Metrics vertx_http_client_active_requests guage can't decrease zero

[Bugxyb]

Version

Vert.x 4.4.6

Context

we use vert.x metrics to count http client active request, and we see the gauge value increase. I stop send traffic to server instance, when the active connections is zero, the vertx_http_client_active_requests guage is not zero.

Extra

1. we scrape the vertx_http_client_active_request and vertx_http_client_active_connections.
2. find the vertx_http_client_active_connections is around 900, but vertx_http_client_active_request continue to increase.
3. we stop traffic when vertx_http_client_active_connections to zero, but vertx_http_client_active_request is equal 4873.

It seems these requests did not successfully call the metrics decrease.

Relate

we open the issue in eclipse-vertx/vert.x#4936

[tsegismont]

Have you been able to create a small reproducer?

[Bugxyb]

Have you been able to create a small reproducer?

it seems difficult to create a small reproduce, those data is provided by online service. but we use the http client to create the request.

    protected Future<WebClientResponse> execute(String url,
                                                WebClientEntity entity,
                                                boolean responseBody) {
        final long time = TIME.get();
        final Promise<WebClientResponse> promise = Promise.promise();
        final RequestOptions requestOptions = new RequestOptions()
                .setMethod(entity.httpMethod())
                .setTimeout(entity.connectTimeoutMs())
                .setAbsoluteURI(url);

        httpClient.request(requestOptions)
                .onComplete(ignored -> recordTimer(this.conncetTimer, time))
                .onSuccess(httpRequest -> request(promise, httpRequest, entity, responseBody))
                .onFailure(throwable -> handleConnectFailed(throwable, promise));

        return promise.future();
    }

    private void request(Promise<WebClientResponse> promise, HttpClientRequest request,
                         WebClientEntity entity, Boolean responseBody) {
        final long time = TIME.get();
        request.headers().addAll(entity.headers());
        request.setTimeout(entity.timeoutMs());
        final Future<HttpClientResponse> httpClientResponse;
        final TimerPromise<WebClientResponse> timerPromise;

        httpClientResponse = entity.withBody() ? request.send(entity.getBody()) : request.send();
        timerPromise = TimerPromise.promise(vertx, entity.timeoutMs(), request::reset);
        httpClientResponse
                .onSuccess(response -> handleResponse(timerPromise, response, responseBody))
                .onFailure(timerPromise::tryFail);
        timerPromise.future()
                .onComplete(ignored -> recordTimer(this.socketTimer, time))
                .onSuccess(promise::complete)
                .onFailure(throwable -> handleFailed(throwable, promise));
    }

timerPromise is timeout to cancel request(using request reset)

[tsegismont]

I wrote this test:

  @Test
  public void shouldDecrementActiveRequestsWhenRequestIsReset(TestContext ctx) {
    vertx = vertx(ctx);
    int numRequests = 10;
    Async doneLatch = ctx.async(numRequests * 2);
    httpServer = vertx.createHttpServer()
      .requestHandler(req -> {
        req.pause();
        vertx.setTimer(500, l -> {
          req.end().onComplete(ctx.asyncAssertSuccess(v1 -> {
            req.response().end().onComplete(v2 -> {
              doneLatch.countDown();
            });
          }));
          req.resume();
        });
      });
    Async listenLatch = ctx.async();
    httpServer
      .listen(9195, "127.0.0.1")
      .onComplete(ctx.asyncAssertSuccess(s -> listenLatch.complete()));
    listenLatch.awaitSuccess(20_000);
    HttpClient client = vertx.createHttpClient();
    for (int i = 0; i < numRequests; i++) {
      client.request(HttpMethod.POST, 9195, "127.0.0.1", "/")
        .onComplete(ctx.asyncAssertSuccess(req -> {
          req
            .response()
            .compose(HttpClientResponse::body);
          req.end("chunk").onComplete(ctx.asyncAssertSuccess(v -> {
            vertx.setTimer(250, l-> {
              ctx.assertTrue(req.reset());
              doneLatch.countDown();
            });
          }));
        }));
    }
    doneLatch.awaitSuccess(20_000);
    List<Datapoint> res = listDatapoints(startsWith("vertx.http.client.active.requests"));
    assertThat(res).extracting(Datapoint::value).contains(0.0);
  }

Tried with both HTTP1 and HTTP2 and failed to reproduce

[tsegismont]

Is it HTTP1 or HTTP2 you're working with?

[Bugxyb]

Is it HTTP1 or HTTP2 you're working with?

use HTTP1

[tsegismont]

@Bugxyb if you can't create a reproducer, would you be able to create a byteman rule and check if there as many calls to io.vertx.core.spi.metrics.ClientMetrics#requestBegin as the sum of calls to io.vertx.core.spi.metrics.ClientMetrics#requestEnd, io.vertx.core.spi.metrics.ClientMetrics#requestReset and io.vertx.core.spi.metrics.ClientMetrics#responseBegin.

That would tell us if the problem comes from Vert.x core or if it's in Vert.x Micrometer.

[tsegismont]

@Bugxyb any news about this?

[Bugxyb]

@Bugxyb any news about this?

sry, I don't have time to deal it with byteman recently.

[tsegismont]

Closing, please reopen in you can provide a reproducer