test: fix flaky test by buffering possibly chunked content

[npepinpe]

Description

This PR attempts to fix a flaky test which looks to be caused by the interaction of Jackson and the underlying body subscriber.

The error was previously: https://github.com/camunda/zeebe/actions/runs/8846555512/job/24292686857

java.io.IOException: chunked transfer encoding, state: READING_LENGTH
	at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:964)
	at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:133)
	at io.camunda.zeebe.it.management.ControlledActorClockEndpointIT.testPinningTime(ControlledActorClockEndpointIT.java:68)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at java.base/java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:194)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
Caused by: java.io.IOException: chunked transfer encoding, state: READING_LENGTH
	at java.net.http/jdk.internal.net.http.common.Utils.wrapWithExtraDetail(Utils.java:391)
	at java.net.http/jdk.internal.net.http.Http1Response$BodyReader.onReadError(Http1Response.java:676)
	at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.checkForErrors(Http1AsyncReceiver.java:302)
	at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.flush(Http1AsyncReceiver.java:268)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$LockingRestartableTask.run(SequentialScheduler.java:182)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:149)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:207)
	at java.net.http/jdk.internal.net.http.HttpClientImpl$DelegatingExecutor.execute(HttpClientImpl.java:177)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:282)
	at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:251)
	at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.onReadError(Http1AsyncReceiver.java:516)
	at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.lambda$handlePendingDelegate$3(Http1AsyncReceiver.java:380)
	at java.net.http/jdk.internal.net.http.Http1AsyncReceiver$Http1AsyncDelegateSubscription.cancel(Http1AsyncReceiver.java:163)
	at java.net.http/jdk.internal.net.http.common.HttpBodySubscriberWrapper$SubscriptionWrapper.cancel(HttpBodySubscriberWrapper.java:92)
	at java.net.http/jdk.internal.net.http.ResponseSubscribers$HttpResponseInputStream.close(ResponseSubscribers.java:653)
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._closeInput(UTF8StreamJsonParser.java:251)
	at com.fasterxml.jackson.core.base.ParserBase.close(ParserBase.java:412)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4913)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3885)
	at io.camunda.zeebe.it.management.ControlledActorClockEndpointIT$ResponseHandler.unsafeRead(ControlledActorClockEndpointIT.java:138)
	at java.base/java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:684)
	at java.base/java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:662)
	at java.base/java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:2200)
	at java.base/java.util.concurrent.CompletableFuture$MinimalStage.thenApply(CompletableFuture.java:2948)
	at java.net.http/jdk.internal.net.http.ResponseSubscribers$MappingSubscriber.getBody(ResponseSubscribers.java:843)
	at java.net.http/jdk.internal.net.http.common.HttpBodySubscriberWrapper.getBody(HttpBodySubscriberWrapper.java:360)
	at java.net.http/jdk.internal.net.http.ResponseSubscribers.lambda$getBodyAsync$3(ResponseSubscribers.java:1175)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.io.IOException: subscription cancelled

From it, we can see that the body subscriber (which is an InputStream) ends up getting closed by Jackson. On close, it will cancel the subscription. On cancelling, it will run its action, which calls onReadError with the IOException containing the subscription cancelled message.

The question for me here is, why is this not happening every time? Why is it flaky? 🤔

At any rate, I think we can just work around this by simply changing the underlying subscriber to ofByteArray, which will buffer the complete response first. This response tends to be small anyway, and this is only for a test.

[lenaschoenburg]

I think the reason it's flaky is because it's a race condition between the HTTP handling completing and Jackson finishing the deserialization and closing the InputStream. From BodySubscribers.ofInputStream():

To ensure that all resources associated with the corresponding exchange are properly released the caller must ensure to either read all bytes until EOF is reached, or call InputStream. close if it is unable or unwilling to do so. Calling close before exhausting the stream may cause the underlying HTTP connection to be closed and prevent it from being reused for subsequent operations.

I don't think Jackson waits for EOF necessarily, see FasterXML/jackson-jaxrs-providers#108 (comment)

[backport-action]

Successfully created backport PR for stable/8.2:

• [Backport stable/8.2] test: fix flaky test by buffering possibly chunked content #18064

[backport-action]

Successfully created backport PR for stable/8.3:

• [Backport stable/8.3] test: fix flaky test by buffering possibly chunked content #18065

[backport-action]

Successfully created backport PR for stable/8.4:

• [Backport stable/8.4] test: fix flaky test by buffering possibly chunked content #18066

[backport-action]

Successfully created backport PR for stable/8.5:

• [Backport stable/8.5] test: fix flaky test by buffering possibly chunked content #18067