[CI] GoogleCloudStorageBlobStoreRepositoryTests.testWriteReadLarge (and others) causes assertion error in HTTP server

[dakrone]

On https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+matrix-java-periodic/ES_RUNTIME_JAVA=corretto8,nodes=general-purpose/538/console / https://gradle-enterprise.elastic.co/s/3f3lkoii653zq

The test fails due to

09:42:14     java.lang.AssertionError: Only index blobs should remain in repository but found [foobar]
09:42:14     Expected: a collection with size <0>
09:42:14          but: collection size was <1>
09:42:14         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
09:42:14         at org.junit.Assert.assertThat(Assert.java:956)
09:42:14         at org.elasticsearch.repositories.blobstore.ESMockAPIBasedRepositoryIntegTestCase.tearDownHttpServer(ESMockAPIBasedRepositoryIntegTestCase.java:112)

However, this is largely due in part to the HTTP server dying and causing a bunch of

09:42:14   2> com.google.cloud.storage.StorageException: Read timed out
09:42:14         at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:227)
09:42:14         at com.google.cloud.storage.spi.v1.HttpStorageRpc.write(HttpStorageRpc.java:762)
09:42:14         at com.google.cloud.storage.BlobWriteChannel$1.run(BlobWriteChannel.java:60)
...
09:42:14         Caused by:
09:42:14         java.net.SocketTimeoutException: Read timed out
09:42:14             at java.net.SocketInputStream.socketRead0(Native Method)
09:42:14             at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)

The server seems to die due to a very unhelpful assertion failure in the actual ServerImpl:

09:42:14   2> ADVERTENCIA: Uncaught exception in thread: Thread[Thread-6,5,TGRP-GoogleCloudStorageBlobStoreRepositoryTests]
09:42:14   2> java.lang.AssertionError
09:42:14   2> 	at __randomizedtesting.SeedInfo.seed([E55E3CFA3325D9BE]:0)
09:42:14   2> 	at sun.net.httpserver.ServerImpl.responseCompleted(ServerImpl.java:795)
09:42:14   2> 	at sun.net.httpserver.ServerImpl$Dispatcher.handleEvent(ServerImpl.java:284)
09:42:14   2> 	at sun.net.httpserver.ServerImpl$Dispatcher.run(ServerImpl.java:343)
09:42:14   2> 	at java.lang.Thread.run(Thread.java:748)

I was unable to reproduce this on the 7.x branch.

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore)

[original-brownbear]

This is very likely yet another instance of a JDK bug.
See #51933 for explanation and fix. I'll try to track it down today before my vacation.

[dakrone]

This one looks like it might be related: https://gradle-enterprise.elastic.co/s/vrjiq7uidkdv4 it has the same Read timed out error, but a different (410) exception. I don't see the assertion tripping, but it could also be not provided in the output.

Let me know if it looks related or whether I should open a new issue for it.

[original-brownbear]

@dakrone thanks for posting that one. It's related and explains the issue perfectly :) Somehow we're failing to fully drain a stream there ...

[original-brownbear]

@tlrx I don't have any more time to look into this one I'm afraid. Maybe you can have a look next week while I'm away?

This is definitely the same assertion from https://bugs.openjdk.java.net/browse/JDK-8180754 but I can't find a spot where we wouldn't drain the input stream before writing the response at this point.
My suggestion would be to maybe not waste too much time and simply skip these tests on JDK-8.
I haven't seen this fail on anything but JDK-8 and that's in line with the open JDK bug that seems to only apply to 8. If we just skip 8 we retain all the coverage and this ordeal is finally over with :D

[dakrone]

This one failed again https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+periodic/833/console / https://gradle-enterprise.elastic.co/s/7bhod6xfqi5us

I'm going to mute it for now.