8294047: HttpResponseInputStream swallows interrupts #11323

DarraghClarke · 2022-11-23T14:21:13Z

Currently if a HttpResonseInputStream gets interrupted while reading it will just swallow the exception and continue,

This PR changes it to close the stream and throw an IOException, I added a test to cover this which just uses two threads to read the stream then interrupt it.

Progress

Change must be properly reviewed (1 review required, with at least 1 Reviewer)
Change must not contain extraneous whitespace
Commit message must refer to an issue
Change requires a CSR request to be approved

Issues

JDK-8294047: HttpResponseInputStream swallows interrupts
JDK-8298163: HttpResponseInputStream swallows interrupts (CSR)

Reviewers

Daniel Fuchs (@dfuch - Reviewer)
Vyom Tewari (@vyommani - Committer) ⚠️ Review applies to 5fc55a3a
Jaikiran Pai (@jaikiran - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk pull/11323/head:pull/11323
$ git checkout pull/11323

Update a local copy of the PR:
$ git checkout pull/11323
$ git pull https://git.openjdk.org/jdk pull/11323/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 11323

View PR using the GUI difftool:
$ git pr show -t 11323

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/11323.diff

DarraghClarke · 2022-11-23T14:21:46Z

/csr needed

bridgekeeper · 2022-11-23T14:22:13Z

👋 Welcome back DarraghClarke! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2022-11-23T14:25:57Z

@DarraghClarke has indicated that a compatibility and specification (CSR) request is needed for this pull request.

@DarraghClarke please create a CSR request for issue JDK-8294047 with the correct fix version. This pull request cannot be integrated until the CSR request is approved.

openjdk · 2022-11-23T14:28:26Z

@DarraghClarke The following label will be automatically applied to this pull request:

net

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

mlbridge · 2022-11-23T14:32:21Z

Webrevs

dfuch · 2022-11-23T15:05:51Z

src/java.net.http/share/classes/jdk/internal/net/http/ResponseSubscribers.java

@@ -481,7 +481,8 @@ private ByteBuffer current() throws IOException {
                    if (debug.on()) debug.log("Next Buffer");
                    currentBuffer = currentListItr.next();
                } catch (InterruptedException ex) {
-                    // continue
+                    close();
+                    throw new IOException("interrupted", ex);


I'd suggest to throw InterruptedIOException here instead. Also it might be better to swallow any exception close() might throw - using try-finally around close() should do that.

dfuch · 2022-11-23T15:08:49Z

test/jdk/java/net/httpclient/HttpResponseInputStreamInterruptTest.java

+            Thread clientThread = createClientThread(countDownLatch, port);
+            Thread interrupterThread = new Thread(() -> {
+                try {
+                    countDownLatch.await();


It would be useful to add a comment before waiting on the latch to explain what you are waiting for.

dfuch · 2022-11-23T15:10:14Z

test/jdk/java/net/httpclient/HttpResponseInputStreamInterruptTest.java

+
+    static class Handler implements HttpHandler {
+
+        CountDownLatch countDownLatch;


Maybe a better name than countDownLatch could be found - that could give a clue on what this latch is used to wait for?

I was thinking maybe messageLatch, messageReceivedLatch or clientReadyLatch

Maybe we rename both:

countDownLatch => interruptReadyLatch interruptLatch => interruptDoneLatch

dfuch · 2022-11-24T17:10:03Z

test/jdk/java/net/httpclient/HttpResponseInputStreamInterruptTest.java

+        server = HttpServer.create(addr, 0);
+        port = server.getAddress().getPort();
+        Handler handler = new Handler(countDownLatch, interruptLatch);
+        server.createContext("/", handler);


It would be good to use a path unique for this test to avoid potential interferences. I would suggest registering the context under "/HttpResponseInputStreamInterruptTest/" instead of "/".

And use that when building the URI below as well.

This is important because the handler is using latches, and therefore can be invoked only once...

dfuch · 2022-11-25T11:28:46Z

src/java.net.http/share/classes/jdk/internal/net/http/ResponseSubscribers.java

+                    try {
+                        close();
+                    } catch (IOException ignored) {
+                    }
+                    throw new InterruptedIOException();


Thanks for doing that. Maybe we should keep the caught InterruptedException as the cause of the new InterruptedIOException. I'd suggest to add a new utility method to the ...common.Utils class for that. I mean - something like:

InterruptedIOException Utils.toInterruptedIOException(InterruptedException ex);

So something in utils like return new InterruptedIOException(String.valueOf(exxeption)); ?

No something that would call initCause(ex) to set the root cause - as AFAICS there's no constructor that takes a root cause.

ah ok, so create a new InterruptedIOException and set the initcause to be the interruptedException.
is there anything else that would need to be set?

preserving the root cause is the main thing.

dfuch · 2022-11-25T11:29:47Z

test/jdk/java/net/httpclient/HttpResponseInputStreamInterruptTest.java

+        CountDownLatch interruptReadyLatch;
+        CountDownLatch interruptDoneLatch;


should be final

I believe this change was missed and is pending?

dfuch · 2022-11-25T11:30:54Z

test/jdk/java/net/httpclient/HttpResponseInputStreamInterruptTest.java

-                countDownLatch.countDown();
-                assertThrows(IOException.class, () -> response.body().readAllBytes(), "excepted IOException");
+                interruptReadyLatch.countDown();
+                assertThrows(InterruptedIOException.class, () -> response.body().readAllBytes(), "excepted IOException");


Maybe change the message while you're at it since we're actually expecting InterruptedIOException

dfuch · 2022-11-28T14:09:47Z

src/java.net.http/share/classes/jdk/internal/net/http/ResponseSubscribers.java

@@ -486,7 +486,8 @@ private ByteBuffer current() throws IOException {
                        close();
                    } catch (IOException ignored) {
                    }
-                    throw new InterruptedIOException();
+                    // Throw InterruptedIOException where the initCause is set to the caught InterruptedException


nit: could you break this long line after "is"

dfuch · 2022-11-28T14:10:52Z

src/java.net.http/share/classes/jdk/internal/net/http/common/Utils.java

+        InterruptedIOException interruptedIOException = new InterruptedIOException();
+        interruptedIOException.initCause(ex);
+        return interruptedIOException;
+    }


The copyright years need updating.

dfuch · 2022-11-29T17:31:50Z

Actually, there's one more thing that we probably should do. The API documentation of HttpResponse.BodySubscribers::ofInputStream has an @apiNote that emphasizes the need to read the stream to exhaustion, or call close, in order to release all resources.
We probably should also add an @apiNote to document that the default implementation returned by that method throw InterruptedIOException if the thread is interrupted while reading from the stream.

vyommani · 2022-11-30T05:19:22Z

src/java.net.http/share/classes/jdk/internal/net/http/ResponseSubscribers.java

+                    }
+                    // Throw InterruptedIOException where the initCause is
+                    // set to the caught InterruptedException
+                    throw Utils.toInterruptedIOException(ex);


Don't you think you need to restores the interrupt status of the thread by invoking the Thread.currentThread().interrupt() method and allowing the caller to detect the interrupt if it needs to ?

By convention, any method that exits by throwing an InterruptedException clears interrupt status when it does so. Since we're throwing InterruptedIOException here, I don't think we need to set the interrupt flag. But the caller might do so if they want.

Hmmm... Maybe a better solution would be to throw "ClosedByInterruptException" since we're also closing the input stream. And the specification of "ClosedByInterruptException" clearly state that the interrupt status of the thread will be preserved. Good question @vyommani .

I am not 100% sure but for me the best option is call close() and just restore the interrupt status let the caller decide how to handle the interrupt.

InterruptedIOException is not equivalent to InterrupedException in this way AFAIK. The thread interrupt status should be reset if any exception other than InterruptedException is thrown AFAIK, otherwise thread interruption is likely to be lost.

AlanBateman · 2022-12-01T11:05:06Z

src/java.net.http/share/classes/jdk/internal/net/http/ResponseSubscribers.java

+                    try {
+                        close();
+                    } catch (IOException ignored) {
+                    }


This will swallow the interrupt status, you' need to do Thread.currentThread().interrupt() before throwing the IOException.

DarraghClarke · 2022-12-01T13:32:59Z

Thanks for pointing out those issues, with the last few commits I've hopefully addressed everything, but let me know if there is anything else that need to be changed

dfuch · 2022-12-01T15:50:29Z

test/jdk/java/net/httpclient/HttpResponseInputStreamInterruptTest.java

+
+                // countdown on latch, and assert that an IOException is throw due to the interrupt
+                interruptReadyLatch.countDown();
+                assertThrows(IOException.class, () -> response.body().readAllBytes(), "expected IOException");


I suggest checking that the exception cause is the expected InterruptedException. If I'm not mistaken something like that should do it:

var thrown = assertThrows(IOException.class, () -> response.body().readAllBytes(), "expected IOException"); var cause = thrown.getCause(); assertTrue(cause instanceof InterruptedException, cause + " is not an InterruptedException");

dfuch · 2022-12-01T15:52:45Z

test/jdk/java/net/httpclient/HttpResponseInputStreamInterruptTest.java

+                interruptReadyLatch.countDown();
+                assertThrows(IOException.class, () -> response.body().readAllBytes(), "expected IOException");
+            } catch (Exception e) {
+                throw new RuntimeException(e);


You might want to catch Throwable, then print the stack trace and call Asserts.fail here - because unhandled exceptions in threads might just be swallowed.
Another possibility is to relay the throwable to the main thread and throw it from the main thread.

dfuch · 2022-12-02T15:46:12Z

test/jdk/java/net/httpclient/HttpResponseInputStreamInterruptTest.java

-                throw new RuntimeException(e);
+                var thrown = assertThrows(IOException.class, () -> response.body().readAllBytes(), "expected IOException");
+                var cause = thrown.getCause();
+                assertTrue(cause instanceof InterruptedException, cause + " is not an InterruptedException");


Ah - and of course the test should verify that the thread interrupted status is set too, since we specified that :-)

var thread = Thread.currentThread(); assertTrue(thread.isInterrupted(), "Thread " + thread + " is not interrupted");

dfuch

LGTM - please make sure the test (and tier1 tier2) all pass before integrating

vyommani

Looks OK to me.

jaikiran · 2022-12-05T07:15:41Z

src/java.net.http/share/classes/java/net/http/HttpResponse.java

@@ -1178,6 +1178,12 @@ public static BodySubscriber<Path> ofFile(Path file) {
         * the underlying HTTP connection to be closed and prevent it
         * from being reused for subsequent operations.
         *
+         * @implNote The read method of the default implementation returned


Hello Darragh, should we instead reword this as:

@implNote The {@code read} method of the {@code InputStream} returned by the default implementation of this method will throw an {@code IOException} with the {@link Thread#isInterrupted() thread interrupt status set} if the thread is interrupted while blocking on read. In that case, the request will also be cancelled and the {@code InputStream} will be closed.

Additionally, irrespective of whether or not we decide to reword this, this file will need an update for the copyright year.

Agreed but it should be {@linkplain Thread#isInterrupted() thread interrupt status set}

jaikiran · 2022-12-05T07:17:05Z

src/java.net.http/share/classes/jdk/internal/net/http/ResponseSubscribers.java

@@ -29,6 +29,7 @@
 import java.io.FilePermission;
 import java.io.IOException;
 import java.io.InputStream;
+import java.io.InterruptedIOException;


I think this is now an unused import and we can remove it?

jaikiran · 2022-12-05T07:29:47Z

src/java.net.http/share/classes/jdk/internal/net/http/ResponseSubscribers.java

+                    try {
+                        close();
+                    } catch (IOException ignored) {
+                    }


I don't have specific opinion here and this is more of a question - should we be doing:

} catch (InterruptedException ex) { final IOException toThrow = new IOException(ex); try { close(); } catch (IOException ignored) { toThrow.addSuppressed(ignored); } Thread.currentThread().interrupt(); throw toThrow; }

i.e. should we be adding any failure to close() as a suppressed exception to the IOException that we throw?

I am not sure whether that would be helpful or confusing.

I think that's a good point, I don't really have a strong opinion one way or the other on it but would be curious if anyone else does?

Let's leave it the way you have it now.

jaikiran · 2022-12-05T08:48:55Z

test/jdk/java/net/httpclient/HttpResponseInputStreamInterruptTest.java

+    }
+
+    @Test
+    public void test() {


To keep this method relatively simpler, you could remove the top level try/catch block (which just catches and rethrows) and change this method's signature to throws Exception. If you decide to leave it in the current form, that's fine too.

jaikiran

Thank you for these changes Darragh. Looks fine to me. There was one review comment by Daniel where he noted that the fields of the Handler in the test should be final (which I agree with). From what I can see, that hasn't been addressed in the updates. In the current test, it doesn't functionally impact anything, so it is OK if you want to leave it in this form.

openjdk · 2022-12-06T21:42:43Z

@DarraghClarke This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8294047: HttpResponseInputStream swallows interrupts

Reviewed-by: dfuchs, vtewari, jpai

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 254 new commits pushed to the master branch:

8a9911e: 8295803: Console should be usable in jshell and other environments
5d4c71c: 8281236: (D)TLS key exchange named groups
10356e7: 8298303: (fs) temporarily remove Path.getExtension
3b8c7ef: 8157023: Integrate NMT with JFR
e86f31b: 8298301: C2: assert(main_cmp->in(2)->Opcode() == Op_Opaque1) failed: main loop has no opaque node?
8edb98d: 8165943: LineBreakMeasurer does not measure correctly if TextAttribute.TRACKING is set.
3934484: 8298205: Prefer Member Initialization Lists for JFR classes in os_perf.hpp
389b8f4: 8297298: SequenceInputStream should override transferTo
dd7385d: 8298202: [AIX] Dead code elimination removed jfr constructor used by AIX
29f1c3c: 8298274: Problem list TestSPISigned on Windows
... and 244 more: https://git.openjdk.org/jdk/compare/f4b5065c37e86f4b2ca26da6ce678febe4a52950...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

As you do not have Committer status in this project an existing Committer must agree to sponsor your change. Possible candidates are the reviewers of this PR (@dfuch, @vyommani, @jaikiran) but any other Committer may sponsor as well.

➡️ To flag this PR as ready for integration with the above commit message, type /integrate in a new comment. (Afterwards, your sponsor types /sponsor in a new comment to perform the integration).

DarraghClarke · 2022-12-07T21:52:52Z

/integrate

openjdk · 2022-12-07T21:56:14Z

@DarraghClarke
Your change (at version 9937b57) is now ready to be sponsored by a Committer.

jaikiran · 2022-12-08T01:36:02Z

/sponsor

openjdk · 2022-12-08T01:39:29Z

Going to push as commit 3aa4070.
Since your change was applied there have been 256 commits pushed to the master branch:

af8fb7e: 8282578: AIOOBE in javax.sound.sampled.Clip
8b69a2e: 8298099: [JVMCI] decouple libgraal from JVMCI module at runtime
8a9911e: 8295803: Console should be usable in jshell and other environments
5d4c71c: 8281236: (D)TLS key exchange named groups
10356e7: 8298303: (fs) temporarily remove Path.getExtension
3b8c7ef: 8157023: Integrate NMT with JFR
e86f31b: 8298301: C2: assert(main_cmp->in(2)->Opcode() == Op_Opaque1) failed: main loop has no opaque node?
8edb98d: 8165943: LineBreakMeasurer does not measure correctly if TextAttribute.TRACKING is set.
3934484: 8298205: Prefer Member Initialization Lists for JFR classes in os_perf.hpp
389b8f4: 8297298: SequenceInputStream should override transferTo
... and 246 more: https://git.openjdk.org/jdk/compare/f4b5065c37e86f4b2ca26da6ce678febe4a52950...master

Your commit was automatically rebased without conflicts.

openjdk · 2022-12-08T01:39:54Z

@jaikiran @DarraghClarke Pushed as commit 3aa4070.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

DarraghClarke and others added 3 commits November 22, 2022 17:01

close HttpResponseInputStream and throw exception on interrupt

8aa21d1

cleanup of test

333949d

Merge branch 'openjdk:master' into JDK-8294047

0d8dd4b

openjdk bot added rfr Pull request is ready for review csr Pull request needs approved CSR before integration labels Nov 23, 2022

openjdk bot added the net net-dev@openjdk.org label Nov 23, 2022

dfuch reviewed Nov 23, 2022

View reviewed changes

dfuch reviewed Nov 24, 2022

View reviewed changes

implemented feedback

560719e

dfuch suggested changes Nov 25, 2022

View reviewed changes

set initCause for InterruptedIOException

928176a

dfuch reviewed Nov 28, 2022

View reviewed changes

fixed copyright year and formatting of a comment

83ff9c1

dfuch approved these changes Nov 28, 2022

View reviewed changes

vyommani reviewed Nov 30, 2022

View reviewed changes

DarraghClarke added 2 commits December 1, 2022 10:55

added implementation note and changed exception thrown to IOException

f34f42c

removed changes from Utils

b6a0912

AlanBateman reviewed Dec 1, 2022

View reviewed changes

set interrupt status on thread before throwing exception

6b9be8a

dfuch suggested changes Dec 1, 2022

View reviewed changes

check cause of exception in test

5f83141

dfuch reviewed Dec 2, 2022

View reviewed changes

check thread interrupt status is set

5fc55a3

dfuch approved these changes Dec 2, 2022

View reviewed changes

vyommani reviewed Dec 3, 2022

View reviewed changes

vyommani approved these changes Dec 4, 2022

View reviewed changes

jaikiran reviewed Dec 5, 2022

View reviewed changes

cleanup

9937b57

dfuch approved these changes Dec 5, 2022

View reviewed changes

jaikiran approved these changes Dec 6, 2022

View reviewed changes

openjdk bot removed the csr Pull request needs approved CSR before integration label Dec 6, 2022

openjdk bot added the ready Pull request is ready to be integrated label Dec 6, 2022

openjdk bot added the sponsor Pull request is ready to be sponsored label Dec 7, 2022

openjdk bot added the integrated Pull request has been integrated label Dec 8, 2022

openjdk bot closed this Dec 8, 2022

openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review sponsor Pull request is ready to be sponsored labels Dec 8, 2022


		static class Handler implements HttpHandler {

		CountDownLatch countDownLatch;

		CountDownLatch interruptReadyLatch;
		CountDownLatch interruptDoneLatch;

8294047: HttpResponseInputStream swallows interrupts #11323

8294047: HttpResponseInputStream swallows interrupts #11323

Conversation

DarraghClarke commented Nov 23, 2022 • edited by openjdk bot Loading

Progress

Issues

Reviewers

Reviewing

DarraghClarke commented Nov 23, 2022

bridgekeeper bot commented Nov 23, 2022

openjdk bot commented Nov 23, 2022

openjdk bot commented Nov 23, 2022

mlbridge bot commented Nov 23, 2022 • edited Loading

Webrevs

dfuch Nov 23, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dfuch commented Nov 29, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DarraghClarke commented Dec 1, 2022

Choose a reason for hiding this comment

dfuch Dec 1, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dfuch left a comment

Choose a reason for hiding this comment

vyommani left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jaikiran Dec 6, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jaikiran left a comment

Choose a reason for hiding this comment

openjdk bot commented Dec 6, 2022 • edited Loading

DarraghClarke commented Dec 7, 2022

openjdk bot commented Dec 7, 2022

jaikiran commented Dec 8, 2022

openjdk bot commented Dec 8, 2022

openjdk bot commented Dec 8, 2022

DarraghClarke commented Nov 23, 2022 •

edited by openjdk bot

Loading

mlbridge bot commented Nov 23, 2022 •

edited

Loading

dfuch Nov 23, 2022 •

edited

Loading

dfuch commented Nov 29, 2022 •

edited

Loading

dfuch Dec 1, 2022 •

edited

Loading

jaikiran Dec 6, 2022 •

edited

Loading

openjdk bot commented Dec 6, 2022 •

edited

Loading