8343855: HTTP/2 ConnectionWindowUpdateSender may miss some unprocessed DataFrames from closed streams #21991

dfuch · 2024-11-08T17:42:22Z

JDK-8342075 has introduced more flow controls checks, but also introduced a race condition where DataFrames for closed streams may fail to be discounted from the connection window.

The consequence is that WINDOW_UPDATE frames for the connection window may not be sent when they should, preventing the server from making progress and stalling the connection.

This can be shown by modifying the StreamFlowControlTest to send less but bigger frames (e.g. chunks of 1600 bytes instead of chunks of 12 bytes). With such a modification the test can be seen failing intermittently, when sameClient=true.

The race happens when frames that have been added to Stream::inputQ fail to be drained after the stream is closed (or continue to be added to the inputQ after the stream is closed).

The fix ensures that Stream::drainInputQueue() is called when the stream is closed, and that no further data farme will be added to the inputQ after the stream is marked closed.

The modified StreamFlowControlTest could be observed failing relatively frequently on linux-aarch64 without the fix.
With the fix the test no longer fails.

Progress

Change must be properly reviewed (1 review required, with at least 1 Reviewer)
Change must not contain extraneous whitespace
Commit message must refer to an issue

Issue

JDK-8343855: HTTP/2 ConnectionWindowUpdateSender may miss some unprocessed DataFrames from closed streams (Bug - P3)

Reviewers

Jaikiran Pai (@jaikiran - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/21991/head:pull/21991
$ git checkout pull/21991

Update a local copy of the PR:
$ git checkout pull/21991
$ git pull https://git.openjdk.org/jdk.git pull/21991/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 21991

View PR using the GUI difftool:
$ git pr show -t 21991

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/21991.diff

Using Webrev

Link to Webrev Comment

…d DataFrames from closed streams

bridgekeeper · 2024-11-08T17:43:18Z

👋 Welcome back dfuchs! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2024-11-08T17:43:53Z

@dfuch This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8343855: HTTP/2 ConnectionWindowUpdateSender may miss some unprocessed DataFrames from closed streams

Reviewed-by: jpai

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 72 new commits pushed to the master branch:

c00e20c: 8343285: java.lang.Process is unresponsive and CPU usage spikes to 100%
cc2acd1: 8343286: Missing unchecked cast warning in polymorphic method call
b80ca49: 8344124: JDK-8341411 Broke the build
a08d67c: 8344080: Return type mismatch for jfr_unregister_stack_filter
4c5bc5f: 8343923: GHA: Switch to Xcode 15 on MacOS AArch64 runners
8af304c: 8341411: C2: remove slice parameter from GraphKit::make_load() and GraphKit::store_to_memory()
b72fe75: 8343140: JfrJavaSupport uses the wrong accessors for sub-int fields
133f8f3: 8344014: Simplify TracePhase constructor
d334af0: 8344089: Fix wrong location of TestWrongMinLWiden.java
79345bb: 8339783: Implement JEP 479: Remove the Windows 32-bit x86 Port
... and 62 more: https://git.openjdk.org/jdk/compare/0c281acfb4c87436096cb562d70f800dffa3671a...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

openjdk · 2024-11-08T17:44:27Z

@dfuch The following label will be automatically applied to this pull request:

net

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

mlbridge · 2024-11-08T17:48:36Z

Webrevs

dfuch · 2024-11-08T18:09:02Z

/issue 8343855

openjdk · 2024-11-08T18:09:34Z

@dfuch This issue is referenced in the PR title - it will now be updated.

test/jdk/java/net/httpclient/http2/StreamFlowControlTest.java

Co-authored-by: Andrey Turbanov <turbanoff@gmail.com>

jaikiran · 2024-11-13T10:23:44Z

src/java.net.http/share/classes/jdk/internal/net/http/Stream.java

@@ -1547,6 +1590,8 @@ void cancelImpl(final Throwable e, final int resetFrameErrCode) {
            }
        } catch (Throwable ex) {
            Log.logError(ex);
+        } finally {
+            drainInputQueue();


Hello Daniel, now that the cancelImpl(...) itself drains the queue in a finally block, do you think we should remove the call to drainInputQueue() from the finally block of schedule() after we call cancelImpl(...) there?

Good point.

jaikiran · 2024-11-13T10:30:55Z

test/jdk/java/net/httpclient/http2/StreamFlowControlTest.java

+            } finally {
+                if (t instanceof FCHttp2TestExchange fct) {
+                    fct.responseSent(query);
+                } else fail("Exchange is not %s but %s"


Nit, while we are here, I think it might be better to wrap the contents of the else block in a {} for consistency.

jaikiran · 2024-11-13T10:37:19Z

test/jdk/java/net/httpclient/lib/jdk/httpclient/test/lib/common/HttpServerAdapters.java

+                    t.sendResponseHeaders(405, 0);
+                }
+            }
+            t.getResponseBody().close();


It's a HttpTestExchange, so we don't have clear defined semantics for what happens when HttpTestExchange.getResponseBody() is invoked after the response body is already close()d previously. I think it might be better to move this t.getResponseBody().close() into the individual case blocks to avoid calling t.getResponseBody() after the case GET already closes the response body in its try-with-resources.

OutputStream::close is supposed to be idempotent and I do believe it is in this implementation. It would be a bug if getResponseBody().close() were not. I have followed your suggestion though.

It would be a bug if getResponseBody().close() were not.

Sorry, my comment wasn't clear. What I meant was, I think a second call to getResponseBody() (in theory) could throw an exception if close() has already been called on the OutputStream returned by the previous call to getResponseBody().

Oh - thanks yes. Should be good now.

jaikiran · 2024-11-13T10:38:39Z

test/jdk/java/net/httpclient/lib/jdk/httpclient/test/lib/common/HttpServerAdapters.java

+     * an empty body.
+     * The response is always returned with fixed length.
+     */
+    public static class HttpHeadHandler implements HttpTestHandler {


The class name is slightly misleading since this also handles GET. Perhaps HeadOrGetHandler would be appropriate?

And maybe we should make this a top level public class in this test package to reduce the amount of code we have in the HttpServerAdapters file.

Renamed HttpHeadOrGetHandler. I'd rather keep the implementation in this file. We already have an HttpEchoHandler here, and there are two many classes bearing this name in the test class hierarchy to consider making it top level.

jaikiran

Thank you for the updates, Daniel. This looks good to me.

dfuch · 2024-11-14T08:53:51Z

/integrate

openjdk · 2024-11-14T08:54:51Z

Going to push as commit bd6152f.
Since your change was applied there have been 95 commits pushed to the master branch:

c3776db: 8342936: Enhance java.io.IO with parameter-less println() and readln()
b54bd82: 8344025: Remove unused ISO2022.Encoder.maximumDesignatorLength
abacece: 8344011: Remove usage of security manager from Class and reflective APIs
c977ef7: 8342047: Create Template Assertion Predicates with Halt nodes only instead of uncommon traps
23a8c71: 8341790: Fix ExceptionOccurred in java.desktop
1e97c1c: 8335989: Implement JEP 494: Module Import Declarations (Second Preview)
e7d90b9: 8343460: ZGC: Crash in ZRemembered::scan_page_and_clear_remset
95a00f8: 8343875: Minor improvements of jpackage test library
90e9234: 8344074: RISC-V: C1: More accurate _exception_handler_size and _deopt_handler_size
3b28354: 8339288: Improve diagnostic logging runtime/cds/DeterministicDump.java
... and 85 more: https://git.openjdk.org/jdk/compare/0c281acfb4c87436096cb562d70f800dffa3671a...master

Your commit was automatically rebased without conflicts.

openjdk · 2024-11-14T08:54:59Z

@dfuch Pushed as commit bd6152f.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

8342075: HTTP/2 ConnectionWindowUpdateSender may miss some unprocesse…

a28b5d1

…d DataFrames from closed streams

openjdk bot added the rfr Pull request is ready for review label Nov 8, 2024

openjdk bot added the net net-dev@openjdk.org label Nov 8, 2024

Add missing bug id

318b181

dfuch changed the title ~~8342075: HTTP/2 ConnectionWindowUpdateSender may miss some unprocessed DataFrames from closed streams~~ 8343855: HTTP/2 ConnectionWindowUpdateSender may miss some unprocessed DataFrames from closed streams Nov 8, 2024

Add more comments

9d21e58

turbanoff reviewed Nov 12, 2024

View reviewed changes

test/jdk/java/net/httpclient/http2/StreamFlowControlTest.java Outdated Show resolved Hide resolved

Update test/jdk/java/net/httpclient/http2/StreamFlowControlTest.java

7c3ad2c

Co-authored-by: Andrey Turbanov <turbanoff@gmail.com>

jaikiran reviewed Nov 13, 2024

View reviewed changes

dfuch added 2 commits November 13, 2024 14:40

Integrated review feedback

d401c83

Integrated review feedback

baf9cd6

jaikiran approved these changes Nov 13, 2024

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Nov 13, 2024

openjdk bot added the integrated Pull request has been integrated label Nov 14, 2024

openjdk bot closed this Nov 14, 2024

openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Nov 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

8343855: HTTP/2 ConnectionWindowUpdateSender may miss some unprocessed DataFrames from closed streams #21991

8343855: HTTP/2 ConnectionWindowUpdateSender may miss some unprocessed DataFrames from closed streams #21991

dfuch commented Nov 8, 2024 •

edited by openjdk bot

Loading

bridgekeeper bot commented Nov 8, 2024

openjdk bot commented Nov 8, 2024 •

edited

Loading

openjdk bot commented Nov 8, 2024

mlbridge bot commented Nov 8, 2024 •

edited

Loading

dfuch commented Nov 8, 2024

openjdk bot commented Nov 8, 2024

jaikiran Nov 13, 2024 •

edited

Loading

dfuch Nov 13, 2024

jaikiran Nov 13, 2024

jaikiran Nov 13, 2024

dfuch Nov 13, 2024

jaikiran Nov 13, 2024

dfuch Nov 13, 2024

jaikiran Nov 13, 2024

jaikiran Nov 13, 2024

dfuch Nov 13, 2024

jaikiran left a comment

dfuch commented Nov 14, 2024

openjdk bot commented Nov 14, 2024

openjdk bot commented Nov 14, 2024

8343855: HTTP/2 ConnectionWindowUpdateSender may miss some unprocessed DataFrames from closed streams #21991

8343855: HTTP/2 ConnectionWindowUpdateSender may miss some unprocessed DataFrames from closed streams #21991

Conversation

dfuch commented Nov 8, 2024 • edited by openjdk bot Loading

Progress

Issue

Reviewers

Reviewing

bridgekeeper bot commented Nov 8, 2024

openjdk bot commented Nov 8, 2024 • edited Loading

openjdk bot commented Nov 8, 2024

mlbridge bot commented Nov 8, 2024 • edited Loading

Webrevs

dfuch commented Nov 8, 2024

openjdk bot commented Nov 8, 2024

jaikiran Nov 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jaikiran left a comment

Choose a reason for hiding this comment

dfuch commented Nov 14, 2024

openjdk bot commented Nov 14, 2024

openjdk bot commented Nov 14, 2024

dfuch commented Nov 8, 2024 •

edited by openjdk bot

Loading

openjdk bot commented Nov 8, 2024 •

edited

Loading

mlbridge bot commented Nov 8, 2024 •

edited

Loading

jaikiran Nov 13, 2024 •

edited

Loading