8245245: WebSocket can lose the URL encoding of URI query parameters #1558

Karm · 2022-11-30T01:52:47Z

Proposes to backport JDK-8245245.

The backport is clean as far as the actual OpeningHandshake.java goes. The test needed a little tweak so as to compile with SimpleSSLContext and also to handle the fact that the erroneous response does not bring a response body.

The test passes with the patch, fails without it.

$ make clean run-test TEST="jtreg:test/jdk/java/net/httpclient/websocket/HandshakeUrlEncodingTest.java"
...
==============================
Test summary
==============================
   TEST                                              TOTAL  PASS  FAIL ERROR   
   jtreg:test/jdk/java/net/httpclient/websocket/HandshakeUrlEncodingTest.java
                                                         1     1     0     0   
==============================
TEST SUCCESS

Stopping sjavac server
Finished building targets 'clean run-test' in configuration 'linux-x86_64-normal-server-release'

In addition to that, I compiled and executed the original WebSocketTest.java reproducer found on JDK-8245245 JIRA.

Unpatched Temurin-11.0.17+8 ❌

$ java WebSocketTest 
Http Request
http://localhost:8000/?raw=abc+def/ghi=xyz&encoded=abc%2Bdef%2Fghi%3Dxyz
Server RequestURI: /?raw=abc+def/ghi=xyz&encoded=abc%2Bdef%2Fghi%3Dxyz
WebSocket Request
ws://localhost:8000/?&raw=abc+def/ghi=xyz&encoded=abc%2Bdef%2Fghi%3Dxyz
Server RequestURI: /?&raw=abc+def/ghi=xyz&encoded=abc+def/ghi=xyz

Patched jdk11u ✔️

$ java WebSocketTest 
Http Request
http://localhost:8000/?raw=abc+def/ghi=xyz&encoded=abc%2Bdef%2Fghi%3Dxyz
Server RequestURI: /?raw=abc+def/ghi=xyz&encoded=abc%2Bdef%2Fghi%3Dxyz
WebSocket Request
ws://localhost:8000/?&raw=abc+def/ghi=xyz&encoded=abc%2Bdef%2Fghi%3Dxyz
Server RequestURI: /?&raw=abc+def/ghi=xyz&encoded=abc%2Bdef%2Fghi%3Dxyz

The patched version correctly leaves the latter part of the query param encoded.

Progress

Change must be properly reviewed (1 review required, with at least 1 Reviewer)
Change must not contain extraneous whitespace
Commit message must refer to an issue

Issues

JDK-8245245: WebSocket can lose the URL encoding of URI query parameters
JDK-8298588: WebSockets: HandshakeUrlEncodingTest unnecessarily depends on a response body

Reviewers

Paul Hohensee (@phohensee - Reviewer)
Severin Gehwolf (@jerboaa - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk11u-dev pull/1558/head:pull/1558
$ git checkout pull/1558

Update a local copy of the PR:
$ git checkout pull/1558
$ git pull https://git.openjdk.org/jdk11u-dev pull/1558/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 1558

View PR using the GUI difftool:
$ git pr show -t 1558

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk11u-dev/pull/1558.diff

bridgekeeper · 2022-11-30T01:53:23Z

👋 Welcome back Karm! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

jerboaa · 2022-11-30T09:34:22Z

@Karm Please change the PR title to Backport c07ce7eec71aefbd3cb624e03ca53f5148d01f19 so that the bots recognize this as a backport. Please be sure to run :jdk_net test group before/after as well.

openjdk · 2022-11-30T09:54:50Z

This backport pull request has now been updated with issue and summary from the original commit.

mlbridge · 2022-11-30T09:59:35Z

Webrevs

phohensee

Looks good except for using getResponse().uri() and a few formatting issues in HandshakeUrlEncodingTest.java.

The change to use getResponse().uri() instead of getResponse().body() fixes a bug in tip, so can't be part of a backport. The thing to do is use the buggy code from the original commit in this backport, fix the problem in tip, then backport that fix to 19u, 17u, and 11u.

On formatting issues, future backports are easier if you match the original formatting as closely as possible. Looks like you're using and IDE that auto-formats.

At line 59:

import static java.lang.System.out;
import static java.net.http.HttpClient.Builder.NO_PROXY;
import static org.testng.Assert.*;

should be

import static java.net.http.HttpClient.Builder.NO_PROXY;
import static org.testng.Assert.*;
import static java.lang.System.out;

to more closely follow the original. In fact, if all the test uses is assertEquals, assertNotNull, and fail, just use the original commit code.

Indent/format the same as the original commit: lines 85-88, 89-92, 106-108, 111, 115, 133-135, 137, 141-146, 153-158, 174-176, 188-190, and missing blank line after 196. Might have missed somthing, but you get the idea.

Karm · 2022-12-02T04:21:31Z

@phohensee Thank you for the review.

Ad formatting: Ack. Will do.

Ad

The change to use getResponse().uri() instead of getResponse().body() fixes a bug in tip, so can't be part of a backport.

Oh...I see. So the fact that the body is returned even though there was a failure is a bug, not a feature/desirable chage? I will browse the Jira and check the spec to make sure I am not bending the test to a faulty implementation then.

Cheers
Karm

jerboaa · 2022-12-02T08:52:48Z

Oh...I see. So the fact that the body is returned even though there was a failure is a bug, not a feature/desirable chage? I will browse the Jira and check the spec to make sure I am not bending the test to a faulty implementation then.

Even if you discover a discrepancy, this would need to get fixed in JDK head first and then backported. Please keep that in mind.

Karm · 2022-12-02T16:01:02Z

Helo, @phohensee,

I changed the behavior of the test in the tip (HEAD), so as I can then backport it along the JDK-8245245 patch to 19u, 17u, and 11u.

openjdk/jdk#11486

@jerboaa If I am reading the spec right, there is IMHO no need to change the behavior of the implementation.

Karm · 2022-12-02T17:50:58Z

@jerboaa I realized the patch is already backported to JDK 19 and JDK 17. So it's just JDK 11 I'd like to backport JDK-8245245 to. And IIUC, the change in openjdk/jdk#11486 needed for JDK 11 will need backporting to JDK 19, JDK 17 too.

jerboaa · 2022-12-05T15:36:48Z

@jerboaa I realized the patch is already backported to JDK 19 and JDK 17. So it's just JDK 11 I'd like to backport JDK-8245245 to. And IIUC, the change in openjdk/jdk#11486 needed for JDK 11 will need backporting to JDK 19, JDK 17 too.

Yes.

jerboaa · 2022-12-06T10:05:55Z

FWIW, since https://bugs.openjdk.org/browse/JDK-8240666 is not in 11u, it would be acceptable to alter the test to account for this. A comment in the test could explain that. I don't think backporting JDK-8240666 would be appropriate (behaviour change) and it looks like getting the test change (account for the optional body) in JDK head might not be wanted.

dfuch · 2022-12-06T13:47:09Z

It is important that the test checks the URI received by the server, and that's probably why the body was used here. The server writes the URI it receives in the response body. This provides an end-to-end check that what was received is what we expected to send. Note that the server doesn't actually supports WebSocket and that's why it always replies with 400.

Karm · 2022-12-12T15:05:01Z

Hello @dfuch, @jerboaa, @phohensee, @jaikiran

I'd like this tets change to go forward: openjdk/jdk#11486
It correctly fails without JDK-8245245 and passes with it. It is not necessary to be checking the body.

When that test change is integrated, I'd initiate its backport. Close this pr (#1558) and open a new one, that would just cleanly add JDK-8245245...

I hope that is all right process wise.

dfuch · 2022-12-12T15:19:58Z

It correctly fails without JDK-8245245 and passes with it. It is not necessary to be checking the body.

Thanks for double checking. In that case I believed you're covered, I personally have no objection in that case. I mostly commented because I don't think there's a bug in the original test. It just uses a different method to check for success. As for approval I'll defer to the JDK 11 maintainers.

Karm · 2022-12-16T14:32:59Z

Hello @dfuch, @phohensee,

With jdk/pull/11486 merged, I would kindly ask for a review of this backport to JDK11u.

the patch in OpeningHandshake.java did not require any changes
the diff for the test between what is in mainline and what I propose here is this: https://editor.mergely.com/Hh1c3Z1N/

thx
K.

jerboaa

Please use /issue add JDK-8298588 as the test now contains those changes too.

test/jdk/java/net/httpclient/websocket/HandshakeUrlEncodingTest.java

phohensee

Lgtm with Severin's recommendations.

openjdk · 2022-12-16T16:32:44Z

⚠️ @Karm the full name on your profile does not match the author name in this pull requests' HEAD commit. If this pull request gets integrated then the author name from this pull requests' HEAD commit will be used for the resulting commit. If you wish to push a new commit with a different author name, then please run the following commands in a local repository of your personal fork:

$ git checkout backport-JDK-8245245
$ git commit --author='Preferred Full Name <you@example.com>' --allow-empty -m 'Update full name'
$ git push

openjdk · 2022-12-16T16:32:46Z

@Karm This change now passes all automated pre-integration checks.

After integration, the commit message for the final commit will be:

8245245: WebSocket can lose the URL encoding of URI query parameters
8298588: WebSockets: HandshakeUrlEncodingTest unnecessarily depends on a response body

The fix updates jdk.internal.net.http.websocket.OpeningHandshake to avoid double encoding and decoding of URL

Reviewed-by: phh, sgehwolf

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 40 new commits pushed to the master branch:

c46dcc5: 8252715: Problem list java/awt/event/KeyEvent/KeyTyped/CtrlASCII.java on Linux
ca49dbf: 8298527: Cygwin's uname -m returns different string than before
bcda041: 8256240: Reproducible builds should turn on the "deterministic" flag for Visual Studio
ffea913: 8244592: Start supporting SOURCE_DATE_EPOCH
3e65a7c: 8282511: Use fixed certificate validation date in SSLExampleCert template
23aeca9: 8296611: Problemlist several sun/security tests until JDK-8295343 is resolved
a3d0522: 8282398: EndingDotHostname.java test fails because SSL cert expired
c356873: Merge
88954e1: 8298737: 8296772 backport to jdk11u caused build error on sparc
77d919a: 8297804: (tz) Update Timezone Data to 2022g
... and 30 more: https://git.openjdk.org/jdk11u-dev/compare/b5555c191537c27e6b7dcf7ac15c4dfe2967688e...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

As you do not have Committer status in this project an existing Committer must agree to sponsor your change. Possible candidates are the reviewers of this PR (@phohensee, @jerboaa) but any other Committer may sponsor as well.

➡️ To flag this PR as ready for integration with the above commit message, type /integrate in a new comment. (Afterwards, your sponsor types /sponsor in a new comment to perform the integration).

phohensee · 2022-12-16T17:34:17Z

/issue add JDK-8298588

test/jdk/java/net/httpclient/websocket/HandshakeUrlEncodingTest.java

RealCLanger · 2022-12-19T22:02:35Z

@Karm Thanks for contributing this backport to JDK 11 Updates. This looks good to me. As @jerboaa mentioned, the next step would be to add the fix request label and information to the JBS bug.

I'd have one further request: Can you please backport your fix for JDK-8298588 to JDK 20 and JDK17 that we don't miss it out there. For JDK20, you can use the "/backport jdk20" comment on the commit in head. If the backport is clean, which I assume, you can right away integrate as it complies to the RDP1 rules (testfix). For jdk17u-dev, you'll need a PR and maintainer approval by labeling jdk17u-fix-request

jerboaa · 2022-12-20T11:29:53Z

For JDK20, you can use the "/backport jdk20" comment on the commit in head.

I think that's only available for committers. Just FYI.

Karm · 2022-12-20T11:51:05Z

For JDK20, you can use the "/backport jdk20" comment on the commit in head.

I think that's only available for committers. Just FYI.

It is. I tried that the other day: openjdk/jdk@c07ce7e#commitcomment-91811154

RealCLanger · 2022-12-20T16:06:25Z

OK, let me try to trigger the backports for you, see here...

Karm · 2022-12-21T09:42:18Z

Thanks, lemme process it today.

Karm · 2022-12-21T10:05:48Z

Hello, @jerboaa, This is the JBS text proposal, formatted for JIRA:


I would like [JDK-8245245|https://bugs.openjdk.org/browse/JDK-8245245] to get backported to JDK 11
so as more libraries can start using JDK's own WebSocket client instead of depending
on other implementations.

For instance, [Fabric8 Kubernetes client|https://github.com/fabric8io/kubernetes-client/blob/master/httpclient-jdk/README.md#jdk-client-for-fabric8]
has this very issue with JDK's WebSocket client and it uses OkHttp3 or Vert.x implementations instead.

I used these [JBang|https://www.jbang.dev/] scripts to briefly showcase that both Vert.x and OkHttp3 implementations
are fine running on JDK 11 and JDK 11's WebSocket client needs fixing: (It uses Undertow as the server)

[JDKClient.java|https://gist.github.com/Karm/58959f7bc1d3ef675eecd1e12e56094c]
[OKHttp3Client.java|https://gist.github.com/Karm/6ed845a22a8b8331f95292bb992ee7e0]
[VertXClient.java|https://gist.github.com/Karm/a04a778f352e8d29667eb38a219d4e4b]

{code}
$ java --version
openjdk 11.0.17 2022-10-18

$ ./JDKClient.java
[jbang] Building jar...
The query string was: &raw=abc+def/ghi=xyz&encoded=abc+def/ghi=xyz

$ ./OKHttp3Client.java
[jbang] Building jar...
The query string was: raw=abc+def/ghi=xyz&encoded=abc%2Bdef%2Fghi%3Dxyz

$ ./VertXClient.java
[jbang] Building jar...
The query string was: raw=abc+def/ghi=xyz&encoded=abc%2Bdef%2Fghi%3Dxyz
{code}

Patched, see the JDK client fixed:
{code}
$ java --version
openjdk 11.0.255-internal 2023-01-17

$ ./JDKClient.java 
The query string was: &raw=abc+def/ghi=xyz&encoded=abc%2Bdef%2Fghi%3Dxyz

$ ./OKHttp3Client.java 
The query string was: raw=abc+def/ghi=xyz&encoded=abc%2Bdef%2Fghi%3Dxyz

$ ./VertXClient.java 
The query string was: raw=abc+def/ghi=xyz&encoded=abc%2Bdef%2Fghi%3Dxyz
{code}

Thanks
K.

Edit, short version:

Fix Request
Approve backporting JDK-8245245 to 11u. It fixes a WebSocket client related query param encoding issue and should be a low risk one.
Testing: ran jdk_net tests without regressions. Patch doesn't apply clean as the test needed changing.
Reviewed by Paul Hohensee <phh>, Daniel Fuchs <dfuchs>, Severin Gehwolf <sgehwolf>.

jerboaa

Seems OK to me.

phohensee

Lgtm.

phohensee · 2022-12-21T19:20:55Z

Tagged JDK-8298588. Commit pending jdk11u-fix-yes on JDk-8245245 and JDK-8298588.

GoeLin · 2022-12-22T09:22:44Z

Hi @Karm,
I saw your JBS comment. Next time please reason why you consider the risk low.
Thanks.

Karm · 2022-12-23T13:31:40Z

OK, let me try to trigger the backports for you, see here...

THX @RealCLanger
JDK 17 openjdk/jdk17u-dev#982
JDK 20 openjdk/jdk20#76

Karm · 2022-12-23T13:44:59Z

Hi @Karm, I saw your JBS comment. Next time please reason why you consider the risk low. Thanks.

@GoeLin Ack. Will do.

My own thinking is along the lines that it is a well isolated change in a very comprehensible part of code, it aligns with the spec and it does not bring any surprising behavior when compared to some popular WebSocket client libs used with JDK 11, as noted in #1558 (comment). If there is code out there workarounding it, it's likely based on a string substitution and will turn to no-op now.

I will try to be more eloquent about risks of particular backports in the it in future.

RealCLanger · 2022-12-25T06:56:13Z

OK, let me try to trigger the backports for you, see here...

THX @RealCLanger JDK 17 openjdk/jdk17u-dev#982 JDK 20 openjdk/jdk20#76

Hi @Karm, I already did the backports, they are already integrated. You can close your PRs... 😄

Karm · 2022-12-25T23:35:27Z

OK, let me try to trigger the backports for you, see here...

THX @RealCLanger JDK 17 openjdk/jdk17u-dev#982 JDK 20 openjdk/jdk20#76

Hi @Karm, I already did the backports, they are already integrated. You can close your PRs... smile

@RealCLanger, Oh, I see :-D Sorry for the noise.

phohensee · 2022-12-27T20:49:21Z

@Karm, please add /integrate and I'll sponsor.

Karm · 2022-12-27T21:34:12Z

/integrate

Karm · 2022-12-27T21:34:28Z

THX @phohensee : )

openjdk · 2022-12-27T21:35:20Z

@Karm
Your change (at version 7306614) is now ready to be sponsored by a Committer.

phohensee · 2022-12-27T21:53:43Z

/sponsor

openjdk · 2022-12-27T21:55:39Z

Going to push as commit ce10688.
Since your change was applied there have been 40 commits pushed to the master branch:

c46dcc5: 8252715: Problem list java/awt/event/KeyEvent/KeyTyped/CtrlASCII.java on Linux
ca49dbf: 8298527: Cygwin's uname -m returns different string than before
bcda041: 8256240: Reproducible builds should turn on the "deterministic" flag for Visual Studio
ffea913: 8244592: Start supporting SOURCE_DATE_EPOCH
3e65a7c: 8282511: Use fixed certificate validation date in SSLExampleCert template
23aeca9: 8296611: Problemlist several sun/security tests until JDK-8295343 is resolved
a3d0522: 8282398: EndingDotHostname.java test fails because SSL cert expired
c356873: Merge
88954e1: 8298737: 8296772 backport to jdk11u caused build error on sparc
77d919a: 8297804: (tz) Update Timezone Data to 2022g
... and 30 more: https://git.openjdk.org/jdk11u-dev/compare/b5555c191537c27e6b7dcf7ac15c4dfe2967688e...master

Your commit was automatically rebased without conflicts.

openjdk · 2022-12-27T21:55:56Z

@phohensee @Karm Pushed as commit ce10688.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

Backports JDK-8245245: WebSocket can lose the URL encoding

3f2772f

Karm changed the title ~~WebSocket can lose the URL encoding of URI query parameters~~ Backport c07ce7eec71aefbd3cb624e03ca53f5148d01f19 Nov 30, 2022

openjdk bot changed the title ~~Backport c07ce7eec71aefbd3cb624e03ca53f5148d01f19~~ 8245245: WebSocket can lose the URL encoding of URI query parameters Nov 30, 2022

openjdk bot added backport rfr Pull request is ready for review labels Nov 30, 2022

phohensee suggested changes Dec 1, 2022

View reviewed changes

Karm mentioned this pull request Dec 13, 2022

8298588: WebSockets: HandshakeUrlEncodingTest unnecessarily depends on a response body openjdk/jdk#11486

Closed

3 tasks

Karm added 2 commits December 13, 2022 15:43

Makes test independent on JDK-8240666

5feb6ef

Synchronized with jdk/pull/11486

3e5b03e

Karm requested a review from phohensee December 16, 2022 14:34

jerboaa reviewed Dec 16, 2022

View reviewed changes

test/jdk/java/net/httpclient/websocket/HandshakeUrlEncodingTest.java Show resolved Hide resolved

phohensee approved these changes Dec 16, 2022

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Dec 16, 2022

jerboaa reviewed Dec 19, 2022

View reviewed changes

test/jdk/java/net/httpclient/websocket/HandshakeUrlEncodingTest.java Outdated Show resolved Hide resolved

Replaces a comment about JDK-8240666

7306614

openjdk bot removed the sponsor Pull request is ready to be sponsored label Dec 19, 2022

jerboaa approved these changes Dec 21, 2022

View reviewed changes

phohensee approved these changes Dec 21, 2022

View reviewed changes

openjdk bot added the sponsor Pull request is ready to be sponsored label Dec 27, 2022

openjdk bot added the integrated Pull request has been integrated label Dec 27, 2022

openjdk bot closed this Dec 27, 2022

openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review sponsor Pull request is ready to be sponsored labels Dec 27, 2022

8245245: WebSocket can lose the URL encoding of URI query parameters #1558

8245245: WebSocket can lose the URL encoding of URI query parameters #1558

Uh oh!

Conversation

Karm commented Nov 30, 2022 • edited by openjdk bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Unpatched Temurin-11.0.17+8 ❌

Patched jdk11u ✔️

Progress

Issues

Reviewers

Reviewing

Uh oh!

bridgekeeper bot commented Nov 30, 2022

Uh oh!

jerboaa commented Nov 30, 2022

Uh oh!

openjdk bot commented Nov 30, 2022

Uh oh!

mlbridge bot commented Nov 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Webrevs

Uh oh!

phohensee left a comment

Choose a reason for hiding this comment

Uh oh!

Karm commented Dec 2, 2022

Uh oh!

jerboaa commented Dec 2, 2022

Uh oh!

Karm commented Dec 2, 2022

Uh oh!

Karm commented Dec 2, 2022

Uh oh!

jerboaa commented Dec 5, 2022

Uh oh!

jerboaa commented Dec 6, 2022

Uh oh!

dfuch commented Dec 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Karm commented Dec 12, 2022

Uh oh!

dfuch commented Dec 12, 2022

Uh oh!

Karm commented Dec 16, 2022

Uh oh!

jerboaa left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

phohensee left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

openjdk bot commented Dec 16, 2022

Uh oh!

openjdk bot commented Dec 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

phohensee commented Dec 16, 2022

Uh oh!

Uh oh!

RealCLanger commented Dec 19, 2022

Uh oh!

jerboaa commented Dec 20, 2022

Uh oh!

Karm commented Dec 20, 2022

Uh oh!

RealCLanger commented Dec 20, 2022

Uh oh!

Karm commented Dec 21, 2022

Uh oh!

Karm commented Dec 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jerboaa left a comment

Choose a reason for hiding this comment

Uh oh!

phohensee left a comment

Karm commented Nov 30, 2022 •

edited by openjdk bot

Loading

mlbridge bot commented Nov 30, 2022 •

edited

Loading

dfuch commented Dec 6, 2022 •

edited

Loading

phohensee left a comment •

edited

Loading

openjdk bot commented Dec 16, 2022 •

edited

Loading

Karm commented Dec 21, 2022 •

edited

Loading

phohensee commented Dec 21, 2022 •

edited

Loading

Karm commented Dec 23, 2022 •

edited

Loading