Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

8245245: WebSocket can lose the URL encoding of URI query parameters #1558

Closed
wants to merge 5 commits into from

Conversation

Karm
Copy link
Contributor

@Karm Karm commented Nov 30, 2022

Proposes to backport JDK-8245245.

The backport is clean as far as the actual OpeningHandshake.java goes. The test needed a little tweak so as to compile with SimpleSSLContext and also to handle the fact that the erroneous response does not bring a response body.

The test passes with the patch, fails without it.

$ make clean run-test TEST="jtreg:test/jdk/java/net/httpclient/websocket/HandshakeUrlEncodingTest.java"
...
==============================
Test summary
==============================
   TEST                                              TOTAL  PASS  FAIL ERROR   
   jtreg:test/jdk/java/net/httpclient/websocket/HandshakeUrlEncodingTest.java
                                                         1     1     0     0   
==============================
TEST SUCCESS

Stopping sjavac server
Finished building targets 'clean run-test' in configuration 'linux-x86_64-normal-server-release'

In addition to that, I compiled and executed the original WebSocketTest.java reproducer found on JDK-8245245 JIRA.

Unpatched Temurin-11.0.17+8 ❌

$ java WebSocketTest 
Http Request
http://localhost:8000/?raw=abc+def/ghi=xyz&encoded=abc%2Bdef%2Fghi%3Dxyz
Server RequestURI: /?raw=abc+def/ghi=xyz&encoded=abc%2Bdef%2Fghi%3Dxyz
WebSocket Request
ws://localhost:8000/?&raw=abc+def/ghi=xyz&encoded=abc%2Bdef%2Fghi%3Dxyz
Server RequestURI: /?&raw=abc+def/ghi=xyz&encoded=abc+def/ghi=xyz

Patched jdk11u ✔️

$ java WebSocketTest 
Http Request
http://localhost:8000/?raw=abc+def/ghi=xyz&encoded=abc%2Bdef%2Fghi%3Dxyz
Server RequestURI: /?raw=abc+def/ghi=xyz&encoded=abc%2Bdef%2Fghi%3Dxyz
WebSocket Request
ws://localhost:8000/?&raw=abc+def/ghi=xyz&encoded=abc%2Bdef%2Fghi%3Dxyz
Server RequestURI: /?&raw=abc+def/ghi=xyz&encoded=abc%2Bdef%2Fghi%3Dxyz

The patched version correctly leaves the latter part of the query param encoded.


Progress

  • Change must be properly reviewed (1 review required, with at least 1 Reviewer)
  • Change must not contain extraneous whitespace
  • Commit message must refer to an issue

Issues

  • JDK-8245245: WebSocket can lose the URL encoding of URI query parameters
  • JDK-8298588: WebSockets: HandshakeUrlEncodingTest unnecessarily depends on a response body

Reviewers

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk11u-dev pull/1558/head:pull/1558
$ git checkout pull/1558

Update a local copy of the PR:
$ git checkout pull/1558
$ git pull https://git.openjdk.org/jdk11u-dev pull/1558/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 1558

View PR using the GUI difftool:
$ git pr show -t 1558

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk11u-dev/pull/1558.diff

@bridgekeeper
Copy link

bridgekeeper bot commented Nov 30, 2022

👋 Welcome back Karm! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

@jerboaa
Copy link
Contributor

jerboaa commented Nov 30, 2022

@Karm Please change the PR title to Backport c07ce7eec71aefbd3cb624e03ca53f5148d01f19 so that the bots recognize this as a backport. Please be sure to run :jdk_net test group before/after as well.

@Karm Karm changed the title WebSocket can lose the URL encoding of URI query parameters Backport c07ce7eec71aefbd3cb624e03ca53f5148d01f19 Nov 30, 2022
@openjdk openjdk bot changed the title Backport c07ce7eec71aefbd3cb624e03ca53f5148d01f19 8245245: WebSocket can lose the URL encoding of URI query parameters Nov 30, 2022
@openjdk
Copy link

openjdk bot commented Nov 30, 2022

This backport pull request has now been updated with issue and summary from the original commit.

@openjdk openjdk bot added backport rfr Pull request is ready for review labels Nov 30, 2022
@mlbridge
Copy link

mlbridge bot commented Nov 30, 2022

Webrevs

Copy link
Member

@phohensee phohensee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good except for using getResponse().uri() and a few formatting issues in HandshakeUrlEncodingTest.java.

The change to use getResponse().uri() instead of getResponse().body() fixes a bug in tip, so can't be part of a backport. The thing to do is use the buggy code from the original commit in this backport, fix the problem in tip, then backport that fix to 19u, 17u, and 11u.

On formatting issues, future backports are easier if you match the original formatting as closely as possible. Looks like you're using and IDE that auto-formats.

At line 59:

import static java.lang.System.out;
import static java.net.http.HttpClient.Builder.NO_PROXY;
import static org.testng.Assert.*;

should be

import static java.net.http.HttpClient.Builder.NO_PROXY;
import static org.testng.Assert.*;
import static java.lang.System.out;

to more closely follow the original. In fact, if all the test uses is assertEquals, assertNotNull, and fail, just use the original commit code.

Indent/format the same as the original commit: lines 85-88, 89-92, 106-108, 111, 115, 133-135, 137, 141-146, 153-158, 174-176, 188-190, and missing blank line after 196. Might have missed somthing, but you get the idea.

@Karm
Copy link
Contributor Author

Karm commented Dec 2, 2022

@phohensee Thank you for the review.

Ad formatting: Ack. Will do.

Ad

The change to use getResponse().uri() instead of getResponse().body() fixes a bug in tip, so can't be part of a backport.

Oh...I see. So the fact that the body is returned even though there was a failure is a bug, not a feature/desirable chage? I will browse the Jira and check the spec to make sure I am not bending the test to a faulty implementation then.

Cheers
Karm

@jerboaa
Copy link
Contributor

jerboaa commented Dec 2, 2022

Oh...I see. So the fact that the body is returned even though there was a failure is a bug, not a feature/desirable chage? I will browse the Jira and check the spec to make sure I am not bending the test to a faulty implementation then.

Even if you discover a discrepancy, this would need to get fixed in JDK head first and then backported. Please keep that in mind.

@Karm
Copy link
Contributor Author

Karm commented Dec 2, 2022

Helo, @phohensee,

I changed the behavior of the test in the tip (HEAD), so as I can then backport it along the JDK-8245245 patch to 19u, 17u, and 11u.

openjdk/jdk#11486

@jerboaa If I am reading the spec right, there is IMHO no need to change the behavior of the implementation.

@Karm
Copy link
Contributor Author

Karm commented Dec 2, 2022

@jerboaa I realized the patch is already backported to JDK 19 and JDK 17. So it's just JDK 11 I'd like to backport JDK-8245245 to. And IIUC, the change in openjdk/jdk#11486 needed for JDK 11 will need backporting to JDK 19, JDK 17 too.

@jerboaa
Copy link
Contributor

jerboaa commented Dec 5, 2022

@jerboaa I realized the patch is already backported to JDK 19 and JDK 17. So it's just JDK 11 I'd like to backport JDK-8245245 to. And IIUC, the change in openjdk/jdk#11486 needed for JDK 11 will need backporting to JDK 19, JDK 17 too.

Yes.

@jerboaa
Copy link
Contributor

jerboaa commented Dec 6, 2022

FWIW, since https://bugs.openjdk.org/browse/JDK-8240666 is not in 11u, it would be acceptable to alter the test to account for this. A comment in the test could explain that. I don't think backporting JDK-8240666 would be appropriate (behaviour change) and it looks like getting the test change (account for the optional body) in JDK head might not be wanted.

@dfuch
Copy link
Member

dfuch commented Dec 6, 2022

It is important that the test checks the URI received by the server, and that's probably why the body was used here. The server writes the URI it receives in the response body. This provides an end-to-end check that what was received is what we expected to send. Note that the server doesn't actually supports WebSocket and that's why it always replies with 400.

@Karm
Copy link
Contributor Author

Karm commented Dec 12, 2022

Hello @dfuch, @jerboaa, @phohensee, @jaikiran

I'd like this tets change to go forward: openjdk/jdk#11486
It correctly fails without JDK-8245245 and passes with it. It is not necessary to be checking the body.

When that test change is integrated, I'd initiate its backport. Close this pr (#1558) and open a new one, that would just cleanly add JDK-8245245...

I hope that is all right process wise.

@dfuch
Copy link
Member

dfuch commented Dec 12, 2022

It correctly fails without JDK-8245245 and passes with it. It is not necessary to be checking the body.

Thanks for double checking. In that case I believed you're covered, I personally have no objection in that case. I mostly commented because I don't think there's a bug in the original test. It just uses a different method to check for success. As for approval I'll defer to the JDK 11 maintainers.

@Karm
Copy link
Contributor Author

Karm commented Dec 16, 2022

Hello @dfuch, @phohensee,

With jdk/pull/11486 merged, I would kindly ask for a review of this backport to JDK11u.

  • the patch in OpeningHandshake.java did not require any changes
  • the diff for the test between what is in mainline and what I propose here is this: https://editor.mergely.com/Hh1c3Z1N/

thx
K.

@Karm Karm requested a review from phohensee December 16, 2022 14:34
Copy link
Contributor

@jerboaa jerboaa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use /issue add JDK-8298588 as the test now contains those changes too.

Copy link
Member

@phohensee phohensee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm with Severin's recommendations.

@openjdk
Copy link

openjdk bot commented Dec 16, 2022

⚠️ @Karm the full name on your profile does not match the author name in this pull requests' HEAD commit. If this pull request gets integrated then the author name from this pull requests' HEAD commit will be used for the resulting commit. If you wish to push a new commit with a different author name, then please run the following commands in a local repository of your personal fork:

$ git checkout backport-JDK-8245245
$ git commit --author='Preferred Full Name <you@example.com>' --allow-empty -m 'Update full name'
$ git push

@openjdk
Copy link

openjdk bot commented Dec 16, 2022

@Karm This change now passes all automated pre-integration checks.

After integration, the commit message for the final commit will be:

8245245: WebSocket can lose the URL encoding of URI query parameters
8298588: WebSockets: HandshakeUrlEncodingTest unnecessarily depends on a response body

The fix updates jdk.internal.net.http.websocket.OpeningHandshake to avoid double encoding and decoding of URL

Reviewed-by: phh, sgehwolf

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 40 new commits pushed to the master branch:

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

As you do not have Committer status in this project an existing Committer must agree to sponsor your change. Possible candidates are the reviewers of this PR (@phohensee, @jerboaa) but any other Committer may sponsor as well.

➡️ To flag this PR as ready for integration with the above commit message, type /integrate in a new comment. (Afterwards, your sponsor types /sponsor in a new comment to perform the integration).

@openjdk openjdk bot added the ready Pull request is ready to be integrated label Dec 16, 2022
@phohensee
Copy link
Member

/issue add JDK-8298588

@openjdk openjdk bot removed the sponsor Pull request is ready to be sponsored label Dec 19, 2022
@RealCLanger
Copy link
Contributor

@Karm Thanks for contributing this backport to JDK 11 Updates. This looks good to me. As @jerboaa mentioned, the next step would be to add the fix request label and information to the JBS bug.

I'd have one further request: Can you please backport your fix for JDK-8298588 to JDK 20 and JDK17 that we don't miss it out there. For JDK20, you can use the "/backport jdk20" comment on the commit in head. If the backport is clean, which I assume, you can right away integrate as it complies to the RDP1 rules (testfix). For jdk17u-dev, you'll need a PR and maintainer approval by labeling jdk17u-fix-request

@jerboaa
Copy link
Contributor

jerboaa commented Dec 20, 2022

For JDK20, you can use the "/backport jdk20" comment on the commit in head.

I think that's only available for committers. Just FYI.

@Karm
Copy link
Contributor Author

Karm commented Dec 20, 2022

For JDK20, you can use the "/backport jdk20" comment on the commit in head.

I think that's only available for committers. Just FYI.

It is. I tried that the other day: openjdk/jdk@c07ce7e#commitcomment-91811154

@RealCLanger
Copy link
Contributor

OK, let me try to trigger the backports for you, see here...

@Karm
Copy link
Contributor Author

Karm commented Dec 21, 2022

Thanks, lemme process it today.

@Karm
Copy link
Contributor Author

Karm commented Dec 21, 2022

Hello, @jerboaa, This is the JBS text proposal, formatted for JIRA:


I would like [JDK-8245245|https://bugs.openjdk.org/browse/JDK-8245245] to get backported to JDK 11
so as more libraries can start using JDK's own WebSocket client instead of depending
on other implementations.

For instance, [Fabric8 Kubernetes client|https://github.com/fabric8io/kubernetes-client/blob/master/httpclient-jdk/README.md#jdk-client-for-fabric8]
has this very issue with JDK's WebSocket client and it uses OkHttp3 or Vert.x implementations instead.

I used these [JBang|https://www.jbang.dev/] scripts to briefly showcase that both Vert.x and OkHttp3 implementations
are fine running on JDK 11 and JDK 11's WebSocket client needs fixing: (It uses Undertow as the server)

[JDKClient.java|https://gist.github.com/Karm/58959f7bc1d3ef675eecd1e12e56094c]
[OKHttp3Client.java|https://gist.github.com/Karm/6ed845a22a8b8331f95292bb992ee7e0]
[VertXClient.java|https://gist.github.com/Karm/a04a778f352e8d29667eb38a219d4e4b]

{code}
$ java --version
openjdk 11.0.17 2022-10-18

$ ./JDKClient.java
[jbang] Building jar...
The query string was: &raw=abc+def/ghi=xyz&encoded=abc+def/ghi=xyz

$ ./OKHttp3Client.java
[jbang] Building jar...
The query string was: raw=abc+def/ghi=xyz&encoded=abc%2Bdef%2Fghi%3Dxyz

$ ./VertXClient.java
[jbang] Building jar...
The query string was: raw=abc+def/ghi=xyz&encoded=abc%2Bdef%2Fghi%3Dxyz
{code}

Patched, see the JDK client fixed:
{code}
$ java --version
openjdk 11.0.255-internal 2023-01-17

$ ./JDKClient.java 
The query string was: &raw=abc+def/ghi=xyz&encoded=abc%2Bdef%2Fghi%3Dxyz

$ ./OKHttp3Client.java 
The query string was: raw=abc+def/ghi=xyz&encoded=abc%2Bdef%2Fghi%3Dxyz

$ ./VertXClient.java 
The query string was: raw=abc+def/ghi=xyz&encoded=abc%2Bdef%2Fghi%3Dxyz
{code}

Thanks
K.

Edit, short version:

Fix Request
Approve backporting JDK-8245245 to 11u. It fixes a WebSocket client related query param encoding issue and should be a low risk one.
Testing: ran jdk_net tests without regressions. Patch doesn't apply clean as the test needed changing.
Reviewed by Paul Hohensee <phh>, Daniel Fuchs <dfuchs>, Severin Gehwolf <sgehwolf>.

Copy link
Contributor

@jerboaa jerboaa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems OK to me.

Copy link
Member

@phohensee phohensee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm.

@phohensee
Copy link
Member

phohensee commented Dec 21, 2022

Tagged JDK-8298588. Commit pending jdk11u-fix-yes on JDk-8245245 and JDK-8298588.

@GoeLin
Copy link
Member

GoeLin commented Dec 22, 2022

Hi @Karm,
I saw your JBS comment. Next time please reason why you consider the risk low.
Thanks.

@Karm
Copy link
Contributor Author

Karm commented Dec 23, 2022

OK, let me try to trigger the backports for you, see here...

THX @RealCLanger
JDK 17 openjdk/jdk17u-dev#982
JDK 20 openjdk/jdk20#76

@Karm
Copy link
Contributor Author

Karm commented Dec 23, 2022

Hi @Karm, I saw your JBS comment. Next time please reason why you consider the risk low. Thanks.

@GoeLin Ack. Will do.

My own thinking is along the lines that it is a well isolated change in a very comprehensible part of code, it aligns with the spec and it does not bring any surprising behavior when compared to some popular WebSocket client libs used with JDK 11, as noted in #1558 (comment). If there is code out there workarounding it, it's likely based on a string substitution and will turn to no-op now.

I will try to be more eloquent about risks of particular backports in the it in future.

@RealCLanger
Copy link
Contributor

OK, let me try to trigger the backports for you, see here...

THX @RealCLanger JDK 17 openjdk/jdk17u-dev#982 JDK 20 openjdk/jdk20#76

Hi @Karm, I already did the backports, they are already integrated. You can close your PRs... 😄

@Karm
Copy link
Contributor Author

Karm commented Dec 25, 2022

OK, let me try to trigger the backports for you, see here...

THX @RealCLanger JDK 17 openjdk/jdk17u-dev#982 JDK 20 openjdk/jdk20#76

Hi @Karm, I already did the backports, they are already integrated. You can close your PRs... smile

@RealCLanger, Oh, I see :-D Sorry for the noise.

@phohensee
Copy link
Member

@Karm, please add /integrate and I'll sponsor.

@Karm
Copy link
Contributor Author

Karm commented Dec 27, 2022

/integrate

@Karm
Copy link
Contributor Author

Karm commented Dec 27, 2022

THX @phohensee : )

@openjdk openjdk bot added the sponsor Pull request is ready to be sponsored label Dec 27, 2022
@openjdk
Copy link

openjdk bot commented Dec 27, 2022

@Karm
Your change (at version 7306614) is now ready to be sponsored by a Committer.

@phohensee
Copy link
Member

/sponsor

@openjdk
Copy link

openjdk bot commented Dec 27, 2022

Going to push as commit ce10688.
Since your change was applied there have been 40 commits pushed to the master branch:

Your commit was automatically rebased without conflicts.

@openjdk openjdk bot added the integrated Pull request has been integrated label Dec 27, 2022
@openjdk openjdk bot closed this Dec 27, 2022
@openjdk openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review sponsor Pull request is ready to be sponsored labels Dec 27, 2022
@openjdk
Copy link

openjdk bot commented Dec 27, 2022

@phohensee @Karm Pushed as commit ce10688.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport integrated Pull request has been integrated
6 participants