%2F if specified in path is getting double encoded as %252F iresspective of setting RestAssured.urlEncodingEnabled=false

[johanhaleby]

From alok.me...@julysystems.com on June 27, 2012 09:20:47

What steps will reproduce the problem? RestAssured.urlEncodingEnabled=false
given().pathParam("pageUrl", "Product%2F1756%2FMilk-Bone-Flavor-Snacks-Biscuits-for-Dogs").
get("/findItem;pageUrl={pageUrl}")

OR

get("/findItem;pageUrl=Product%2F1756%2FMilk-Bone-Flavor-Snacks-Biscuits-for-Dogs;sortby=true;test=abc123")

The server is receiving the pageURl as "Product%252F1756%252FMilk-Bone-Flavor-Snacks-Biscuits-for-Dogs" .
you can clearly see %2F is getting double encoded as %252F.

if you use any of the above methods you can see when the pageUrl parameter passed to the server is double encoded. There is no way to pass a matrixparam to a rest service which contains "/" . if possible add support for matrixparam as well, so that we will be able to test rest services which takes multiple matrixparam as input. What is the expected output? What do you see instead? i should have seen pageUrl=Product%2F1756%2FMilk-Bone-Flavor-Snacks-Biscuits-for-Dogs at the server. Bu i see its coming as Product%252F1756%252FMilk-Bone-Flavor-Snacks-Biscuits-for-Dogs What version of the product are you using? On what operating system? 1.6.2 on Windows 7 Please provide any additional information below.

Original issue: http://code.google.com/p/rest-assured/issues/detail?id=181

[johanhaleby]

From mi.koel...@googlemail.com on June 28, 2012 07:06:46

Was already reported: https://code.google.com/p/rest-assured/issues/detail?id=175 However, that ticket has status "Fixed" although the problem was only partly resolved.

[johanhaleby]

From alok.me...@julysystems.com on June 29, 2012 00:50:33

The issue still persist . Any workarounds ?

[johanhaleby]

From johan.ha...@gmail.com on August 16, 2012 03:57:06

You can try if using

given().urlEncodingEnabled(false).pathParam(...)

works.

[johanhaleby]

From mi.koel...@googlemail.com on September 03, 2012 02:26:17

No, in 1.6.2 that does not help.

[johanhaleby]

From johan.ha...@gmail.com on September 03, 2012 02:28:04

Ok thanks for trying.

Status: Accepted

[johanhaleby]

From kwig...@gmail.com on September 07, 2012 09:13:21

I tried with both 1.6.1 and 1.6.2

When I turn urlEscapeEnabled(false) RA allows me to pass query string values that are already escaped in and the items are NOT double escaped:

given().urlEncodingEnabled(false)
.get("/rest/api/2.0.alpha1/search?foo%20=%20BAM%20AND%20issuetype%20=%20Bug")

GET /rest/api/2.0.alpha1/search?foo%20=%20BAM%20AND%20issuetype%20=%20Bug HTTP/1.1

However if the escaping occurs in the actual path, double escaping occurs

given().urlEncodingEnabled(false)
.get("/rest/api/2.0.alpha1/search/foo%20=%20BAM%20AND%20issuetype%20=%20Bug")

GET /rest/api/2.0.alpha1/search/foo%2520=%2520BAM%2520AND%2520issuetype%2520=%2520Bug HTTP/1.1

I have some fancy URIs that I want to test with, but can't figure out how to NOT get double escaping (thus failing my tests). For instance I have some URI path segments that have the ?

I believe 188 is a subset of this problem. Either I need to escape the full path myself (urlEncodingEnabled(false) working) or some more characters need to be better encoded. Or both. Below is my list:

These get encoded correctly

"dot .", "EURO \u20ac", "PARA \u00a7", "colon :",
"asterisk *", "parens ()", "curly {}", "angle <>", "square []", "percent %", "and &",
"comma ,", "hash #", "plus +", "tilde ~"

and the ones that are not escaped correctly

"qmark ?" // bug 188 "backslash ",
"slash /",
"semicolon ;",
"quotes "'",
"surrogate pair \ud800\udc00"

Any quick workarounds?

[johanhaleby]

From anthony....@gmail.com on September 20, 2012 09:03:01

also got bitten by this issue (1.6.2 also).
hopefully, the resource I was testing was just a GET resource, so I used java.net.URL and hamcrest matchers as a workaround:
URL url = new URL(RestAssured.baseURI + ":" + RestAssured.port + "/api/agents/probeUrl/" + URLEncoder.encode("funky://characters&to&*escape", "UTF-8"));
InputStream openStream = url.openStream();
String result = new java.util.Scanner(openStream).useDelimiter("\A").next();
String expectedContent = "{"agentId":"embedded","available":true";
assertThat(result, containsString(expectedContent));

[johanhaleby]

From johan.ha...@gmail.com on October 11, 2012 00:06:25

Status: Fixed

[johanhaleby]

From 0xffff...@gmail.com on October 25, 2012 02:19:08

Can you describe the work around?

When I try to send this URL with the code shown below I get a 404 because the %2F, is rest assured decoding the %2F somewhere?

http://localhost:8097/something/ceec6bdd-3734-41f3-b13d-ab722a45eee6/QumrSExq2yLcuISsau2%2Fj0BIdwuQATk4wUCCTBHzO5k%3D/integrationTestGame/WIzIgbLkEEgMp9cixKmC23FvqzYOhEVUFnmzpkWC3jo%3D

given().urlEncodingEnabled(false).get(url).asString();

[johanhaleby]

From johan.ha...@gmail.com on November 03, 2012 21:57:45

Does it work if you're not using rest assured?

[johanhaleby]

From John.Pr...@gmail.com on October 30, 2013 06:59:57

Looks like the original reporter dropped out, but I can vouch that we're still seeing this bug in 1.8.1. I ran some tests and checked what the server ended up receiving in each case.

String val = "yes/no";
String encoded = URLEncoder.encode(val, "UTF-8");

// Case 1: val is encoded
given().queryParam("val", val).get(" https://example.com/foo"); // /foo?val=yes%2Fno

// Case 2: val is not encoded
given().urlEncodingEnabled(false).queryParam("val", val).get(" https://example.com/foo"); // /foo?val=yes/no

// Case 3: val is encoded
given().pathParam("val", val).get(" https://example.com/foo?val={val}"); // /foo?val=yes%2Fno

// Case 4: val is not encoded
given().pathParam("val", val).get(" https://example.com/foo/{val}/something"); // /foo/yes/no/something

// Case 5: "encoded" is encoded (so now it's double-encoded)
given().pathParam("encoded", encoded).get(" https://example.com/foo/{encoded}/something"); // /foo/yes%252Fno/something

// Case 6: val is not encoded
given().urlEncodingEnabled(false).pathParam("val", val).get(" https://example.com/foo/{val}/something"); // /foo/yes/no/something

// Case 7: encoded has been un-encoded
given().urlEncodingEnabled(false).pathParam("encoded", encoded).get(" https://example.com/foo/{encoded}/something"); // /foo/yes/no/something

Cases 1-3 work as I would expect.

Case 4 seems inconsistent with case 3, but maybe that's intentional. However, if case 4 is correct, then I would not expect the double-encoding in case 5.

Case 6 seems correct, but then case 7 is baffling; why does the value get un-encoded?

In the end, I seem to be able to send requests that are either double-encoded or not encoded at all, but never just properly encoded.

[johanhaleby]

From johan.ha...@gmail.com on October 30, 2013 07:02:58

Thanks for your thorough investigation. I've broken my hand and have a really hard time to write and code right now so please help out!

Status: Accepted

[johanhaleby]

From gavinhac...@gmail.com on October 31, 2013 10:28:16

I'm also experiencing John's Case 7; my "%2F"s are getting decoded to "/".

[johanhaleby]

From John.Pr...@gmail.com on October 31, 2013 10:48:18

I poked through the code yesterday but couldn't find anything obvious. Unfortunately, I was unable to get Groovy debugging working well enough to really track things down and eventually had to move on to other work. If I get a chance to dig further I'll post any findings, or maybe the test cases will be of use to someone else.

[johanhaleby]

From johan.ha...@gmail.com on November 15, 2013 01:24:06

I don't think it's valid to have a "/" in the "resource path" (e.g. yes/no before the ?) before the query parameters even though you encode it. From my understanding severs decode everything specified after the host (e.g. https:://example/foo). So if "yes/no" would be properly URL encoded by REST Assured it still wouldn't work on the server side.

[johanhaleby]

From johan.ha...@gmail.com on November 15, 2013 01:25:17

What I mean by "wouldn't work on the server side" is that if the server expects a path parameter called "yes/no".

[johanhaleby]

From John.Pr...@gmail.com on November 15, 2013 07:36:35

We're using Jersey on the server side and it does work if things are properly encoded by the client. Eg. given

@path("/foo/{val}/something")
public void doGet(@PathParam("val") String val)

the value of "val" will be "foo/bar" if the slash was encoded as %2F by the client.

I did a quick search and it appears that some servers disable support for this (eg. Apache: http://stackoverflow.com/questions/3235219/urlencoded-forward-slash-is-breaking-url ), but it's not strictly "wrong" as far as I can tell.

That said, in the most recent version of our API we've eliminated the need to ever do this, but we still need to maintain tests for the old version until we can actually remove support for it.

[johanhaleby]

From johan.ha...@gmail.com on November 15, 2013 09:06:27

Thanks for sharing. I'm working on a fix for this, lots of changes though but hopefully I'll be able to get it right sometime next week.

[johanhaleby]

From johan.ha...@gmail.com on November 18, 2013 06:41:19

I've implemented a new snapshot release in which I've tried to fix all URL encoding issues. Please try it out by depending on version 1.8.2-SNAPSHOT after having added the following maven repo:

sonatype https://oss.sonatype.org/content/repositories/snapshots/

It would be great if you can try out asap since I really want to make a new release.

[johanhaleby]

From anthony....@gmail.com on November 18, 2013 09:25:02

hello Johan,
I run several tests with 1.8.2-SNAPSHOT, nothing changed for me : https://gist.github.com/anthonydahanne/7531730

[johanhaleby]

From johan.ha...@gmail.com on November 18, 2013 12:52:00

Thanks a lot for trying it out so soon. There seem to be several (new) issues that you've discovered that my tests doesn't find. I'll try to look into them soon.

[johanhaleby]

From johan.ha...@gmail.com on November 19, 2013 04:45:54

@anthony: I've looked into the code that you provided and these are my findings:

1. I believe this to be correct since the URL is more or less invalid if you disable URL encoding in this case?
2. This is probably correct because there's no way for REST Assured to know how to URL encode the path: "http://localhost/agents/probeUrl/https://localhost:9888 How should RA know that it only should url encode stuff after "http://localhost/agents/probeUrl/ ? If you use named or unnamed path parameters then it'll work, for example given().pathParam("x", "https://localhost:9888").when().get("/agents/probeUrl/{x}"); .
3. I cannot reproduce this. I've checked both the REST Assured logs and tried caught the request with Wireshark and RA is not doing double URL encoding as far as I can tell.
4. Again I've checked with Wireshark and RA does indeed send the request (and does double URL encoding). I've added test cases as well (see the last four tests in URLEncodingITest)

Please let me know if you still think there's something wrong.

[johanhaleby]

From johan.ha...@gmail.com on November 19, 2013 04:57:24

Actually regarding (1) I think that it ought to be possible to specify URL's with "://" in the path (besides the scheme). I'll fix this but it's actually another issue.

[johanhaleby]

From anthony....@gmail.com on November 19, 2013 06:53:59

@johan,
thanks a lot for investigating.

1. yeah this is expected behavior
2. I did not try with .pathParam() ; I just tested it, it works (I updated the gist at https://gist.github.com/anthonydahanne/7531730 3) that is weird, if I have time I'll create a standalone maven project to reproduce the issue
   4 ) ok, that sounds expected anyway.

all in all, I like the pathParam solution; as you said, it makes sense to extract the part of the url that needs encoding into a path param, so that RA knows what needs to be encoded.
thanks a lot for your help, now I can get rid of HttpClient in my tests :-)

[johanhaleby]

From johan.ha...@gmail.com on November 19, 2013 09:59:02

You can also use unnamed path parameters like this: .. .get("/agents/probeUrl/{x}", "https://localhost:9888");

I'll consider this issue fixed now. If anyone find any further issues please let me know asap.

Thanks for your patience and help!

Status: Fixed

[gideonaina]

Another workaround ... See
https://stackoverflow.com/questions/55019246/restassured-disable-url-encoding-not-working-correctly/57610677#57610677

Basically, it involves setting this in your RequestSpecification
new RequestSpecBuilder().setUrlEncodingEnabled(false)