HADOOP-18928: S3AFileSystem URL encodes twice where Path has trailing / #6167
+16
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Change-Id: I4062dc67013511ea0d240b9461aff8063c00ea39
|
🎊 +1 overall
This message was automatically generated. |
ayushtkn
reviewed
Oct 17, 2023
Member
ayushtkn
left a comment
ayushtkn
left a comment
There was a problem hiding this comment.
Patches against object stores require manual testing:
https://cwiki.apache.org/confluence/display/hadoop/how+to+contribute#HowToContribute-SubmittingpatchesagainstobjectstoressuchasAmazonS3,OpenStackSwiftandMicrosoftAzure
|
🎊 +1 overall
This message was automatically generated. |
steveloughran
requested changes
Oct 25, 2023
Contributor
steveloughran
left a comment
steveloughran
left a comment
There was a problem hiding this comment.
we've had problems with spaces in things before (s3a committers) because there's not enough testing/production use of them. So I accept that this is probably an issue.
- can you do what i've suggested about pulling the code from s3afs into a static S3AUtils method, so it can be tested in a (new) unit test.
- and as noted, all hadoop-aws tests require the submitter to run the entire suite against an s3 store -and tell us which endpoint/arguments you used. Yetus can't be given the credentials, see.
| try { | ||
| q = new Path(new URI(urlString.substring(0, urlString.length() - 1))); | ||
| } catch (URISyntaxException e) { | ||
| LOG.error(String.format("Error removing trailing / from path %s", path.toString())); |
Contributor
There was a problem hiding this comment.
- We'd want a LogExactlyOnce so if there is a problem it only ever gets logged once, not every time a path was loaded.
- it should be at warn, not error
- +use slf4j {} expansions.
thanks
| if (urlString.endsWith(Path.SEPARATOR)) { | ||
| // this is a path which needs root stripping off to avoid | ||
| // confusion, See HADOOP-15430 | ||
| LOG.debug("Stripping trailing '/' from {}", q); |
Contributor
There was a problem hiding this comment.
how about pulling out everything in this method after the super.makeQualified(path); call into a static method in S3AUtils? that way it could be tested in a unit test rather than waiting until someone runs the integration tests. faster and with yetus coverage
Contributor
|
We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. |
Contributor
|
@ahmarsuhail @mukund-thakur @shameersss1 what do people think here? |
Contributor
|
As per the description the issue happens with S3Guard #1646 (comment) which is no longer there |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As per the comment at #1646 (comment) ...
due to HADOOP-15430 a Path
s3://my.bucket/my folder/becomes
s3://my.bucket/my%20folderwhich upon HTTP request creation becomes
"GET ...&prefix=my%2520folder which is a wrong path.This PR re-adds the missing step of converting the modified String back to URI before creating a Path out of it.
cc: @steveloughran @bgaborg @whenamanlies