HADOOP-18830. Cut S3 Select #6144
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Given I wrote this it's kind of sad, but openFile() will outlive it as it has many more uses. |
|
🎊 +1 overall
This message was automatically generated. |
|
Tested s3 london |
|
with this PR
Retained
I have a concern about retention #2; it uses the unshaded package we may want to cut it -we can always add a ref in the package docs to say its' been cut and which git commit to go to so as to find it. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
ahmarsuhail
reviewed
Oct 17, 2023
| @@ -35,7 +35,6 @@ full details. | |||
|
|
|||
| * [Encryption](./encryption.html) | |||
| * [Performance](./performance.html) | |||
| * [S3Guard](./s3guard.html) | |||
There was a problem hiding this comment.
why are we removing this? think the S3Guard page is still there
There was a problem hiding this comment.
think i moved it
ahmarsuhail
reviewed
Oct 17, 2023
There was a problem hiding this comment.
Sad to see this go too..it complicated the SDK V2 work and we had to add in a lot of code to get this work.
Looks good, few things:
- Remove the CsvFile class?
- In S3Utils we have SC_405_METHOD_NOT_ALLOWED, comment says "seen on S3 Select" . maybe we don't need that anymore
- Couple of open todos in
ITestSelectUnsupported
|
And agree with your comment, should cut |
|
keeping but moving csv file. it's actually very useful in places; in cloudstore we use it in places like "bandwidth" to track block upload times. so it may be useful in future...leaving it in test/ means there's no commitment to maintain |
steveloughran
force-pushed
the
s3/HADOOP-18830-cut-s3-select
branch
from
616e55a
to
a4346dd
Compare
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
steveloughran
force-pushed
the
s3/HADOOP-18830-cut-s3-select
branch
from
a4346dd
to
88f202d
Compare
|
🎊 +1 overall
This message was automatically generated. |
steveloughran
force-pushed
the
s3/HADOOP-18830-cut-s3-select
branch
2 times, most recently
from
8029a13
to
eadbd95
Compare
|
latest change is just a rebase; not retested |
|
💔 -1 overall
This message was automatically generated. |
steveloughran
force-pushed
the
s3/HADOOP-18830-cut-s3-select
branch
from
eadbd95
to
a254800
Compare
|
rebased pr with retest. failures unrelated; the signing one has an active pr to fix, the committer one looks like my config is at fault (bucket overrides not being cut) |
|
🎊 +1 overall
This message was automatically generated. |
|
@ahmarsuhail @mukund-thakur can I get this in? it'll help on test time... |
ahmarsuhail
approved these changes
Jan 25, 2024
| @@ -17,7 +17,12 @@ | |||
| */ | |||
|
|
|||
| /** | |||
| * Support for S3 Select. | |||
| * Was the location for support for S3 Select. | |||
| * Now removed apart from some constants.f | |||
| " WHERE s._1 = 'foo'"; | ||
|
|
||
| /** | ||
| * A {@code .must(SELECT_SQL, _)} option MSUT not fail the build. |
There was a problem hiding this comment.
nit: typo in MUST
and this isn't clear to me, .must should fail and throw Unsupported Exception right?
There was a problem hiding this comment.
correct. updated this and other javadocs, named methods appropriately
|
💔 -1 overall
This message was automatically generated. |
|
thanks. not sure what is up with docker. will rebase and force push with test run |
Cut out S3 Select * leave public/unstable constants alone * s3guard tool will fail with error * s3afs. path capability will fail * openFile() will fail with specific error * s3 select doc updated * Cut eventstream jar * New test: ITestSelectUnsupported verifies new failure handling above Everything else cut, including tests. Change-Id: Iff8c1e6562ed2d47abefe2cfef2909c00c78a55d
+update doc with explicit consequences of removal, including the "hadoop s3guard select" command Change-Id: Ica3ffb33d803f8a03e0d733447e4f026d547f8c1
steveloughran
force-pushed
the
s3/HADOOP-18830-cut-s3-select
branch
from
1ff7c34
to
736a3cc
Compare
|
🎊 +1 overall
This message was automatically generated. |
Review in docs/core-siten references to landsat and CSV; update as appropriate. This includes * A review of use of .endpoint vs .endpoint.region and a move to fs.s3a.endpoint.region as much as possible. * reorg of some sections * fixup of a name= headers. Change-Id: I5e07cdf153cacf0ce1ee6673d3a094c2d5eaf5a2
|
💔 -1 overall
This message was automatically generated. |
asfgit
pushed a commit
that referenced
this pull request
Jan 30, 2024
Cut out S3 Select * leave public/unstable constants alone * s3guard tool will fail with error * s3afs. path capability will fail * openFile() will fail with specific error * s3 select doc updated * Cut eventstream jar * New test: ITestSelectUnsupported verifies new failure handling above Contributed by Steve Loughran
4 tasks
steveloughran
added a commit
to steveloughran/hadoop
that referenced
this pull request
Feb 2, 2024
Cut out S3 Select * leave public/unstable constants alone * s3guard tool will fail with error * s3afs. path capability will fail * openFile() will fail with specific error * s3 select doc updated * New test: ITestSelectUnsupported verifies new failure handling above This is the v1 SDK version of the feature. Contributed by Steve Loughran Change-Id: Ic18c49562e5143a2a2204d66840be149db486b02
steveloughran
added a commit
to steveloughran/hadoop
that referenced
this pull request
Feb 2, 2024
Cut out S3 Select * leave public/unstable constants alone * s3guard tool will fail with error * s3afs. path capability will fail * openFile() will fail with specific error * s3 select doc updated * New test: ITestSelectUnsupported verifies new failure handling above This is the v1 SDK version of the feature. Contributed by Steve Loughran Change-Id: Ic18c49562e5143a2a2204d66840be149db486b02
jiajunmao
pushed a commit
to jiajunmao/hadoop-MLEC
that referenced
this pull request
Feb 6, 2024
Cut out S3 Select * leave public/unstable constants alone * s3guard tool will fail with error * s3afs. path capability will fail * openFile() will fail with specific error * s3 select doc updated * Cut eventstream jar * New test: ITestSelectUnsupported verifies new failure handling above Contributed by Steve Loughran
steveloughran
added a commit
that referenced
this pull request
Feb 8, 2024
Cut out S3 Select * leave public/unstable constants alone * s3guard tool will fail with error * s3afs. path capability will fail * openFile() will fail with specific error * s3 select doc updated * New test: ITestSelectUnsupported verifies new failure handling above This is the v1 SDK version of the change. Contributed by Steve Loughran
slfan1989
pushed a commit
that referenced
this pull request
Feb 10, 2024
Cut out S3 Select * leave public/unstable constants alone * s3guard tool will fail with error * s3afs. path capability will fail * openFile() will fail with specific error * s3 select doc updated * Cut eventstream jar * New test: ITestSelectUnsupported verifies new failure handling above Contributed by Steve Loughran
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cut out S3 Select
Everything else cut, including tests.
New test: ITestSelectUnsupported
verifies new failure handling above
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?