HADOOP-18925. S3A: option "fs.s3a.optimized.copy.from.local.enabled" to control CopyFromLocalOperation #6163
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
🎊 +1 overall
This message was automatically generated. |
mehakmeet
reviewed
Oct 11, 2023
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
Outdated
Show resolved
Hide resolved
| @@ -4086,7 +4103,8 @@ public void copyLocalFileFromTo(File file, Path from, Path to) throws IOExceptio | |||
| Progressable progress = null; | |||
| PutObjectRequest.Builder putObjectRequestBuilder = | |||
| newPutObjectRequestBuilder(key, file.length(), false); | |||
| S3AFileSystem.this.invoker.retry("putObject(" + "" + ")", to.toString(), true, | |||
| final String d = to.toString(); | |||
There was a problem hiding this comment.
nit: Can have a better variable name
There was a problem hiding this comment.
done, reviewed that code and fixed the span logic for a single span across the entire process.
steveloughran
changed the title
|
@mehakmeet addressed your issues, option is now fs.s3a.optimized.copy.from.local.enabled
|
|
💔 -1 overall
This message was automatically generated. |
|
There are conflicts in the PR |
steveloughran
force-pushed
the
s3/HADOOP-18925-disable-copy-from-local
branch
from
f4dcedf
to
6e7c975
Compare
|
💔 -1 overall
This message was automatically generated. |
steveloughran
force-pushed
the
s3/HADOOP-18925-disable-copy-from-local
branch
from
6e7c975
to
3084bc1
Compare
jojochuang
reviewed
Oct 19, 2023
|
💔 -1 overall
This message was automatically generated. |
|
test failure is unrelated and will go away on a rebase |
|
🎊 +1 overall
This message was automatically generated. |
Add a new option fs.s3a.optimized.copy.from.local.enabled This will enable (default) or disable the optimized CopyFromLocalOperation upload operation when copyFromLocalFile() is invoked. When false the superclass implementation is used; duration statistics are still collected, though audit span entries in logs will be for the individual fs operations, not the overall operation. ITestS3ACopyFromLocalFile parameterized to run with/without option. This is for debugging. Change-Id: I208b8babd9c9951acbd603d811b4f5f94620892d
steveloughran
force-pushed
the
s3/HADOOP-18925-disable-copy-from-local
branch
from
3084bc1
to
757207b
Compare
|
rebased and testing with ITestS3ACopyFromLocalFile against s3 london; all good |
|
💔 -1 overall
This message was automatically generated. |
|
@ahmarsuhail could i get a quick look at this? thanks |
|
thanks. I'll backport it as needed. |
steveloughran
added a commit
to steveloughran/hadoop
that referenced
this pull request
Nov 6, 2023
…pache#6163) Add a new option: fs.s3a.optimized.copy.from.local.enabled This will enable (default) or disable the optimized CopyFromLocalOperation upload operation when copyFromLocalFile() is invoked. When false the superclass implementation is used; duration statistics are still collected, though audit span entries in logs will be for the individual fs operations, not the overall operation. Contributed by Steve Loughran Change-Id: I4debb420ea238d0b41ef60b78425a62e963232fe
steveloughran
added a commit
to steveloughran/hadoop
that referenced
this pull request
Nov 6, 2023
…pache#6163) Add a new option: fs.s3a.optimized.copy.from.local.enabled This will enable (default) or disable the optimized CopyFromLocalOperation upload operation when copyFromLocalFile() is invoked. When false the superclass implementation is used; duration statistics are still collected, though audit span entries in logs will be for the individual fs operations, not the overall operation. Contributed by Steve Loughran Change-Id: I4debb420ea238d0b41ef60b78425a62e963232fe
4 tasks
steveloughran
added a commit
to steveloughran/hadoop
that referenced
this pull request
Nov 13, 2023
…pache#6163) Add a new option: fs.s3a.optimized.copy.from.local.enabled This will enable (default) or disable the optimized CopyFromLocalOperation upload operation when copyFromLocalFile() is invoked. When false the superclass implementation is used; duration statistics are still collected, though audit span entries in logs will be for the individual fs operations, not the overall operation. Contributed by Steve Loughran Change-Id: I4debb420ea238d0b41ef60b78425a62e963232fe
ahmarsuhail
pushed a commit
to ahmarsuhail/hadoop
that referenced
this pull request
Dec 4, 2023
…pache#6163) Add a new option: fs.s3a.optimized.copy.from.local.enabled This will enable (default) or disable the optimized CopyFromLocalOperation upload operation when copyFromLocalFile() is invoked. When false the superclass implementation is used; duration statistics are still collected, though audit span entries in logs will be for the individual fs operations, not the overall operation. Contributed by Steve Loughran
ahmarsuhail
pushed a commit
to ahmarsuhail/hadoop
that referenced
this pull request
Dec 5, 2023
…pache#6163) Add a new option: fs.s3a.optimized.copy.from.local.enabled This will enable (default) or disable the optimized CopyFromLocalOperation upload operation when copyFromLocalFile() is invoked. When false the superclass implementation is used; duration statistics are still collected, though audit span entries in logs will be for the individual fs operations, not the overall operation. Contributed by Steve Loughran
ahmarsuhail
pushed a commit
to ahmarsuhail/hadoop
that referenced
this pull request
Dec 5, 2023
…pache#6163) Add a new option: fs.s3a.optimized.copy.from.local.enabled This will enable (default) or disable the optimized CopyFromLocalOperation upload operation when copyFromLocalFile() is invoked. When false the superclass implementation is used; duration statistics are still collected, though audit span entries in logs will be for the individual fs operations, not the overall operation. Contributed by Steve Loughran
steveloughran
added a commit
that referenced
this pull request
Dec 7, 2023
…6163) (#6259) Add a new option fs.s3a.optimized.copy.from.local.enabled This will enable (default) or disable the optimized CopyFromLocalOperation upload operation when copyFromLocalFile() is invoked. When false the superclass implementation is used; duration statistics are still collected, though audit span entries in logs will be for the individual fs operations, not the overall operation. Contributed by Steve Loughran
jiajunmao
pushed a commit
to jiajunmao/hadoop-MLEC
that referenced
this pull request
Feb 6, 2024
…pache#6163) Add a new option: fs.s3a.optimized.copy.from.local.enabled This will enable (default) or disable the optimized CopyFromLocalOperation upload operation when copyFromLocalFile() is invoked. When false the superclass implementation is used; duration statistics are still collected, though audit span entries in logs will be for the individual fs operations, not the overall operation. Contributed by Steve Loughran
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds a new option to enable or disable the
optimized CopyFromLocalOperation upload operation
when copyFromLocalFile() is invoked.
When false the superclass implementation is used; duration statistics are still collected, though audit span entries in logs will be for the individual fs operations, not the overall operation.
How was this patch tested?
-Dparallel-tests -DtestsThreadCount=10 -Dscale -DprefetchFor code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?