HADOOP-18930. Make fs.s3a.create.performance a bucket-wide setting #6168
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
creation tests fail. propose: parameterize |
|
+ITestDirectoryMarkerListing |
|
testing: updated test against s3 london from ide and maven. |
xinglin
reviewed
Oct 11, 2023
| @@ -1183,12 +1183,17 @@ private Constants() { | |||
|
|
|||
| /** | |||
| * Flag for create performance. | |||
| * This is *not* a configuration option; it is for use in the | |||
| * {code createFile()} builder. | |||
| * This can be set in the {code createFile()} builder. | |||
There was a problem hiding this comment.
can we add what "create performance" means or change it to a more meaningful name, such as fs.s3a.create.skip.file.existence.check or fs.s3a.create.overwrite-if-exist, or just as simple as fs.s3a.create.overwrite and then add a comment about what it means?
There was a problem hiding this comment.
i'd rather not because its the existing openFile option. And it gives us options to do more later, if we can think of
There was a problem hiding this comment.
sure, understood. we can keep this option. But can we add a comment about what it does today?
I don't get what it meant by reading "create performance".
There was a problem hiding this comment.
should be covered in the openfile docs
steveloughran
changed the title
xinglin
reviewed
Oct 14, 2023
| @@ -656,6 +657,11 @@ public void initialize(URI name, Configuration originalConf) | |||
| // verify there's no S3Guard in the store config. | |||
| checkNoS3Guard(this.getUri(), getConf()); | |||
|
|
|||
| // performance creation flag for code which wants performance | |||
There was a problem hiding this comment.
can we move comment here to where this property MACRO is defined? easy to find there.
steveloughran
force-pushed
the
s3/HADOOP-18930-create-performance
branch
from
99f879a
to
f234e28
Compare
|
lots of work on the tests, including a fix in production code (create("/") must raise IOE) still ITestS3ADeleteCost failing |
steveloughran
force-pushed
the
s3/HADOOP-18930-create-performance
branch
from
43bed65
to
f670582
Compare
If fs.s3a.create.performance is set on a bucket - All file overwrite checks are skipped, even if the caller says otherwise. - All directory existence checks are skipped. - Marker deletion is *always* skipped. This eliminates a HEAD and a LIST for every creation. * New path capability "fs.s3a.create.performance.enabled" true if the option is enabled. * Parameterize ITestS3AContractCreate to expect the different outcomes * Parameterize ITestCreateFileCost similarly, with changed cost assertions there. * create(/) raises an IOE. existing bug only noticed here. Change-Id: Ie997290254ae6afd5bc1d52789f57a9b68eb536b
steveloughran
force-pushed
the
s3/HADOOP-18930-create-performance
branch
from
f670582
to
7bdf972
Compare
|
🎊 +1 overall
This message was automatically generated. |
mukund-thakur
approved these changes
Oct 26, 2023
ahmarsuhail
pushed a commit
to ahmarsuhail/hadoop
that referenced
this pull request
Dec 4, 2023
…pache#6168) If fs.s3a.create.performance is set on a bucket - All file overwrite checks are skipped, even if the caller says otherwise. - All directory existence checks are skipped. - Marker deletion is *always* skipped. This eliminates a HEAD and a LIST for every creation. * New path capability "fs.s3a.create.performance.enabled" true if the option is enabled. * Parameterize ITestS3AContractCreate to expect the different outcomes * Parameterize ITestCreateFileCost similarly, with changed cost assertions there. * create(/) raises an IOE. existing bug only noticed here. Contributed by Steve Loughran
ahmarsuhail
pushed a commit
to ahmarsuhail/hadoop
that referenced
this pull request
Dec 5, 2023
…pache#6168) If fs.s3a.create.performance is set on a bucket - All file overwrite checks are skipped, even if the caller says otherwise. - All directory existence checks are skipped. - Marker deletion is *always* skipped. This eliminates a HEAD and a LIST for every creation. * New path capability "fs.s3a.create.performance.enabled" true if the option is enabled. * Parameterize ITestS3AContractCreate to expect the different outcomes * Parameterize ITestCreateFileCost similarly, with changed cost assertions there. * create(/) raises an IOE. existing bug only noticed here. Contributed by Steve Loughran
ahmarsuhail
pushed a commit
to ahmarsuhail/hadoop
that referenced
this pull request
Dec 5, 2023
…pache#6168) If fs.s3a.create.performance is set on a bucket - All file overwrite checks are skipped, even if the caller says otherwise. - All directory existence checks are skipped. - Marker deletion is *always* skipped. This eliminates a HEAD and a LIST for every creation. * New path capability "fs.s3a.create.performance.enabled" true if the option is enabled. * Parameterize ITestS3AContractCreate to expect the different outcomes * Parameterize ITestCreateFileCost similarly, with changed cost assertions there. * create(/) raises an IOE. existing bug only noticed here. Contributed by Steve Loughran
jiajunmao
pushed a commit
to jiajunmao/hadoop-MLEC
that referenced
this pull request
Feb 6, 2024
…pache#6168) If fs.s3a.create.performance is set on a bucket - All file overwrite checks are skipped, even if the caller says otherwise. - All directory existence checks are skipped. - Marker deletion is *always* skipped. This eliminates a HEAD and a LIST for every creation. * New path capability "fs.s3a.create.performance.enabled" true if the option is enabled. * Parameterize ITestS3AContractCreate to expect the different outcomes * Parameterize ITestCreateFileCost similarly, with changed cost assertions there. * create(/) raises an IOE. existing bug only noticed here. Contributed by Steve Loughran
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
HADOOP-18930. Make fs.s3a.create.performance a bucket-wide settings.
Adds the switch;
How was this patch tested?
doesn't change the tests which will fail when enabled, or provide and explicit way to set in a maven profile.
likely outcome
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?