-
Notifications
You must be signed in to change notification settings - Fork 8.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HADOOP-18930. Make fs.s3a.create.performance a bucket-wide setting #6168
HADOOP-18930. Make fs.s3a.create.performance a bucket-wide setting #6168
Conversation
creation tests fail. propose: parameterize
|
+ITestDirectoryMarkerListing |
testing: updated test against s3 london from ide and maven. |
@@ -1183,12 +1183,17 @@ private Constants() { | |||
|
|||
/** | |||
* Flag for create performance. | |||
* This is *not* a configuration option; it is for use in the | |||
* {code createFile()} builder. | |||
* This can be set in the {code createFile()} builder. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we add what "create performance" means or change it to a more meaningful name, such as fs.s3a.create.skip.file.existence.check
or fs.s3a.create.overwrite-if-exist
, or just as simple as fs.s3a.create.overwrite
and then add a comment about what it means?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i'd rather not because its the existing openFile option. And it gives us options to do more later, if we can think of
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sure, understood. we can keep this option. But can we add a comment about what it does today?
I don't get what it meant by reading "create performance".
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should be covered in the openfile docs
@@ -656,6 +657,11 @@ public void initialize(URI name, Configuration originalConf) | |||
// verify there's no S3Guard in the store config. | |||
checkNoS3Guard(this.getUri(), getConf()); | |||
|
|||
// performance creation flag for code which wants performance |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we move comment here to where this property MACRO is defined? easy to find there.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
change lgtm.
99f879a
to
f234e28
Compare
lots of work on the tests, including a fix in production code (create("/") must raise IOE) still ITestS3ADeleteCost failing |
43bed65
to
f670582
Compare
If fs.s3a.create.performance is set on a bucket - All file overwrite checks are skipped, even if the caller says otherwise. - All directory existence checks are skipped. - Marker deletion is *always* skipped. This eliminates a HEAD and a LIST for every creation. * New path capability "fs.s3a.create.performance.enabled" true if the option is enabled. * Parameterize ITestS3AContractCreate to expect the different outcomes * Parameterize ITestCreateFileCost similarly, with changed cost assertions there. * create(/) raises an IOE. existing bug only noticed here. Change-Id: Ie997290254ae6afd5bc1d52789f57a9b68eb536b
f670582
to
7bdf972
Compare
🎊 +1 overall
This message was automatically generated. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM +1
Looks good to me overall.
…pache#6168) If fs.s3a.create.performance is set on a bucket - All file overwrite checks are skipped, even if the caller says otherwise. - All directory existence checks are skipped. - Marker deletion is *always* skipped. This eliminates a HEAD and a LIST for every creation. * New path capability "fs.s3a.create.performance.enabled" true if the option is enabled. * Parameterize ITestS3AContractCreate to expect the different outcomes * Parameterize ITestCreateFileCost similarly, with changed cost assertions there. * create(/) raises an IOE. existing bug only noticed here. Contributed by Steve Loughran
…pache#6168) If fs.s3a.create.performance is set on a bucket - All file overwrite checks are skipped, even if the caller says otherwise. - All directory existence checks are skipped. - Marker deletion is *always* skipped. This eliminates a HEAD and a LIST for every creation. * New path capability "fs.s3a.create.performance.enabled" true if the option is enabled. * Parameterize ITestS3AContractCreate to expect the different outcomes * Parameterize ITestCreateFileCost similarly, with changed cost assertions there. * create(/) raises an IOE. existing bug only noticed here. Contributed by Steve Loughran
…pache#6168) If fs.s3a.create.performance is set on a bucket - All file overwrite checks are skipped, even if the caller says otherwise. - All directory existence checks are skipped. - Marker deletion is *always* skipped. This eliminates a HEAD and a LIST for every creation. * New path capability "fs.s3a.create.performance.enabled" true if the option is enabled. * Parameterize ITestS3AContractCreate to expect the different outcomes * Parameterize ITestCreateFileCost similarly, with changed cost assertions there. * create(/) raises an IOE. existing bug only noticed here. Contributed by Steve Loughran
…pache#6168) If fs.s3a.create.performance is set on a bucket - All file overwrite checks are skipped, even if the caller says otherwise. - All directory existence checks are skipped. - Marker deletion is *always* skipped. This eliminates a HEAD and a LIST for every creation. * New path capability "fs.s3a.create.performance.enabled" true if the option is enabled. * Parameterize ITestS3AContractCreate to expect the different outcomes * Parameterize ITestCreateFileCost similarly, with changed cost assertions there. * create(/) raises an IOE. existing bug only noticed here. Contributed by Steve Loughran
Description of PR
HADOOP-18930. Make fs.s3a.create.performance a bucket-wide settings.
Adds the switch;
How was this patch tested?
doesn't change the tests which will fail when enabled, or provide and explicit way to set in a maven profile.
likely outcome
For code changes:
LICENSE
,LICENSE-binary
,NOTICE-binary
files?