-
Notifications
You must be signed in to change notification settings - Fork 8.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HADOOP-17559. S3guard import OOM. #2734
HADOOP-17559. S3guard import OOM. #2734
Conversation
Remove all tracking of files from DDB AncestorState; dirs in import tool. Reduces size of the cache to O(dirs). Test change is to reduce brittleness to clock skew on loaded test runs; removes an intermittent failure where the existence assert was triggering a s3guard update -which then broke the assert about the number of writes Change-Id: I9251f64beb0fec225b0b4ba71bc16f3e116bc758
tested: s3 london with s3guard. Some failures I'm fixing in the audit patch; only new failure was the intermittent one in ITestCommitOperations which I suspected was a regression, but tracked it down in the logs to clock skew: initial PUT of the output file uses local clock, but s3 uses its clock, and on HEAD request this is updated. |
Stack from the test failure fixed here as seen in the audit PR. Shows its not directly related to this, though I had to stop through to make sure
|
+Add a test to verify we can import with spaces in the filenames |
closing as unmerged; leaving the PR up |
Remove all tracking of files from DDB AncestorState; dirs in import tool.
Reduces size of the cache to O(dirs).
Test change is to reduce brittleness to clock skew on loaded test runs;
removes an intermittent failure where the existence assert was triggering
a s3guard update -which then broke the assert about the number of writes
Change-Id: I9251f64beb0fec225b0b4ba71bc16f3e116bc758