Skip to content

[FLINK-8148][yarn/s3] fix test instability in YarnFileStageTestS3ITCase#5066

Closed
NicoK wants to merge 1 commit intoapache:masterfrom
NicoK:flink-8148
Closed

[FLINK-8148][yarn/s3] fix test instability in YarnFileStageTestS3ITCase#5066
NicoK wants to merge 1 commit intoapache:masterfrom
NicoK:flink-8148

Conversation

@NicoK
Copy link
Contributor

@NicoK NicoK commented Nov 24, 2017

What is the purpose of the change

YarnFileStageTestS3ITCase.testRecursiveUploadForYarn verifies that the test directory used is cleaned up by YarnFileStageTest.testCopyFromLocalRecursive which should clean up the directory (in a finally block). However, for S3, we may not always see our own deletes.

Quoting from https://aws.amazon.com/s3/faqs/ here:

Q: What data consistency model does Amazon S3 employ?
Amazon S3 buckets in all Regions provide read-after-write consistency for PUTS of new objects and eventual consistency for overwrite PUTS and DELETES.

Brief change log

  • Remove a check for a deleted directory since we may not see our own delete yet
    with S3.

Verifying this change

This change is a trivial rework / code cleanup without any test coverage.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? not applicable

Remove a check for a deleted directory since we may not see our own delete yet
with S3.
@StephanEwen
Copy link
Contributor

Makes sense, +1

@zentol
Copy link
Contributor

zentol commented Nov 29, 2017

merging.

zentol pushed a commit to zentol/flink that referenced this pull request Nov 29, 2017
Remove a check for a deleted directory since we may not see our own delete yet
with S3.

This closes apache#5066.
@zentol
Copy link
Contributor

zentol commented Nov 29, 2017

will fix the checkstyle violation on the way.

@NicoK
Copy link
Contributor Author

NicoK commented Nov 29, 2017

oh, sorry for that

@asfgit asfgit closed this in 4bd0ef1 Dec 4, 2017
zentol pushed a commit to zentol/flink that referenced this pull request Dec 4, 2017
Remove a check for a deleted directory since we may not see our own delete yet
with S3.

This closes apache#5066.
asfgit pushed a commit that referenced this pull request Dec 4, 2017
Remove a check for a deleted directory since we may not see our own delete yet
with S3.

This closes #5066.
glaksh100 pushed a commit to lyft/flink that referenced this pull request Jun 6, 2018
Remove a check for a deleted directory since we may not see our own delete yet
with S3.

This closes apache#5066.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants