[FLINK-7196][blob] add a TTL to all transient BLOBs #4381
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
NicoK
force-pushed
the
flink-6916-7196
branch
2 times, most recently
from
569cd83 to
8f401d5
Compare
|
rebased onto the latest #4359 |
added 2 commits
October 6, 2017 11:34
This should guard us from uploading (and deleting) the same file more than once and also from hash collisions.
Transient BLOB files should not exist for long and are only deleted manually from the caches and after access on the server. This uses the BLOB storage's cleanup interval to set a TTL on all transient BLOB files as a backup cleanup path. The cleanup task itself runs every cleanupInterval seconds and removes all transient BLOBs for which the TTL is older than the current time. This way, a transient BLOB stays at most 2*cleanupInterval seconds before getting deleted automatically.
tillrohrmann
approved these changes
Oct 20, 2017
There was a problem hiding this comment.
Changes look good to me. Thanks for your contribution @NicoK. I'll throw again on Travis and if Travis gives green light, then I'll merge it.
tillrohrmann
pushed a commit
to tillrohrmann/flink
that referenced
this pull request
Oct 20, 2017
Transient BLOB files should not exist for long and are only deleted manually from the caches and after access on the server. This uses the BLOB storage's cleanup interval to set a TTL on all transient BLOB files as a backup cleanup path. The cleanup task itself runs every cleanupInterval seconds and removes all transient BLOBs for which the TTL is older than the current time. This way, a transient BLOB stays at most 2*cleanupInterval seconds before getting deleted automatically. This closes apache#4381.
tillrohrmann
pushed a commit
to tillrohrmann/flink
that referenced
this pull request
Oct 23, 2017
Transient BLOB files should not exist for long and are only deleted manually from the caches and after access on the server. This uses the BLOB storage's cleanup interval to set a TTL on all transient BLOB files as a backup cleanup path. The cleanup task itself runs every cleanupInterval seconds and removes all transient BLOBs for which the TTL is older than the current time. This way, a transient BLOB stays at most 2*cleanupInterval seconds before getting deleted automatically. This closes apache#4381.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
Transient BLOB files are currently only deleted manually and may thus linger around if not cleaned up. We should introduce a default cleanup time, i.e. a time-to-live (TTL), as a backup cleanup path.
Brief change log
TransientBlobService, i.e.BlobServerandTransientBlobCache, gets an additional member keeping track of all expiry times of transient BLOBs.TransientBlobService, its TTL will be set tocurrentTime + cleanupInterval.TransientBlobCleanupTask) is executed everycleanupIntervalseconds and deletes all expired transient BLOBs.2 * cleanupIntervalseconds before getting deleted automatically.Please note that this PR is based on #4359 in a series to implement FLIP-19.
Verifying this change
This change adds tests and can be verified as follows:
BlobServerandTransientBlobCachewith and without a job ID) that after adding two transient BLOBs with further accesses only to one of them, that the accessed one remains and the other is deleted. For the existence check, however, there is a slight race condition: if the test is slow and delayed by 1s, the cleanup may kick in and the test may fail. This is hard to test otherwise without further intrusion into the blob store classes.Does this pull request potentially affect one of the following parts:
@Public(Evolving): (no)Documentation