DM-31941: Allow datastore.trash to remove unknown files and ingest to overwrite #578
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
timj
force-pushed
the
tickets/DM-31941
branch
from
0dac861
to
b6ca422
Compare
Codecov Report
@@ Coverage Diff @@
## master #578 +/- ##
==========================================
+ Coverage 83.54% 83.60% +0.05%
==========================================
Files 242 242
Lines 30197 30306 +109
Branches 4493 4507 +14
==========================================
+ Hits 25229 25337 +108
- Misses 3780 3782 +2
+ Partials 1188 1187 -1
Continue to review full report at Codecov.
|
timj
force-pushed
the
tickets/DM-31941
branch
from
b6ca422
to
5ab5bc5
Compare
|
@andy-slac I updated the trash commit to add some test coverage by turning on trust mode. |
andy-slac
approved these changes
Sep 29, 2021
This matches datastore.put behavior and trusts that the registry would not be asking for an overwrite unless it had good reason to do so.
If you asked to delete two rows it would double up every row that did not match. The "else" was indented the wrong amount.
This allows some areas to be tested that use set()
datastore.trash() is optimized to scan the records table and quickly move those records to the trash table for later deletion. For execution mode the files might be there but the records won't be. Therefore, to allow consistency for dataset pruning, when trust mode is enabled the file system is examined and files removed if they are present even if there are no records.
timj
force-pushed
the
tickets/DM-31941
branch
from
5ab5bc5
to
fb328a3
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Checklist
doc/changes