Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-35458: Always write local datastore files to temporaries #705

Merged
merged 4 commits into from Jul 7, 2022

Conversation

timj
Copy link
Member

@timj timj commented Jul 5, 2022

This allows for an atomic rename in case something goes wrong
in the file writing.

Checklist

  • ran Jenkins
  • added a release note for user-visible changes to doc/changes

@codecov
Copy link

codecov bot commented Jul 5, 2022

Codecov Report

Merging #705 (08a51ac) into main (df4b34c) will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #705      +/-   ##
==========================================
+ Coverage   84.54%   84.55%   +0.01%     
==========================================
  Files         243      243              
  Lines       31825    31835      +10     
  Branches     5966     5969       +3     
==========================================
+ Hits        26906    26918      +12     
+ Misses       3747     3745       -2     
  Partials     1172     1172              
Impacted Files Coverage Δ
python/lsst/daf/butler/datastores/fileDatastore.py 81.10% <100.00%> (+0.17%) ⬆️
tests/test_datastore.py 99.48% <100.00%> (+<0.01%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update df4b34c...08a51ac. Read the comment docs.

@timj timj marked this pull request as ready for review July 6, 2022 21:47
Copy link
Contributor

@ktlim ktlim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks OK. Any chance to test the new path? (Maybe start writing a large file, then use inotify to see that the temp file has shown up?)

timj added 2 commits July 6, 2022 15:35
This allows for an atomic rename in case something goes wrong
in the file writing.
@timj
Copy link
Member Author

timj commented Jul 6, 2022

There is not a special path for local files any more. Everything uses a temporary and I can confirm that the log messages are showing that the temporary is in use and is then moved over to the final location.

@ktlim
Copy link
Contributor

ktlim commented Jul 6, 2022

But you don't check those log messages in a test case, do you? In this case the side effect of the implementation choice is significant.

@timj
Copy link
Member Author

timj commented Jul 6, 2022

I don't check the log message. What I can do is add a check that the resources logger is reporting a move transfer for local writes. The only code choice there is is whether a move or copy is happening to the temporary file (and whether the temp file is inside datastore or not).

@timj timj merged commit a69945d into main Jul 7, 2022
@timj timj deleted the tickets/DM-35458 branch July 7, 2022 04:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants