-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
proc: process info files should be locked during writing #53
Labels
bug
Something isn't working
Comments
karajan1001
added a commit
that referenced
this issue
Jun 16, 2022
fix: #53 Currently, the dump and load process info file at a time may cause some error. Write to a temp file and then rename it eases this problem. But a final solution would still be required in someday later.
pmrowla
pushed a commit
that referenced
this issue
Jun 16, 2022
fix: #53 Currently, the dump and load process info file at a time may cause some error. Write to a temp file and then rename it eases this problem. But a final solution would still be required in someday later.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In certain situations it's possible that a reader tries to load proc info from file while it's being written to which causes a jsondecodeerror. ProcessManager should provide minimal amount of synchronization for these cases.
Alternatively, at a minimum we should just write to temporary paths (i.e.
output_path.json.<random suffix>
) and then fs.move to replace the existing file in:dvc-task/src/dvc_task/proc/process.py
Lines 133 to 138 in d132caa
The text was updated successfully, but these errors were encountered: