BF: Handle IncompleteResultsError in cloning dataset #98
Conversation
Cloning a dataset sometimes can fail with a IncompleteResultsError. Simply waiting and retrying will resolve the problem in most cases
Codecov ReportBase: 95.43% // Head: 94.54% // Decreases project coverage by
Additional details and impacted files@@ Coverage Diff @@
## master #98 +/- ##
==========================================
- Coverage 95.43% 94.54% -0.90%
==========================================
Files 26 26
Lines 1030 1044 +14
==========================================
+ Hits 983 987 +4
- Misses 47 57 +10
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
datalad_registry/tasks.py
Outdated
| try: | ||
| ds = dl.clone(url, ds_path_str) | ||
| except IncompleteResultsError: | ||
| lgr.warning(f"Incomplete results error in cloning {url}. Trying again.") |
There was a problem hiding this comment.
os.unlink(ds_path_str) before retrying to ensure that follow up re-cloning is not somehow skipping since target exists but not fully kosher.
|
in the future checkout the tenacity library for generic framework for retries |
Cloning a dataset sometimes can fail with an IncompleteResultsError. Simply retrying will resolve the problem in most cases.
This PR is only a step toward a solution for #93. A complete solution would require the handling of the case in which all retries of cloning fail. However, with this PR, the population of datasets would be stable enough for the work for #89 to start.