ENH: Give clone candidates a priority #4619

mih · 2020-06-09T15:46:50Z

This is achieved by prefixing their existing label with three digits and sorting them before processing them.

This implementation makes no change to the previously established order, it just makes that order explicit in the labels.

The default setup uses the priorities in the second half of the spectrum:

500 for remote URL + submodule path
600 for the configured submodule URL
700 for any unprioritized 'datalad.get.subdataset-source-candidate' config
900 for the local subdataset path

The first half of the spectrum and the spaces inbetween are available for re-prioritizing clone attempts. The idea is to be able to configure

datalad.get.subdataset-source-candidate-000takemefirst = http://some

and have this be processed first.

I have confirmed manually that this solution enables successful clones on first attempt, and substantially speeds up dataset installation. Likewise using '999label' it is possible to configure a fallback source candidate.

Towards a resolution of gh-4613

TODO:

adjust clone to use this feature
adjust documentation
add test
~~potentially prevent "flexibilizing" of explicitly configured URLs (e.g. /.git appending)~~

This change has no effect, because the final unique() immediately takes them out again (and always did), but it is still wrong and confusing.

codecov · 2020-06-09T20:25:48Z

Codecov Report

Merging #4619 into master will increase coverage by 39.76%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##           master    #4619       +/-   ##
===========================================
+ Coverage   49.82%   89.59%   +39.76%     
===========================================
  Files         286      288        +2     
  Lines       38822    39016      +194     
===========================================
+ Hits        19344    34957    +15613     
+ Misses      19478     4059    -15419

Impacted Files	Coverage Δ
datalad/core/distributed/clone.py	`89.26% <ø> (+43.86%)`	⬆️
datalad/core/distributed/tests/test_clone.py	`93.41% <100.00%> (+87.59%)`	⬆️
datalad/distribution/get.py	`96.22% <100.00%> (+17.20%)`	⬆️
datalad/distribution/tests/test_get.py	`100.00% <100.00%> (+93.04%)`	⬆️
datalad/tests/utils_cached_dataset.py	`92.30% <0.00%> (ø)`
datalad/tests/test_utils_cached_dataset.py	`100.00% <0.00%> (ø)`
datalad/core/local/tests/test_create.py	`100.00% <0.00%> (+0.44%)`	⬆️
datalad/local/subdatasets.py	`95.65% <0.00%> (+0.86%)`	⬆️
datalad/core/local/diff.py	`95.29% <0.00%> (+1.17%)`	⬆️
datalad/support/external_versions.py	`96.07% <0.00%> (+1.30%)`	⬆️
... and 181 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f0adaed...a30b7c6. Read the comment docs.

mih · 2020-06-11T10:53:17Z

Crippled failure, not yet replicatable locally...

======================================================================
FAIL: datalad.core.distributed.tests.test_push.test_push(True,)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.5.9/x64/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/opt/hostedtoolcache/Python/3.5.9/x64/lib/python3.5/site-packages/datalad/tests/utils.py", line 714, in newfunc
    return t(*(arg + (filename,)), **kw)
  File "/opt/hostedtoolcache/Python/3.5.9/x64/lib/python3.5/site-packages/datalad/tests/utils.py", line 714, in newfunc
    return t(*(arg + (filename,)), **kw)
  File "/opt/hostedtoolcache/Python/3.5.9/x64/lib/python3.5/site-packages/datalad/core/distributed/tests/test_push.py", line 131, in check_push
    assert_result_count(res, 2 if annex else 1)
  File "/opt/hostedtoolcache/Python/3.5.9/x64/lib/python3.5/site-packages/datalad/tests/utils.py", line 1305, in assert_result_count
    _format_res(results)))
AssertionError: Got 3 instead of 2 expected results matching
  {}
...

Because it is spurious... :(

mih · 2020-06-11T14:46:03Z

I will not add the ability to disable automatically appending /.git to all relevant URLs. It is unclear to me when that is possible without impairing legitimate use cases. The sorting already enables to get in front of the list of candidate and provide a URL that works as the first one to be tried.

This is achieved by simply prefixing their existing label with two digits and sorting them before processing them. This implementation makes no change to the previously established order, it just makes that order explicit in the labels. The default setup uses the priorities in the second half of the spectrum: 50 for remote URL + submodule path 60 for the configured submodule URL 70 for any unprioritized 'datalad.get.subdataset-source-candidate' config 90 for the local subdataset path The first half of the spectrum and the spaces inbetween are available for re-prioritizing clone attempts. The idea is to be able to configure datalad.get.subdataset-source-candidate-00takemefirst = http://some and have this be processed first. Likewise using '99label' it is possible to configure a fallback source candidate.

… clones This avoid 3-5 unsuccessful clone attempts per subdataset in all cases with subdataset can be found in the same store. It added 1-2 failed attempts when this is not the case. However, those were already present before.

yarikoptic

Left some ideas to make it work with already existing clones, and making it more explicit than embedding into a variable name, and a typo fix.
Also, I think looking forward it might be better to make priorities (or costs) to align with git-annex which uses three digits level (as e.g. debian does for apt priorities). That might allow later to align priorities of remotes with e.g. priorities for annex remotes and/or URLs (based on regexes).

datalad/distribution/get.py

yarikoptic · 2020-06-11T15:31:17Z

datalad/core/distributed/clone.py

+        # we pick a priority of 20 to sort it before datalad's default candidates
+        # for non-RIA URLs, because they prioritize hierarchical layouts that
+        # cannot be found in a RIA store
+        'datalad.get.subdataset-source-candidate-20origin',


I wondered if it might make sense to separate out priority from the variable: i.e. introduce e.g.datalad.get.subdataset-source-candidate-origin-priority which if not set would get 20. That would

make existing clones work without any changes

make it more explicit

internal implementation could do whatever it does now (I would have preferred to have priority an explicit variable/tuple element to avoid all the splitting etc) and could be refactored later if desired/needed

If a second variable is introduced, one would also need to make sure that they stay in sync across all sources we read config from. I do not know how to achieve that, and how to test whether I have achieved that. It is perfectly possible that datalad.get.subdataset-source-candidate-origin-priority comes from user config and datalad.get.subdataset-source-candidate-origin does not.

I am not much concerned about existing clones -- I have all of them in front of me.

I addressed the topic of the messy internal implementation in 05e4d8f and it also switched to a three-digit priority setup.

It is perfectly possible that datalad.get.subdataset-source-candidate-origin-priority comes from user config and datalad.get.subdataset-source-candidate-origin does not.

Wouldn't that be a feature - I could boost or penalize priority of all ria stores with one config setting?

Besides I would follow annex and call it cost and not priority (ambiguous - is higher number a higher priority? More obvious with cost, and is taking the one with lower cost first).

And in current approach it would be easy only to raise priority (lower cost) but not to lower it since the original variable would still be available somewhere with higher cost?

Wouldn't that be a feature - I could boost or penalize priority of all ria stores with one config setting?

That may be, but a feature that I do not comprehend enough to be the one implementing it, and there wasn't a use case for it either.

Besides I would follow annex and call it cost and not priority (ambiguous - is higher number a higher priority? More obvious with cost, and is taking the one with lower cost first).

I can do this, if needed (but it may not be enough, see below).

And in current approach it would be easy only to raise priority (lower cost) but not to lower it since the original variable would still be available somewhere with higher cost?

There is no concept of raising or lowering, there are just candidates with (default) priorities.

I suspect that you want to aim for a bigger solution. If you insist, I can obscure the fact that the sorting can be used to achieve a prioritizing and just talk about sorting. In this case the space would be open for the feature that you have in mind. I just want to get a handle on gh-4613, have that quickly, and then I will detach from this topic.

There is no concept of raising or lowering, there are just candidates with (default) priorities.

ah -- then there should be no need for any config changes? just RF code (if still needed) so priority for RIA remotes is higher and be done?

I suspect that you want to aim for a bigger solution. If you insist, ...

oh no -- I am not insisting or even suggesting to implement anything bigger solution here. I just want this solution not require changes to user visible parts (e.g. config) later if we decide to go with some global user visible approach.

If an existing unprioritized candidate configuration is found, it automatically gets priority 700 such that identical behavior is maintained. This doesn't fix gh-4613, but I see no reason for doing such intervention in repos where this wasn't a problem so far (and I believe there are just a handful). All new clones will get the new behavior.

Follows up on a aspected of datalad#4619 (comment) Stop merging and splitting strings, but use more appropriate data types (dicts for records, int for priorities). Also switch to a three-digit priority to align with other setups (datalad#4619 (review)).

Addresses: datalad#4619 (comment)

This .split() call was dropped with the switch to structured records. [ci skip]

[ci skip]

This no longer applies after the switch to structured records. [ci skip]

kyleam

The approach looks good to me.

To resolve the polarity issue pointed out in datalad#4619 (comment)

mih · 2020-06-15T05:25:58Z

Thanks for the reviews and for the approval @kyleam

I took the liberty to still rename priority to cost to resolve interpretation issue pointed out in #4619 (comment)

Will merge when green.

BF: Do not add duplicate source candidates per each remote

c69ddae

This change has no effect, because the final unique() immediately takes them out again (and always did), but it is still wrong and confusing.

mih force-pushed the bf-4613 branch from 80ec947 to 71045a3 Compare Jun 9, 2020

mih force-pushed the bf-4613 branch from 71045a3 to e6f1436 Compare Jun 11, 2020

mih mentioned this pull request Jun 11, 2020

Document how subdatasets can be cloned from different URLS datalad-handbook/book#523

Closed

mih added 2 commits Jun 11, 2020

mih force-pushed the bf-4613 branch from e6f1436 to 47a1aa0 Compare Jun 11, 2020

yarikoptic reviewed Jun 11, 2020

View changes

mih and others added 6 commits Jun 12, 2020

RF: Make more obvious what is being stripped

e0798e9

Addresses: datalad#4619 (comment)

RF: Adjust clone to use three-digit priority variables

a80c660

DOC: get: Remove a stale comment

9855323

This .split() call was dropped with the switch to structured records. [ci skip]

DOC: clone/get: Update stale bits related to 3 digit priority switch

bc8d3c7

[ci skip]

DOC: get: Remove a comment about ":" priority delimiter

aaf0cc5

This no longer applies after the switch to structured records. [ci skip]

kyleam approved these changes Jun 12, 2020

View changes

RF: priority -> cost

a30b7c6

To resolve the polarity issue pointed out in datalad#4619 (comment)

mih merged commit 896aa20 into datalad:master Jun 15, 2020
12 checks passed

mih deleted the bf-4613 branch Jun 15, 2020

bpoldrack mentioned this pull request May 20, 2021

RIA subdataset install "broken" by early success #4613

Closed

ENH: Give clone candidates a priority #4619

ENH: Give clone candidates a priority #4619

mih commented Jun 9, 2020 •

edited

codecov bot commented Jun 9, 2020 •

edited

mih commented Jun 11, 2020 •

edited

mih commented Jun 11, 2020 •

edited

yarikoptic left a comment

yarikoptic Jun 11, 2020

mih Jun 12, 2020 •

edited

mih Jun 12, 2020

yarikoptic Jun 12, 2020

yarikoptic Jun 12, 2020

yarikoptic Jun 12, 2020

mih Jun 12, 2020

yarikoptic Jun 12, 2020

mih Jun 15, 2020

kyleam left a comment

mih commented Jun 15, 2020

ENH: Give clone candidates a priority #4619

ENH: Give clone candidates a priority #4619

Conversation

mih commented Jun 9, 2020 • edited

codecov bot commented Jun 9, 2020 • edited

Codecov Report

mih commented Jun 11, 2020 • edited

mih commented Jun 11, 2020 • edited

yarikoptic left a comment

Choose a reason for hiding this comment

mih Jun 12, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kyleam left a comment

mih commented Jun 15, 2020

mih commented Jun 9, 2020 •

edited

codecov bot commented Jun 9, 2020 •

edited

mih commented Jun 11, 2020 •

edited

mih commented Jun 11, 2020 •

edited

mih Jun 12, 2020 •

edited