More flexible clone URL generation for submodules #3828

mih · 2019-10-23T16:07:36Z

This fixes gh-3827, while also tweaking a little bit on the sides -- this is rather old code.

This is technically ready. Once agreed on, there will be a use case description in the handbook on feature.

For which our wrapper had to be extended. One could argue that `.call_git_items_()` would be at least as adequate, but I chose to invest in my prior investments ;-)

The desire is to slim the interface.

To align with get_hexsha()

No StopIteration could have come out of that code block

codecov · 2019-10-23T17:04:25Z

Codecov Report

Merging #3828 into master will increase coverage by 0.02%.
The diff coverage is 92.59%.

@@            Coverage Diff             @@
##           master    #3828      +/-   ##
==========================================
+ Coverage   80.71%   80.73%   +0.02%     
==========================================
  Files         273      273              
  Lines       35905    35925      +20     
==========================================
+ Hits        28979    29004      +25     
+ Misses       6926     6921       -5

Impacted Files	Coverage Δ
datalad/distribution/utils.py	`89.74% <ø> (ø)`	⬆️
datalad/metadata/aggregate.py	`14.24% <0%> (ø)`	⬆️
datalad/metadata/search.py	`35.39% <0%> (ø)`	⬆️
datalad/distribution/tests/test_get.py	`100% <100%> (ø)`	⬆️
datalad/distribution/get.py	`81.73% <100%> (+1.73%)`	⬆️
datalad/support/tests/test_gitrepo.py	`99.88% <100%> (ø)`	⬆️
datalad/support/gitrepo.py	`83.5% <71.42%> (-0.11%)`	⬇️
datalad/distribution/siblings.py	`54.46% <0%> (+0.3%)`	⬆️
datalad/log.py	`65.86% <0%> (+0.96%)`	⬆️
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1592db4...ad04e53. Read the comment docs.

…enerator

Fixes dataladgh-3827

This touches upon an idea by @yaricoptic expressed in datalad#3827 (comment) and merely lays the foundation for further work. Names are not unique (and I don't see how they could be, or what we would gain from it), hence no dict. Choice of names is debatable too. But given that they are not used ATM, it doesn't really matter and can easily be changed.

[ci skip]

kyleam · 2019-10-25T00:37:27Z

datalad/support/gitrepo.py

@@ -1308,6 +1309,9 @@ def for_each_ref_(self, fields=('objectname', 'objecttype', 'refname'),
          descending order.
        count : int, optional
          Stop iteration after the given number of matches.
+        contains : str, optional
+          Only list refs which contain the specified commit (HEAD if not
+          specified).


If not specified, it's None, and --contains isn't added. That's not the same as restricting the results to --contains=HEAD.

Fixed, by removing that part from the docs (copied from Git).

kyleam · 2019-10-25T00:37:27Z

datalad/distribution/get.py

+    return [
+        remote
+        for remote in repo.get_remotes(with_urls_only=with_urls_only)
+        if any(rb.startswith(remote + '/') for rb in remote_branches)


I'm not sure I see the point of using get_remote(with_urls_only=...) here, especially given this is only used in one spot. Wouldn't for_each_ref_s output (with an additional layer stripped) be sufficient?

This is just keeping what the keep did before. I tried to get rid of it, but couldn't figure out why things would break. I'll leave it for another time.

This is just keeping what the keep did before.

Yes...

I tried to get rid of it, but couldn't figure out why things would break.

ah, all right, so my guess that it isn't need is wrong. Thanks.

datalad/support/gitrepo.py

kyleam · 2019-10-25T00:37:27Z

datalad/support/gitrepo.py

        except CommandError as e:
-            if 'does not have any commits' in e.stderr:
+            if 'unknown revision or path not in the working tree' in e.stderr:


From my POV it'd be best to avoid brittle error message matching if we can. Given we're in the error handling arm, another git call seems acceptable, so this could be replaced by if self.get_hexsha() is None. (get_hexsha() relies on format_commit which itself does error message matching, but at least we'd avoid introducing another spot with a new message.)

kyleam · 2019-10-25T00:37:27Z

datalad/distribution/get.py

+            # outside. Additional guesswork can only make it slower
+            clone_urls.append((name, url))
+
+    # CANDIDATE: the actual configured gitmodule URL


I find this new wording more confusing than what it replaced. In Git's eyes, relative paths have the following resolution: (1) relative to the URL of the tracking branch, (2) if there is not a tracking branch, relative to the URL of "origin", or (3) if there is not a configured remote, relative to the working directory. So the new description is only accurately describing what Git considers to be the configured URL in cases that fall through to no. 3.

(As a sidenote that's not directly related to these changes, it looks like _get_flexible_source_candidates_for_submodule doesn't handle case no. 2, so there's a scenario where git submodule can clone something while datalad install can't.)

kyleam · 2019-10-25T00:37:27Z

datalad/distribution/get.py

-            ds.path,
-            alternate_suffix=False)
+        clone_urls.extend(
+            ('origin', url)


It sounds like these names aren't used for anything at this point, so it doesn't really matter, but perhaps "local" would be a better description?

Funny enough local is what I had. Now it is local again ;-)

@kyleam

Following a suggestion from @kyleam

@kyleam

Thanks @kyleam

kyleam

@mih Thanks for the updates. Looks good to me (I know you haven't responded to the # CANDIDATE: bit yet, but it's fine with me if that remains unchanged).

mih · 2019-10-28T16:05:57Z

@kyleam Love'ya!

mih added 6 commits Oct 23, 2019

RF: Replace custom git branch call with preferable git for-each-ref

f0f5926

For which our wrapper had to be extended. One could argue that `.call_git_items_()` would be at least as adequate, but I chose to invest in my prior investments ;-)

RF: Relocate single-use internal helper for get()

9424aeb

The desire is to slim the interface.

RF: Reimplement get_last_commit_hash() with Git plumbing

bf32550

RF: get_last_commit_hash() -> get_last_commit_hexsha()

62773e6

To align with get_hexsha()

BF: Turn no longer valid error handling into a plain conditional

116d611

No StopIteration could have come out of that code block

DOC: A bit more insight written down, remove misplace/outdate comment

0ac84c1

RF: Deliver all known submodule props to the clone URL alternatives g…

2650455

…enerator

mih force-pushed the enh-submoduleclone branch from ca65455 to 14f4873 Compare Oct 24, 2019

mih added 2 commits Oct 24, 2019

NF: Support additional subdataset clone source from configuration

fd7ba6a

Fixes dataladgh-3827

mih force-pushed the enh-submoduleclone branch from 14f4873 to 8e99c2f Compare Oct 24, 2019

mih mentioned this pull request Oct 24, 2019

RF: Move tracking branch update functionality into GitRepo.update_submodule() #3831

Merged

TST: clone URL alternative generation

1c2a659

mih marked this pull request as ready for review Oct 24, 2019

DOC: get: Fix some recent docstring typos

7eac837

[ci skip]

kyleam requested changes Oct 25, 2019

View changes

mih added 3 commits Oct 28, 2019

RF: Use 'local' label for gitmodule URL configured for a local submodule

e565841

Following a suggestion from @kyleam

ENH: Rebustify error detection

96b88fc

Thanks @kyleam

DOC: Fixup overreaching copy&paste from Git docs.

ad04e53

mih force-pushed the enh-submoduleclone branch from ac91db6 to ad04e53 Compare Oct 28, 2019

kyleam approved these changes Oct 28, 2019

View changes

mih merged commit 236204c into datalad:master Oct 28, 2019
16 checks passed

mih deleted the enh-submoduleclone branch Oct 28, 2019

More flexible clone URL generation for submodules #3828

More flexible clone URL generation for submodules #3828

mih commented Oct 23, 2019 •

edited

codecov bot commented Oct 23, 2019 •

edited

kyleam Oct 25, 2019

mih Oct 28, 2019

kyleam Oct 25, 2019

mih Oct 28, 2019

kyleam Oct 28, 2019

kyleam Oct 25, 2019

mih Oct 28, 2019

kyleam Oct 25, 2019

kyleam Oct 25, 2019

mih Oct 28, 2019

kyleam left a comment

mih commented Oct 28, 2019

More flexible clone URL generation for submodules #3828

More flexible clone URL generation for submodules #3828

Conversation

mih commented Oct 23, 2019 • edited

codecov bot commented Oct 23, 2019 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kyleam left a comment

mih commented Oct 28, 2019

mih commented Oct 23, 2019 •

edited

codecov bot commented Oct 23, 2019 •

edited