NF: Support cloning of specific repository versions (fixes gh-2109) #4036
Commits on Jan 17, 2020
-
NF: Support cloning of specific repository versions (fixes dataladgh-…
…2109) `GitRepo.clone()` is enhanced to accept arbitrary additional options that are passed on to `git-clone`. We use the `--branch` option (that, despite its name, can handle any relevant version identifier) to let `git-clone` do all the work. `datalad-clone` is now set up to acknowlege a request for a particular version, for any kind of source URL where `decode_source_spec()` yields a non-None `version` property. At the moment this is only the case for `ria+http|ssh://` URL, but future additions only need to alter/enhance `decode_source_spec()` to extend this functionality for other types. This change also sets the stage for dataladgh-4035
-
-
TST: Verify we fail on non-existing version
But: Here is what it looks like when requesting a version that doesn't exist: ``` % datalad clone "ria+http://127.0.0.1:41207/#33c2e000-38fa-11ea-aa8f-f0d5bf7b5561@impossible" [ERROR ] Failed to clone from all attempted sources: ['http://127.0.0.1:41207/33c/2e000-38fa-11ea-aa8f-f0d5bf7b5561', 'http://127.0.0.1:41207/33c/2e000-38fa-11ea-aa8f-f0d5bf7b5561/.git'] [install(/tmp/33c2e000-38fa-11ea-aa8f-f0d5bf7b5561)] install(error): /tmp/33c2e000-38fa-11ea-aa8f-f0d5bf7b5561 (dataset) [Failed to clone from all attempted sources: ['http://127.0.0.1:41207/33c/2e000-38fa-11ea-aa8f-f0d5bf7b5561', 'http://127.0.0.1:41207/33c/2e000-38fa-11ea-aa8f-f0d5bf7b5561/.git']] ``` If we ignore the pointless double-reporting, we see that the error is wrong. The clone worked just fine, but the checkout failed. I would expect `git-clone` to be more clever, and just say it like it is. Turns out it is ``` % git clone --branch impossible http://127.0.0.1:41207/33c/2e000-38fa-11ea-aa8f-f0d5bf7b5561 Cloning into '2e000-38fa-11ea-aa8f-f0d5bf7b5561'... fatal: Remote branch impossible not found in upstream origin ``` ``` (Pdb) GitRepo.clone(url='http://127.0.0.1:41207/33c/2e000-38fa-11ea-aa8f-f0d5bf7b5561', path='/tmp/broken', create=True, clone_options={'branch': 'impossible'}) [DEBUG ] Git clone from http://127.0.0.1:41207/33c/2e000-38fa-11ea-aa8f-f0d5bf7b5561 to /tmp/broken [DEBUG ] HTTP: "GET /33c/2e000-38fa-11ea-aa8f-f0d5bf7b5561/info/refs?service=git-upload-pack HTTP/1.1" 200 - [DEBUG ] HTTP: "GET /33c/2e000-38fa-11ea-aa8f-f0d5bf7b5561/HEAD HTTP/1.1" 200 - *** git.exc.GitCommandError: Cmd('/usr/bin/git') failed due to: exit code(128) cmdline: /usr/bin/git clone --progress -v --branch=impossible http://127.0.0.1:41207/33c/2e000-38fa-11ea-aa8f-f0d5bf7b5561 /tmp/broken ``` Also reveals the issue, but does not seem to include the critical information in the exception ``` (Pdb) gitpy.Repo.clone_from('http://127.0.0.1:32771/2bd/222e6-38fc-11ea-a876-f0d5bf7b5561', '/tmp/broken', multi_options=['--branch=impossible']) [DEBUG ] HTTP: "GET /2bd/222e6-38fc-11ea-a876-f0d5bf7b5561/info/refs?service=git-upload-pack HTTP/1.1" 200 - [DEBUG ] HTTP: "GET /2bd/222e6-38fc-11ea-a876-f0d5bf7b5561/HEAD HTTP/1.1" 200 - *** git.exc.GitCommandError: Cmd('/usr/bin/git') failed due to: exit code(128) cmdline: /usr/bin/git clone -v --branch=impossible http://127.0.0.1:32771/2bd/222e6-38fc-11ea-a876-f0d5bf7b5561 /tmp/broken stderr: 'Cloning into '/tmp/broken'... fatal: Remote branch impossible not found in upstream origin ... -> e_str = exc_str(e) (Pdb) e.stderr '' (Pdb) e.stdout '' (Pdb) e.status 128 (Pdb) str(e) "Cmd('/usr/bin/git') failed due to: exit code(128)\n cmdline: /usr/bin/git clone --progress -v http://127.0.0.1:38771/a23/42448-38fc-11ea-a812-f0d5bf7b5561/subdir/subds /tmp/datalad_temp_tree_test_ria_httph0n4roy6/clone/subdir/subds" ``` Possibly the progress reporting makes the output vanish ``` (Pdb) git_progress = GitPythonProgressBar("Cloning") (Pdb) gitpy.Repo.clone_from('http://127.0.0.1:38771/a23/42448-38fc-11ea-a812-f0d5bf7b5561', '/tmp/broken', multi_options=['--branch=impossible'], progress=git_progress) [DEBUG ] HTTP: "GET /a23/42448-38fc-11ea-a812-f0d5bf7b5561/info/refs?service=git-upload-pack HTTP/1.1" 200 - [DEBUG ] HTTP: "GET /a23/42448-38fc-11ea-a812-f0d5bf7b5561/HEAD HTTP/1.1" 200 - *** git.exc.GitCommandError: Cmd('/usr/bin/git') failed due to: exit code(128) cmdline: /usr/bin/git clone --progress -v --branch=impossible http://127.0.0.1:38771/a23/42448-38fc-11ea-a812-f0d5bf7b5561 /tmp/broken ``` Seems to be the case :( But it is not git's fault: ``` % /usr/bin/git clone --progress -v --branch=impossible http://127.0.0.1:38771/a23/42448-38fc-11ea-a812-f0d5bf7b5561 /tmp/broken Cloning into '/tmp/broken'... fatal: Remote branch impossible not found in upstream origin ``` @bpoldrack points out that this line may give a hint on where we are discarding this information https://github.com/datalad/datalad/blame/master/datalad/support/gitrepo.py#L480/datalad/datalad/commit/0610dda16d4181d2525af0f5dd90c4da370ed840
-
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.