initremote type=git fresh "regression"

[yarikoptic]

This is a sattelite of originally filed only in git-annex for @joeyh at https://git-annex.branchable.com/bugs/initremote_type__61__git_fresh___34__regression__34__/ and which nobody yet troubleshooted to resolution yet, but which started to appear in our CI due to recent release of git-annex.

[mih]

I can confirm the test failure with git-annex version: 10.20220526-gc6b112108

However, it is not that this is "broken" now, it just requires the actual URL to be specified.

The test aspect

datalad/datalad/distribution/tests/test_siblings.py

Lines 540 to 545 in 5eb8358

	res = ds_fromurl.siblings(
	'configure',
	name=DEFAULT_REMOTE,
	as_common_datasrc='mike2',
	result_renderer='disabled',
	)

works and yields (notice the .git extension):

% git cat-file -p git-annex:remote.log
1c48353d-e720-4f40-89e5-ff4ca46a5739 autoenable=true location=http://127.0.0.1:41831/.git name=mike2 type=git timestamp=1656693098.117189159s

The subsequent test aspect

datalad/datalad/distribution/tests/test_siblings.py

Lines 548 to 554 in 5eb8358

	res = ds_fromurl.siblings(
	'add',
	name='fresh',
	url=url,
	as_common_datasrc='fresh-sr',
	result_renderer='disabled',
	)

forces a different url

-> res = ds_fromurl.siblings(
(Pdb) p url
'http://127.0.0.1:41831/'

This is not the URL one can clone from, because the test uses a non-bare repository as subject. Up to this line the test works, because datalad clone through several layers of "convenience" determines a workable URL from the inaccurate input.

The test is broken.

Here is a possible fix

diff --git a/datalad/distribution/tests/test_siblings.py b/datalad/distribution/tests/test_siblings.py
index b5a49ee58..d60f6f1fc 100644
--- a/datalad/distribution/tests/test_siblings.py
+++ b/datalad/distribution/tests/test_siblings.py
@@ -545,6 +545,10 @@ def test_as_common_datasource(testbed=None, viapath=None, viaurl=None, remotepat
     )
     assert_status('ok', res)
     # same thing should be possible by adding a fresh remote
+    # but we must amend the URL given by serve_path_via_http, because
+    # we are serving the root of a non-bare repository, but git-annex needs
+    # to talk to its .git
+    url += '.git'
     res = ds_fromurl.siblings(
         'add',
         name='fresh',

[yarikoptic]

However, it is not that this is "broken" now, it just requires the actual URL to be specified.

I am still curious on just how did it (git-annex) work before - isn't the rest of that test verifies that git/git-annex handles that URL ok even without .git suffix?

[mih]

No, the rest of the test works with the URL that datalad clone amended with the missing suffix.

[yarikoptic]

No, the rest of the test works with the URL that datalad clone amended with the missing suffix.

hm... I thought we are talking about URL within special remote, so not subject to datalad clone and rather git-annex automatically adding it during annex init... doing some code archaeology to discover fun commits like 0cbd5ac and code like https://github.com/datalad/datalad/blob/HEAD/datalad/distribution/siblings.py#L566 . But most fun is getting

$> git show git-annex:remote.log     
 autoenable=true location=http://127.0.0.1:34817/ name=fresh-sr type=git timestamp=1656698564.388750883s
640b3cbe-2ab0-488e-9dde-770f3cb13bfc autoenable=true location=http://127.0.0.1:34817/.git name=mike2 type=git timestamp=1656698511.408632757s

and things still working just fine ;) Robust sandwich we had! ;) but also I think that the test wasn't really testing that particular aspect since you can see above mike2 already having that nice full URL to that repo and even with autoenable=true before we even tried to configure that sibling with a broken URL. They both were pointing to the same thing, just that first one was configureed with correct URL. And thus test worked - we were ignoring error from annex... will test now with fresh annex to see how life is different ;)

dump of our logs with annex 10.20220322-1~ndall+1 while adding sibling with incorrect url

(Pdb) 
[DEBUG  ] Determined class of decorated function: <class 'datalad.distribution.siblings.Siblings'> 
[DEBUG  ] Resolved dataset to configure sibling: /home/yoh/.tmp/datalad_temp_test_as_common_datasource7tdbrez7 
[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'remote', 'add', 'fresh', 'http://127.0.0.1:34817/'] (cwd=/home/yoh/.tmp/datalad_temp_test_as_common_datasource7tdbrez7) 
[DEBUG  ] Run ['git', 'config', '-z', '-l', '--show-origin'] (cwd=/home/yoh/.tmp/datalad_temp_test_as_common_datasource7tdbrez7) 
[DEBUG  ] Finished ['git', 'config', '-z', '-l', '--show-origin'] with status 0 
[DEBUG  ] Run ['git', 'config', '--local', 'remote.fresh.url', 'http://127.0.0.1:34817/'] (cwd=/home/yoh/.tmp/datalad_temp_test_as_common_datasource7tdbrez7) 
[DEBUG  ] Finished ['git', 'config', '--local', 'remote.fresh.url', 'http://127.0.0.1:34817/'] with status 0 
[DEBUG  ] Run ['git', 'config', '-z', '-l', '--show-origin'] (cwd=/home/yoh/.tmp/datalad_temp_test_as_common_datasource7tdbrez7) 
[DEBUG  ] Finished ['git', 'config', '-z', '-l', '--show-origin'] with status 0 
[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'enableremote', 'fresh', '-c', 'annex.dotfiles=true'] (cwd=/home/yoh/.tmp/datalad_temp_test_as_common_datasource7tdbrez7) 
[DEBUG  ] Finished ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'enableremote', 'fresh', '-c', 'annex.dotfiles=true'] with status 1 
[INFO   ] Could not enable annex remote fresh. This is expected if fresh is a pure Git remote, or happens if it is not accessible. 
[DEBUG  ] Exception was: CommandError(CommandError: 'git -c diff.ignoreSubmodules=none annex enableremote fresh -c annex.dotfiles=true' failed with exitcode 1 under /home/yoh/.tmp/datalad_temp_test_as_common_datasource7tdbrez7 [out: 'enableremote fresh 

failed'] [err: 'Remote fresh not usable by git-annex; setting annex-ignore
  http://127.0.0.1:34817//config download failed: File not found
  Remote fresh not usable by git-annex; setting annex-ignore
  http://127.0.0.1:34817//config download failed: File not found
enableremote: 1 failed']) 
[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'initremote', 'fresh-sr', 'type=git', 'location=http://127.0.0.1:34817/', 'autoenable=true', '-c', 'annex.dotfiles=true'] (cwd=/home/yoh/.tmp/datalad_temp_test_as_common_datasource7tdbrez7) 
[DEBUG  ] Finished ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'initremote', 'fresh-sr', 'type=git', 'location=http://127.0.0.1:34817/', 'autoenable=true', '-c', 'annex.dotfiles=true'] with status 0 
[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'info', '--bytes', '--fast', '--json', '--json-error-messages', '-c', 'annex.dotfiles=true'] (cwd=/home/yoh/.tmp/datalad_temp_test_as_common_datasource7tdbrez7) 
[DEBUG  ] Finished ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'info', '--bytes', '--fast', '--json', '--json-error-messages', '-c', 'annex.dotfiles=true'] with status 0 
[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'cat-file', 'blob', 'git-annex:remote.log'] (cwd=/home/yoh/.tmp/datalad_temp_test_as_common_datasource7tdbrez7) 
[DEBUG  ] Run ['git', '-c', 'diff.ignoreSubmodules=none', 'annex', 'wanted', 'fresh', '-c', 'annex.dotfiles=true'] (cwd=/home/yoh/.tmp/datalad_temp_test_as_common_datasource7tdbrez7) 
[DEBUG  ] Run ['git', 'config', '-z', '-l', '--show-origin'] (cwd=/home/yoh/.tmp/datalad_temp_test_as_common_datasource7tdbrez7) 
[DEBUG  ] Finished ['git', 'config', '-z', '-l', '--show-origin'] with status 0 
[WARNING] Could not detect whether fresh carries an annex. If fresh is a pure Git remote, this is expected. Remote was marked by annex as annex-ignore. Edit .git/config to reset if you think that was done by mistake due to absent connection etc 
> /home/yoh/proj/datalad/datalad-maint/datalad/distribution/tests/test_siblings.py(559)test_as_common_datasource()
-> assert_status('ok', res)
(Pdb) 

[yarikoptic]

rright -- and with newer git-annex, that initremote call we do after unsuccessful enableremote now fails similarly and that is what leads to the test to error out. mystery resolved. Funny that the test errorring out is just a "side-effect". May be ideally test should also be adjusted to test that "add"ed with URL remote actually works.