TST: set --allow-unrelated-histories in the mk_push_target setup for Windows #5855

adswa · 2021-08-02T09:06:49Z

The test helper mk_push_target creates a sibling dataset by
creating an unrelated dataset and adding it as a sibling. On
Windows, this was not straightforward, as the datasets had
unrelated histories due to the initial adjusted branch commit
in the target dataset. Until git-annex version 20210720, a git
annex sync would establish a shared history - more recent versions
of git-annex will fail to sync unrelated histories unless
--allow-unrelated-histories is set.

With this patch, the tests that started to fail on Windows in #5811 pass again.

Fixes #5811.

codecov · 2021-08-02T09:16:43Z

Codecov Report

Merging #5855 (c59d3db) into maint (7f543d0) will decrease coverage by 60.74%.
The diff coverage is 32.27%.

❗ Current head c59d3db differs from pull request most recent head e21dde0. Consider uploading reports for the commit e21dde0 to get more accurate results

@@             Coverage Diff             @@
##            maint    #5855       +/-   ##
===========================================
- Coverage   90.28%   29.54%   -60.75%     
===========================================
  Files         300      297        -3     
  Lines       42370    42338       -32     
===========================================
- Hits        38252    12507    -25745     
- Misses       4118    29831    +25713

Impacted Files	Coverage Δ
datalad/__main__.py	`0.00% <0.00%> (-75.00%)`	⬇️
datalad/cmdline/tests/test_helpers.py	`0.00% <0.00%> (-100.00%)`	⬇️
datalad/cmdline/tests/test_main.py	`0.00% <0.00%> (-98.62%)`	⬇️
datalad/config.py	`69.69% <0.00%> (-27.03%)`	⬇️
datalad/consts.py	`96.96% <ø> (-3.04%)`	⬇️
datalad/core/distributed/tests/test_clone.py	`0.00% <0.00%> (-97.24%)`	⬇️
datalad/core/distributed/tests/test_push.py	`0.00% <0.00%> (-99.12%)`	⬇️
datalad/core/local/tests/test_create.py	`0.00% <0.00%> (-100.00%)`	⬇️
datalad/core/local/tests/test_diff.py	`0.00% <0.00%> (-99.54%)`	⬇️
datalad/core/local/tests/test_save.py	`0.00% <0.00%> (-96.71%)`	⬇️
... and 395 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7f543d0...e21dde0. Read the comment docs.

adswa · 2021-08-02T11:25:46Z

The test failure looks unrelated to me:

======================================================================
ERROR: datalad.tests.test_utils_cached_dataset.test_get_cached_dataset
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 189, in _wrap_skip_if_no_network
    return func(*args, **kwargs)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils.py", line 739, in _wrap_with_tempfile
    return t(*(arg + (filename,)), **kw)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/test_utils_cached_dataset.py", line 120, in test_get_cached_dataset
    ds = get_cached_dataset(url, version, keys)
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/tests/utils_cached_dataset.py", line 117, in get_cached_dataset
    for c in ds.repo.fetch(DEFAULT_REMOTE)):
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/support/gitrepo.py", line 1897, in fetch
    return list(
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/support/gitrepo.py", line 1908, in fetch_
    yield from self._fetch_push_helper(
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/support/gitrepo.py", line 2081, in _fetch_push_helper
    out = self._git_runner.run(
  File "/home/appveyor/dlvenv/lib/python3.8/site-packages/datalad/cmd.py", line 168, in run
    raise CommandError(
datalad.support.exceptions.CommandError: CommandError: 'git -c diff.ignoreSubmodules=none fetch --verbose --progress dl-test-remote' failed with exitcode 128 under /home/appveyor/DLTMP/datalad_temp_test_get_cached_datasetdvfszxbw/https___github.com_datalad_testrepo--minimalds [err: 'fatal: unable to access 'https://github.com/datalad/testrepo--minimalds/': Failed to connect to github.com port 443: Connection timed out']

yarikoptic · 2021-08-02T14:12:02Z

I wonder if such a special treatment would only hide an issue which otherwise would eventually hit the users. I still think it should be cleared up with @joeyh since I still feel that it might be just a regression and will be fixed on annex side.

Some jobs stalled in our tests (not a good sign :-/ stalls happened recently in other PRs as well), restarted a 12259.1 on travis

yarikoptic · 2021-08-02T14:17:18Z

oh, just spotted the comment you @adswa left regarding git-annex change http://source.git-annex.branchable.com/?p=source.git;a=commit;h=3d50b47dede6e4e94232718f4c460cd19147dcfb which exposes that option within git-annex and thus may be the change in behavior is intended. But then it seems it would be for us in probably create-sibling*s to handle/do whenever we do an initial sync call, not just in the tests

adswa · 2021-08-02T14:20:57Z

I wonder if such a special treatment would only hide an issue which otherwise would eventually hit the users

I wondered this, too. But this test setup is arguably a quite special case - creating a new dataset and adding it as a sibling to an existing dataset. And it is only on Windows where this involves unifying two unrelated histories. Maybe I don't have enough imagination, but I would think that a typical workflow would rather involve a clone with sibling/remote add or a create-sibling?

adswa · 2021-08-02T14:22:12Z

But then it seems it would be for us in probably create-sibling*s to handle/do whenever we do an initial sync call, not just in the tests

Yes, this may well be. But this PR here just wants to make this test helper functional again

yarikoptic · 2021-08-02T14:26:39Z

And it is only on Windows where this involves unifying two unrelated histories.

isn't it for any remote which would be on a crippled FS (external usb stick), thus adjusted branches etc?
thought to check -- but git-annex seems to fail its own tests ATM on a regular and crippled FS on linux https://github.com/datalad/git-annex/actions/runs/1088733834 (but passes with crippled HOME, odd)

adswa · 2021-08-02T14:27:32Z

isn't it for any remote which would be on a crippled FS (external usb stick), thus adjusted branches etc?

oh yes, probably! Sorry, I keep forgetting about crippled FSs

mih · 2021-08-02T15:38:48Z

... But then it seems it would be for us in probably create-sibling*s to handle/do whenever we do an initial sync call, not just in the tests

Is create-sibling now functional on windows? The reason for these test helpers was that datalad is not capable to do this on windows. I am all in favor of fixing the helpers, and leaving bigger plans aside.

Filed #5857 to make things a bit more obvious.

…Windows The test helper mk_push_target creates a sibling dataset by creating an unrelated dataset and adding it as a sibling. On Windows, this was not straightforward, as the datasets had unrelated histories due to the initial adjusted branch commit in the target dataset. Until git-annex version 20210720, a git annex sync would establish a shared history - more recent versions of git-annex will fail to sync unrelated histories unless --allow-unrelated-histories is set.

adswa · 2021-08-03T14:05:15Z

rebased to maint

mih

This looks good to me. Thx!

mih · 2021-08-03T15:03:28Z

The failing linting is not the fault of this PR.

adswa requested a review from yarikoptic August 2, 2021 09:07

adswa added the semver-internal Changes only affect the internal API label Aug 2, 2021

adswa added platform-windows Issue concerned with Windows tests Add or improve existing tests labels Aug 2, 2021

mih mentioned this pull request Aug 2, 2021

create-sibling not functional/tested on windows #5857

Closed

adswa force-pushed the tst-pushtarget branch from 91469d7 to e21dde0 Compare August 3, 2021 14:04

adswa changed the base branch from master to maint August 3, 2021 14:04

mih approved these changes Aug 3, 2021

View reviewed changes

mih merged commit 85e2e96 into datalad:maint Aug 3, 2021

adswa deleted the tst-pushtarget branch August 3, 2021 15:10

yarikoptic removed the semver-internal Changes only affect the internal API label Aug 3, 2021

mih mentioned this pull request Oct 5, 2021

a number of push tests (4?) started to fail at 20210720 on windows with some recent annex changes #5811

Closed

TST: set --allow-unrelated-histories in the mk_push_target setup for Windows #5855

TST: set --allow-unrelated-histories in the mk_push_target setup for Windows #5855

adswa commented Aug 2, 2021

codecov bot commented Aug 2, 2021 •

edited

adswa commented Aug 2, 2021

yarikoptic commented Aug 2, 2021

yarikoptic commented Aug 2, 2021

adswa commented Aug 2, 2021

adswa commented Aug 2, 2021

yarikoptic commented Aug 2, 2021

adswa commented Aug 2, 2021

mih commented Aug 2, 2021 •

edited

adswa commented Aug 3, 2021

mih left a comment

mih commented Aug 3, 2021

TST: set --allow-unrelated-histories in the mk_push_target setup for Windows #5855

TST: set --allow-unrelated-histories in the mk_push_target setup for Windows #5855

Conversation

adswa commented Aug 2, 2021

codecov bot commented Aug 2, 2021 • edited

Codecov Report

adswa commented Aug 2, 2021

yarikoptic commented Aug 2, 2021

yarikoptic commented Aug 2, 2021

adswa commented Aug 2, 2021

adswa commented Aug 2, 2021

yarikoptic commented Aug 2, 2021

adswa commented Aug 2, 2021

mih commented Aug 2, 2021 • edited

adswa commented Aug 3, 2021

mih left a comment

Choose a reason for hiding this comment

mih commented Aug 3, 2021

codecov bot commented Aug 2, 2021 •

edited

mih commented Aug 2, 2021 •

edited