save: Update for "no commits" sub-repo fix in Git #3476
Conversation
kyleam
added
the
do not merge
Codecov Report
@@ Coverage Diff @@
## master #3476 +/- ##
==========================================
- Coverage 83.05% 66.5% -16.55%
==========================================
Files 269 269
Lines 34933 34946 +13
==========================================
- Hits 29014 23242 -5772
- Misses 5919 11704 +5785
Continue to review full report at Codecov.
|
|
Marking as "do not merge" because there's a temporary commit to enable the Travis run with the latest Git and because this needs and sits on top of gh-3470.
Looks like ppa:git-core/ppa doesn't have Git v2.22.0 yet. I tested this change locally with v2.22.0, so I'd be ok merging this without confirmation from Travis once gh-3470 is in. Will drop the temporary Travis commit.
|
As the comment says, this is about adjusted branches rather than Windows specifically, so use is_managed_branch(). (That method also returns true for a repository in direct mode, but that's not relevant for master.)
This test is about the unborn branch edge case, but we might as well check that the adjusted branch is in the state that we expect.
kyleam
added
do not merge
kyleam
force-pushed
the
save-no-commits
branch
3 times, most recently
from
dc8d436
to
de8c1c7
Compare
|
OK, since I opened this PR, ppa:git-core/ppa has been updated to include the latest Git (v2.22.0). So testing v2.22.0-specific bits of this PR on Travis should be as easy as a temporary commit that comments out the cron bit of upstream git build. But
Any ideas about what the issue is? |
kyleam
force-pushed
the
save-no-commits
branch
8 times, most recently
from
3fb9b47
to
89b9e15
Compare
|
There's a lot of noise on this PR (mostly from me trying to debug a Travis-specific failure), so let me summarize the outstanding issues:
|
Travis recently switched to using Xenial by default, and the Xenial image, unlike the Trusty one, doesn't have third-party apt-repositories [0]. Selectively enable ppa:git-core/ppa in the upstream Git build because we rely on that repository to get the latest Git. Note that we install git through "addons: apt: packages:" rather than our tools/ci/install-latest-git.sh script. This has the downside that Git is installed unnecessarily, both for builds that have _DL_CRON set and for builds where the bundled Git matches the latest upstream one. We do it because otherwise the build exits (with a status of 0) right after the installation happens [1]. The same happens if we use "dist: trusty", which should have the third-party repositories enabled [2]. [0]: https://docs.travis-ci.com/user/reference/xenial/#third-party-apt-repositories-removed [1]: https://travis-ci.org/datalad/datalad/jobs/550376442 [2]: https://travis-ci.org/datalad/datalad/builds/550835880 Re: datalad#3476
If we see an untracked directory in the get_content_info(..., untracked="all") output, we know it's a sub-repository because otherwise git would show the untracked files individually (or nothing at all in the case of an empty directory). We call add_submodule on these paths, which takes care of adding the paths to the index, there's no reason to later pass these paths to 'git add' or 'git annex add'. Most of the time that is just a noop, but it is problematic in one situation, stemming from a change in how Git treats sub-repositories without any commits checked out. Starting with v2.22.0, these are treated as any other repository, so the call to add_submodule will appropriately fail. Before v2.22.0, untracked files in the sub-repository would confusingly be added to the _parent_ repo. The problems comes when the all of the following conditions are met: DATALAD_USE_DEFAULT_GIT is set, the system git is at least v2.22.0, and the bundled git is below v2.22.0. In this case, 'git annex add empty-subrepo' will have the old and undesirable behavior of adding the file to the parent repo. Avoid this by dropping the unnecessary sub-repository paths from the 'git annex add' call.
Saving an untracked sub-repository without a commit checked out [0]
results in an ugly state: the sub-repository is _not_ added as a
submodule but its untracked files are added as blobs in the
_superdataset_. This is not usually an issue for datasets because
these get initial commits on creation, so we documented this as a
known issue rather than introducing additional checks that would come
with a performance penalty.
The core problem is a combination of 'git submodule add' accepting a
sub-repository on an unborn branch and 'git ls-files' considering such
a sub-repository a directory rather than a repository. Both issues
are fixed in Git 2.22.0 [1]. Update the documentation note and tests
accordingly.
[0] Usually this is just a repository without any commits, but it
could be repository that has commits but is currently on an unborn
branch.
[1] https://public-inbox.org/git/20190409230737.26809-1-kyle@kyleam.com/
Commits e13811189b and b22827045e.
Re: datalad#3139
kyleam
force-pushed
the
save-no-commits
branch
2 times, most recently
from
1daf394
to
1b78c01
Compare
OK, this passed on the latest upstream git run. Explanation in 4e73078 (BF: gitrepo: Don't pass added submodules to git{,-annex} add). The only failing test on the upstream git run was the webui test that was fixed by gh-3492 (which has yet to make its way to master). https://travis-ci.org/datalad/datalad/jobs/551290494 I'll drop 1b78c01 ([drop] un-cron upstream git build for testing) and from my POV this is good to go. |
kyleam
removed
the
do not merge
Travis recently switched to using Xenial by default, and the Xenial image, unlike the Trusty one, doesn't have third-party apt-repositories [0]. Selectively enable ppa:git-core/ppa in the upstream Git build because we rely on that repository to get the latest Git. [0]: https://docs.travis-ci.com/user/reference/xenial/#third-party-apt-repositories-removed Re: datalad#3476
|
Looks proper to me, filed #3500 for appveyor failure, adjusted here test passes there: so merging |
Re: #3139
Marking as "do not merge" because there's a temporary commit to enable the Travis run with the latest Git and because this needs and sits on top of gh-3470.