Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prepare for git-annex's removal of direct mode #3631

Merged
merged 6 commits into from Sep 2, 2019

Conversation

@kyleam
Copy link
Member

commented Aug 30, 2019

This series updates annexrepo.py and the test suite for git-annex's removal of direct mode (gh-3627). It resolves issues that I've caught from running the test suite locally with a git-annex built from d6e1f09ed, but we'd of course want a run on Travis before merging this.

kyleam added 6 commits Aug 29, 2019
TST+RF: test_annexrepo: Split up a test
test_annex_version_handling patches our information on git annex so
that its version matches the minimum requirement, its version is
outdated, or annex is unavailable entirely.

The first scenario will interfere with the logic in
AnnexRepo.__init__() that decides whether the git-annex version
supports direct mode, so we need to skip the test if we're running a
git-annex version that doesn't support direct mode and
datalad.repo.direct is true.  Split this part off so that we can more
easily skip it without skipping the other scenarios.
BF+TST: Avoid leaking patched git-annex version
Several of the tests patch the git-annex version.  One of these,
test_annex_version_comparison, leaks state because it doesn't return
AnnexRepo.git_annex_version back to its original value.

Add a helper that takes care of this, and use it in all the spots that
temporarily override the git-annex version.
ENH: annexrepo: Add check_direct_mode_support method
Using a class method here is a bit ugly.  The main motivation of doing
this---instead of adding a property or regular method to AnnexRepo or
adding a top-level helper to annexrepo.py---is to allow the method to
be called without instantiating the class while still sharing this
information with instantiated objects.
ENH: annexrepo: Refuse to initialize in direct mode if unsupported
When the caller instructs us to use direct mode and git-annex doesn't
support it, issue a warning and ignore the request.
TST: Update tests for a git-annex without direct mode
git-annex dropped direct mode after version 7.20190819.  Skip tests
that rely on it being supported.

Closes #3627.
@yarikoptic

This comment has been minimized.

Copy link
Member

commented Sep 1, 2019

Looks good to me! Anything else left todo to take it out of Draft mode? I will add it to upcoming 0.11.7 milestone so datalad could be ready to "accept" new git annex in *debians

@yarikoptic yarikoptic added this to the Release 0.11.7 milestone Sep 1, 2019

@kyleam

This comment has been minimized.

Copy link
Member Author

commented Sep 1, 2019

Anything else left todo to take it out of Draft mode?

I'll take it out of draft mode now, but I think we still need a Travis build with an unreleased annex.

@kyleam kyleam marked this pull request as ready for review Sep 1, 2019

@yarikoptic

This comment has been minimized.

Copy link
Member

commented Sep 1, 2019

I was building a fresh snapshot, will upload later to NeuroDebian devel, we could test this PR against it. I will push a temp commit for that later today

@codecov

This comment has been minimized.

Copy link

commented Sep 1, 2019

Codecov Report

Merging #3631 into 0.11.x will increase coverage by 23.36%.
The diff coverage is 96.61%.

Impacted file tree graph

@@             Coverage Diff             @@
##           0.11.x    #3631       +/-   ##
===========================================
+ Coverage   57.66%   81.02%   +23.36%     
===========================================
  Files          94      256      +162     
  Lines       15083    33882    +18799     
===========================================
+ Hits         8697    27454    +18757     
- Misses       6386     6428       +42
Impacted Files Coverage Δ
datalad/tests/test_direct_mode.py 30.35% <100%> (ø)
datalad/tests/utils.py 89.12% <100%> (ø)
datalad/support/tests/test_annexrepo.py 94.31% <100%> (ø)
datalad/support/tests/test_external_versions.py 92.05% <100%> (ø)
datalad/support/annexrepo.py 57.66% <77.77%> (+12.05%) ⬆️
datalad/downloaders/tests/utils.py 92.3% <0%> (ø)
datalad/customremotes/tests/__init__.py 91.66% <0%> (ø)
datalad/cmdline/tests/test_helpers.py 100% <0%> (ø)
datalad/support/tests/test_stats.py 100% <0%> (ø)
datalad/tests/test_utils.py 96.51% <0%> (ø)
... and 201 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a2d1bd4...542b677. Read the comment docs.

@codecov

This comment has been minimized.

Copy link

commented Sep 1, 2019

Codecov Report

Merging #3631 into 0.11.x will increase coverage by 19.41%.
The diff coverage is 96.61%.

Impacted file tree graph

@@             Coverage Diff             @@
##           0.11.x    #3631       +/-   ##
===========================================
+ Coverage   57.66%   77.07%   +19.41%     
===========================================
  Files          94      256      +162     
  Lines       15083    33882    +18799     
===========================================
+ Hits         8697    26114    +17417     
- Misses       6386     7768     +1382
Impacted Files Coverage Δ
datalad/tests/utils.py 88.72% <100%> (ø)
datalad/tests/test_direct_mode.py 30.35% <100%> (ø)
datalad/support/tests/test_annexrepo.py 93.21% <100%> (ø)
datalad/support/tests/test_external_versions.py 92.05% <100%> (ø)
datalad/support/annexrepo.py 51.28% <77.77%> (+5.67%) ⬆️
datalad/support/keyring_.py 35.55% <0%> (-48.89%) ⬇️
datalad/distribution/create_sibling_github.py 36.06% <0%> (-47.55%) ⬇️
datalad/ui/base.py 50% <0%> (-45.46%) ⬇️
datalad/interface/unlock.py 34.56% <0%> (-43.21%) ⬇️
datalad/distribution/siblings.py 14.46% <0%> (-39.7%) ⬇️
... and 222 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a2d1bd4...e2f918b. Read the comment docs.

@kyleam

This comment has been minimized.

Copy link
Member Author

commented Sep 2, 2019

I was building a fresh snapshot, will upload later to NeuroDebian devel, we could test this PR against it. I will push a temp commit for that later today

Thank you. All green: https://travis-ci.org/datalad/datalad/builds/579444778

I'll drop the tip commit and merge.

@kyleam kyleam force-pushed the kyleam:directless-annex branch from 542b677 to e2f918b Sep 2, 2019

kyleam added a commit that referenced this pull request Sep 2, 2019

@kyleam kyleam merged commit e2f918b into datalad:0.11.x Sep 2, 2019

4 of 5 checks passed

continuous-integration/appveyor/pr Waiting for AppVeyor build to complete
Details
WIP Ready for review
Details
codecov/patch 96.61% of diff hit (target 57.66%)
Details
codecov/project 81.02% (+23.36%) compared to a2d1bd4
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@kyleam kyleam deleted the kyleam:directless-annex branch Sep 2, 2019

yarikoptic added a commit that referenced this pull request Sep 6, 2019
Merge tag '0.11.7' into debian
0.11.7 (Sep 02, 2019) -- python2-we-still-love-you-but-...

Primarily bugfixes with some optimizations and refactorings.

 Fixes

- [addurls][]
  - now provides better handling when the URL file isn't in the
    expected format.  ([#3579][])
  - always considered a relative file for the URL file argument as
    relative to the current working directory, which goes against the
    convention used by other commands of taking relative paths as
    relative to the dataset argument.  ([#3582][])

- [run-procedure][]
  - hard coded "python" when formatting the command for non-executable
    procedures ending with ".py".  `sys.executable` is now used.
    ([#3624][])
  - failed if arguments needed more complicated quoting than simply
    surrounding the value with double quotes.  This has been resolved
    for systems that support `shlex.quote`, but note that on Windows
    values are left unquoted. ([#3626][])

- [siblings][] now displays an informative error message if a local
  path is given to `--url` but `--name` isn't specified.  ([#3555][])

- [sshrun][], the command DataLad uses for `GIT_SSH_COMMAND`, didn't
  support all the parameters that Git expects it to.  ([#3616][])

- Fixed a number of Unicode py2-compatibility issues. ([#3597][])

 Enhancements and new features

- The [annotate-paths][] helper now caches subdatasets it has seen to
  avoid unnecessary calls.  ([#3570][])

- A repeated configuration query has been dropped from the handling of
  `--proc-pre` and `--proc-post`.  ([#3576][])

- Calls to `git annex find` now use `--in=.` instead of the alias
  `--in=here` to take advantage of an optimization that git-annex (as
  of the current release, 7.20190730) applies only to the
  former. ([#3574][])

- [addurls][] now suggests close matches when the URL or file format
  contains an unknown field.  ([#3594][])

- Shared logic used in the setup.py files of Datalad and its
  extensions has been moved to modules in the _datalad_build_support/
  directory.  ([#3600][])

- Get ready for upcoming git-annex dropping support for direct mode
  ([#3631][])

* tag '0.11.7': (87 commits)
  DOC: Added an entry to changelogn on merged 3631
  ENH: finalizing changelog for 0.11.7
  TST: Update tests for a git-annex without direct mode
  TST: utils: Add decorator that skips when direct mode is unsupported
  ENH: annexrepo: Refuse to initialize in direct mode if unsupported
  ENH: annexrepo: Add check_direct_mode_support method
  BF+TST: Avoid leaking patched git-annex version
  TST+RF: test_annexrepo: Split up a test
  CHANGELOG.md: Second batch for 0.11.7
  TST: run_procedure: Mark test_spaces() as known Windows failure
  TST: run_procedure: Mark test_quoting as known windows failure
  TST: run_procedure: Test more arguments that need quoting
  BF(py2): run_procedure: Avoid encoding error in log message
  TST: add run_procedure test with spaces in file name
  TST/RF: non-hardcoded Python executable
  RF: newline at end of file
  RF: helper instead of conditional
  RF: remove superfluous imports
  BF/TST: remove quoting
  ENH: replace conditionals with helper function
  ...
yarikoptic added a commit that referenced this pull request Sep 6, 2019
Merge tag '0.11.7' into debian
0.11.7 (Sep 02, 2019) -- python2-we-still-love-you-but-...

Primarily bugfixes with some optimizations and refactorings.

 Fixes

- [addurls][]
  - now provides better handling when the URL file isn't in the
    expected format.  ([#3579][])
  - always considered a relative file for the URL file argument as
    relative to the current working directory, which goes against the
    convention used by other commands of taking relative paths as
    relative to the dataset argument.  ([#3582][])

- [run-procedure][]
  - hard coded "python" when formatting the command for non-executable
    procedures ending with ".py".  `sys.executable` is now used.
    ([#3624][])
  - failed if arguments needed more complicated quoting than simply
    surrounding the value with double quotes.  This has been resolved
    for systems that support `shlex.quote`, but note that on Windows
    values are left unquoted. ([#3626][])

- [siblings][] now displays an informative error message if a local
  path is given to `--url` but `--name` isn't specified.  ([#3555][])

- [sshrun][], the command DataLad uses for `GIT_SSH_COMMAND`, didn't
  support all the parameters that Git expects it to.  ([#3616][])

- Fixed a number of Unicode py2-compatibility issues. ([#3597][])

- [download-url][] now will create leading directories of the output path
  if they do not exist ([#3646][])

 Enhancements and new features

- The [annotate-paths][] helper now caches subdatasets it has seen to
  avoid unnecessary calls.  ([#3570][])

- A repeated configuration query has been dropped from the handling of
  `--proc-pre` and `--proc-post`.  ([#3576][])

- Calls to `git annex find` now use `--in=.` instead of the alias
  `--in=here` to take advantage of an optimization that git-annex (as
  of the current release, 7.20190730) applies only to the
  former. ([#3574][])

- [addurls][] now suggests close matches when the URL or file format
  contains an unknown field.  ([#3594][])

- Shared logic used in the setup.py files of Datalad and its
  extensions has been moved to modules in the _datalad_build_support/
  directory.  ([#3600][])

- Get ready for upcoming git-annex dropping support for direct mode
  ([#3631][])

* tag '0.11.7':
  Changelog entry for download-url paths handling
  ENH: downloaders: Ensure directories for target exist
kyleam added a commit to kyleam/datalad that referenced this pull request Sep 6, 2019
ENH: annexrepo: Add check_direct_mode_support method, again
This is a partial cherry pick of 66cf13b (ENH: annexrepo: Add
check_direct_mode_support method, 2019-08-29).  When the 0.11.x
series (dataladgh-3631) that contained it was merged in a19e5c8, I dropped
the method because it had no callers.  But it turns out that I must've
botched my local testing with the unreleased annex and part of
test_direct_cfg does needs to be skipped.

0.11.x calls check_direct_mode_support() in __init__() to ensure that
the supports_direct_mode attribute is set to a non-None value, but
don't bother doing that here given that the only place in master that
needs to check direct mode support is one test.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.