Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RF: Replace custom .gitattributes handling with standard method #2912

Merged
merged 2 commits into from Oct 12, 2018
Merged

Conversation

@mih
Copy link
Member

@mih mih commented Oct 11, 2018

No description provided.

@yarikoptic
Copy link
Member

@yarikoptic yarikoptic commented Oct 11, 2018

travis was flaking out - restarted
here is the failure from one of the build bots:

======================================================================
ERROR: datalad.distribution.tests.test_add.test_add_mimetypes
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/buildbot/datalad-pr-docker-dl-nd16_04/build/datalad/tests/utils.py", line 418, in newfunc
    return t(*(arg + (d,)), **kw)
  File "/home/buildbot/datalad-pr-docker-dl-nd16_04/build/datalad/distribution/tests/test_add.py", line 384, in test_add_mimetypes
    __not_tested__ = ds.add(['file.txt', 'empty'])
  File "/home/buildbot/datalad-pr-docker-dl-nd16_04/build/venv-ci/local/lib/python2.7/site-packages/wrapt/wrappers.py", line 562, in __call__
    args, kwargs)
  File "/home/buildbot/datalad-pr-docker-dl-nd16_04/build/datalad/distribution/dataset.py", line 442, in apply_func
    return f(**kwargs)
  File "/home/buildbot/datalad-pr-docker-dl-nd16_04/build/venv-ci/local/lib/python2.7/site-packages/wrapt/wrappers.py", line 523, in __call__
    args, kwargs)
  File "/home/buildbot/datalad-pr-docker-dl-nd16_04/build/datalad/interface/utils.py", line 479, in eval_func
    return return_func(generator_func)(*args, **kwargs)
  File "/home/buildbot/datalad-pr-docker-dl-nd16_04/build/venv-ci/local/lib/python2.7/site-packages/wrapt/wrappers.py", line 523, in __call__
    args, kwargs)
  File "/home/buildbot/datalad-pr-docker-dl-nd16_04/build/datalad/interface/utils.py", line 467, in return_func
    results = list(results)
  File "/home/buildbot/datalad-pr-docker-dl-nd16_04/build/datalad/interface/utils.py", line 422, in generator_func
    result_renderer, result_xfm, _result_filter, **_kwargs):
  File "/home/buildbot/datalad-pr-docker-dl-nd16_04/build/datalad/interface/utils.py", line 491, in _process_results
    for res in results:
  File "/home/buildbot/datalad-pr-docker-dl-nd16_04/build/datalad/distribution/add.py", line 408, in __call__
    for a in added:
  File "/home/buildbot/datalad-pr-docker-dl-nd16_04/build/datalad/support/annexrepo.py", line 1567, in add_
    expect_stderr=True):
  File "/home/buildbot/datalad-pr-docker-dl-nd16_04/build/datalad/support/annexrepo.py", line 2464, in _run_annex_command_json
    raise e
CommandError: CommandError: command '['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'add', '--json', '--', u'/tmp/datalad_temp_tree_test_add_mimetypesF2_1ck/file.txt', u'/tmp/datalad_temp_tree_test_add_mimetypesF2_1ck/empty']' failed with exitcode 1
Failed to run ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'add', '--json', '--', u'/tmp/datalad_temp_tree_test_add_mimetypesF2_1ck/file.txt', u'/tmp/datalad_temp_tree_test_add_mimetypesF2_1ck/empty'] under '/tmp/datalad_temp_tree_test_add_mimetypesF2_1ck'. Exit code=1.

@mih
Copy link
Member Author

@mih mih commented Oct 11, 2018

If that failure is real, it means that GitRepo.set_gitattributes() itself may be broken.

@mih
Copy link
Member Author

@mih mih commented Oct 12, 2018

Reason for the failure is a .gitattributes files being placed in the dataset by the test that has no final newline. GitRepo.set_gitattributes() writes to this file in append-mode, hence breaks the syntax.

Otherwise a prev missing final newline will break the syntax
@codecov
Copy link

@codecov codecov bot commented Oct 12, 2018

Codecov Report

Merging #2912 into master will increase coverage by 0.03%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2912      +/-   ##
==========================================
+ Coverage   90.28%   90.32%   +0.03%     
==========================================
  Files         246      246              
  Lines       31902    31969      +67     
==========================================
+ Hits        28804    28877      +73     
+ Misses       3098     3092       -6
Impacted Files Coverage Δ
datalad/support/gitrepo.py 88.8% <100%> (ø) ⬆️
datalad/support/annexrepo.py 88.17% <100%> (-0.07%) ⬇️
datalad/plugin/wtf.py 89.47% <0%> (+6.53%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d43abf4...99bce9e. Read the comment docs.

@mih mih merged commit 012203b into master Oct 12, 2018
12 checks passed
@mih mih deleted the rf-annexbackend branch Oct 12, 2018
@yarikoptic yarikoptic added this to the Release 0.10.4 milestone Oct 22, 2018
@yarikoptic yarikoptic removed this from the Release 0.10.4 milestone Oct 22, 2018
@yarikoptic yarikoptic added this to the Release 0.10.4 milestone Oct 22, 2018
@yarikoptic yarikoptic added this to the Release 0.10.4 milestone Oct 22, 2018
@yarikoptic yarikoptic removed this from the Release 0.10.4 milestone Oct 22, 2018
@yarikoptic yarikoptic added this to the Release 0.10.4 milestone Oct 22, 2018
yarikoptic added a commit that referenced this issue Oct 24, 2018
 Upgrade of [git-annex] to the most recent available to your release is
 advisable since a number of issues were resolved at that level.

 ### Major refactoring and deprecations

 - `datalad.consts.LOCAL_CENTRAL_PATH` constant was deprecated in favor
   of `datalad.locations.default-dataset` [configuration] variable
   ([#2835])

 ### Minor refactoring

 - `"notneeded"` messages are no longer reported by default results
   renderer
 - [run] no longer shows commit instructions upon command failure when
   `explicit` is true and no outputs are specified ([#2922])
 - `get_git_dir` moved into GitRepo ([#2886])
 - `_gitpy_custom_call` removed from GitRepo ([#2894])
 - `GitRepo.get_merge_base` argument is now called `commitishes` instead
   of `treeishes` ([#2903])

 ### Fixes

 - [update] should not leave the dataset in non-clean state ([#2858])
   and some other enhancements ([#2859])
 - Fixed chunking of the long command lines to account for decorators
   and other arguments ([#2864])
 - Progress bar should not crash the process on some missing progress
   information ([#2891])
 - Default value for `jobs` set to be `"auto"` (not `None`) to take
   advantage of possible parallel get if in `-g` mode ([#2861])
 - [wtf] must not crash if `git-annex` is not installed etc ([#2865]),
   ([#2865]), ([#2918]), ([#2917])
 - Fixed paths (with spaces etc) handling while reporting annex error
   output ([#2892]), ([#2893])
 - `__del__` should not access `.repo` but `._repo` to avoid attempts
   for reinstantiation etc ([#2901])
 - Fix up submodule `.git` right in `GitRepo.add_submodule` to avoid
   added submodules being non git-annex friendly ([#2909]), ([#2904])
 - [run-procedure] ([#2905])
   - now will provide dataset into the procedure if called within dataset
   - will not crash if procedure is an executable without `.py` or `.sh`
     suffixes
 - Use centralized `.gitattributes` handling while setting annex backend
   ([#2912])
 - `GlobbedPaths.expand(..., full=True)` incorrectly returned relative
    paths when called more than once ([#2921])

 ### Enhancements and new features

 - Report progress on [clone] when installing from "smart" git servers
   ([#2876])
 - Stale/unused `sth_like_file_has_content` was removed ([#2860])
 - Enhancements to [search] to operate on "improved" metadata layouts
   ([#2878])
 - Output of `git annex init` operation is now logged ([#2881])
 - New
   - `GitRepo.cherry_pick` ([#2900])
   - `GitRepo.format_commit` ([#2902])
 - [run-procedure] ([#2905])
   - procedures can now recursively be discovered in subdatasets as well.
     The uppermost has highest priority
   - Procedures in user and system locations now take precedence over
     those in datasets.

* tag '0.11.0':
  Make it a 0.11.0 release since there were some API RFings
  REL: slight tune up to Changelog following the advices
  DOC: v0.10.4: Mention change in procedure precedence (a0cbcba)
  DOC: v0.10.4: Fix description of db715b7
  DOC: v0.10.4: Improve description of 6f615a4
  DOC: v0.10.4: Remove duplicate word
yarikoptic added a commit that referenced this issue Oct 24, 2018
 ## 0.11.0 (Oct 23, 2018) -- Soon-to-be-perfect

 [git-annex] 6.20180913 (or later) is now required - provides a number of
 fixes for v6 mode operations etc.

 ### Major refactoring and deprecations

 - `datalad.consts.LOCAL_CENTRAL_PATH` constant was deprecated in favor
   of `datalad.locations.default-dataset` [configuration] variable
   ([#2835])

 ### Minor refactoring

 - `"notneeded"` messages are no longer reported by default results
   renderer
 - [run] no longer shows commit instructions upon command failure when
   `explicit` is true and no outputs are specified ([#2922])
 - `get_git_dir` moved into GitRepo ([#2886])
 - `_gitpy_custom_call` removed from GitRepo ([#2894])
 - `GitRepo.get_merge_base` argument is now called `commitishes` instead
   of `treeishes` ([#2903])

 ### Fixes

 - [update] should not leave the dataset in non-clean state ([#2858])
   and some other enhancements ([#2859])
 - Fixed chunking of the long command lines to account for decorators
   and other arguments ([#2864])
 - Progress bar should not crash the process on some missing progress
   information ([#2891])
 - Default value for `jobs` set to be `"auto"` (not `None`) to take
   advantage of possible parallel get if in `-g` mode ([#2861])
 - [wtf] must not crash if `git-annex` is not installed etc ([#2865]),
   ([#2865]), ([#2918]), ([#2917])
 - Fixed paths (with spaces etc) handling while reporting annex error
   output ([#2892]), ([#2893])
 - `__del__` should not access `.repo` but `._repo` to avoid attempts
   for reinstantiation etc ([#2901])
 - Fix up submodule `.git` right in `GitRepo.add_submodule` to avoid
   added submodules being non git-annex friendly ([#2909]), ([#2904])
 - [run-procedure] ([#2905])
   - now will provide dataset into the procedure if called within dataset
   - will not crash if procedure is an executable without `.py` or `.sh`
     suffixes
 - Use centralized `.gitattributes` handling while setting annex backend
   ([#2912])
 - `GlobbedPaths.expand(..., full=True)` incorrectly returned relative
    paths when called more than once ([#2921])

 ### Enhancements and new features

 - Report progress on [clone] when installing from "smart" git servers
   ([#2876])
 - Stale/unused `sth_like_file_has_content` was removed ([#2860])
 - Enhancements to [search] to operate on "improved" metadata layouts
   ([#2878])
 - Output of `git annex init` operation is now logged ([#2881])
 - New
   - `GitRepo.cherry_pick` ([#2900])
   - `GitRepo.format_commit` ([#2902])
 - [run-procedure] ([#2905])
   - procedures can now recursively be discovered in subdatasets as well.
     The uppermost has highest priority
   - Procedures in user and system locations now take precedence over
     those in datasets.

* tag '0.11.0':
  CHANGELOG adjusted to reflect new minimal version of git-annex
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants