ENH: Prefer structured over rendered CommandError info #4157
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mih
changed the title
|
Looks better. But I am surprised that it is neither in like in other examples with clone or create (duplication of msg between ERROR log and results renderer aside)$> datalad clone ///asdlkfj
[ERROR ] Failed to clone from any candidate source URL. Encountered errors per each url were:
| - http://datasets.datalad.org/asdlkfj
CommandError: command '['git', 'clone', '--progress', 'http://datasets.datalad.org/asdlkfj', '/tmp/asdlkfj']' failed with exitcode 128
Failed to run ['git', 'clone', '--progress', 'http://datasets.datalad.org/asdlkfj', '/tmp/asdlkfj'].
stdout=
stderr=Cloning into '/tmp/asdlkfj'...
fatal: repository 'http://datasets.datalad.org/asdlkfj/' not found
- http://datasets.datalad.org/asdlkfj/.git
CommandError: command '['git', 'clone', '--progress', 'http://datasets.datalad.org/asdlkfj/.git', '/tmp/asdlkfj']' failed with exitcode 128
Failed to run ['git', 'clone', '--progress', 'http://datasets.datalad.org/asdlkfj/.git', '/tmp/asdlkfj'].
stdout=
stderr=Cloning into '/tmp/asdlkfj'...
fatal: repository 'http://datasets.datalad.org/asdlkfj/.git/' not found
[install(/tmp/asdlkfj)]
install(error): /tmp/asdlkfj (dataset) [Failed to clone from any candidate source URL. Encountered errors per each url were:
- http://datasets.datalad.org/asdlkfj
CommandError: command '['git', 'clone', '--progress', 'http://datasets.datalad.org/asdlkfj', '/tmp/asdlkfj']' failed with exitcode 128
Failed to run ['git', 'clone', '--progress', 'http://datasets.datalad.org/asdlkfj', '/tmp/asdlkfj'].
stdout=
stderr=Cloning into '/tmp/asdlkfj'...
fatal: repository 'http://datasets.datalad.org/asdlkfj/' not found
- http://datasets.datalad.org/asdlkfj/.git
CommandError: command '['git', 'clone', '--progress', 'http://datasets.datalad.org/asdlkfj/.git', '/tmp/asdlkfj']' failed with exitcode 128
Failed to run ['git', 'clone', '--progress', 'http://datasets.datalad.org/asdlkfj/.git', '/tmp/asdlkfj'].
stdout=
stderr=Cloning into '/tmp/asdlkfj'...
fatal: repository 'http://datasets.datalad.org/asdlkfj/.git/' not found
]
$> datalad create /buga
[INFO ] Creating a new annex repo at /buga
[ERROR ] [Errno 13] Permission denied: '/buga' [os.py:makedirs:221] (PermissionError) |
CommandError(msg=) was abused to obfuscate information and cwd etc. Not these are dedicated properties, and a comprehensive str representation is rendered at the latest possible stage, avoiding intermediate overhead. Turns ``` % datalad create blowup --bare [INFO ] Creating a new annex repo at /tmp/blowup CommandError: command '['git', '-c', 'annex.largefiles=nothing', 'add', '--dry-run', '-N', '--ignore-missing', '--verbose', '--', '.gitattributes']' failed with exitcode 128 Failed to run ['git', '-c', 'annex.largefiles=nothing', 'add', '--dry-run', '-N', '--ignore-missing', '--verbose', '--', '.gitattributes'] under '/tmp/blowup'. Exit code=128. out= err=fatal: this operation must be run in a work tree stdout= stderr=fatal: this operation must be run in a work tree fatal: this operation must be run in a work tree ``` into ``` % datalad create blowup --bare [INFO ] Creating a new annex repo at /tmp/blowup CommandError: 'git -c annex.largefiles=nothing add --dry-run -N --ignore-missing --verbose -- .gitattributes' failed with exitcode 128 under /tmp/blowup fatal: this operation must be run in a work tree ```
The demo that I picked isn't a regular or anticipated failure. It is a crash, hence there is nothing that sees this until the top-level catch-all. |
yeap, and I thought that we are catching all of those, unless in |
|
No idea how this change can impact the NFS test only... Same for the windows failure: |
|
indeed, eyeballed the diff and didn't spot anything obvious which could have left a process holding that path. There is a helper mode |
FWIW, NFS issue replicates locally:$> python -m nose -s -v datalad/support/tests/test_annexrepo.py:test_dropkey
datalad.support.tests.test_annexrepo.test_dropkey(False,) ... ok
datalad.support.tests.test_annexrepo.test_dropkey(True,) ... ERROR
Versions: appdirs=1.4.3 boto=2.49.0 cmd:7z=16.02 cmd:annex=7.20200204+git1-gb2c529b88-1~ndall+1 cmd:bundled-git=2.24.0 cmd:git=2.24.0 cmd:system-git=2.24.0 cmd:system-ssh=8.1p1 exifread=2.1.2 git=3.0.5 gitdb=2.0.6 humanize=0.5.1 iso8601=0.1.11 keyring=18.0.1 keyrings.alt=3.1.1 msgpack=0.5.6 mutagen=1.40.0 requests=2.21.0 wrapt=1.11.2
Obscure filename: str=b' "\';a&b&c\xce\x94\xd0\x99\xd7\xa7\xd9\x85\xe0\xb9\x97\xe3\x81\x82 `| ' repr=' "\';a&b&cΔЙקم๗あ `| '
Encodings: default='utf-8' filesystem='utf-8' locale.prefered='UTF-8'
Environment: LANG='en_US.UTF-8' GIT_PAGER='less --no-init --quit-if-one-screen' PATH='/home/yoh/gocode/bin:/home/yoh/proj/datalad/datalad-master/venvs/dev3/bin:/home/yoh/gocode/bin:/home/yoh/gocode/bin:/home/yoh/bin:/home/yoh/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/sbin:/usr/sbin:/usr/local/sbin:/home/yoh/.mutt/soft' GIT_PYTHON_GIT_EXECUTABLE='/usr/lib/git-annex.linux/git'
======================================================================
ERROR: datalad.support.tests.test_annexrepo.test_dropkey(True,)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/home/yoh/proj/datalad/datalad-master/datalad/tests/utils.py", line 588, in newfunc
rmtemp(d)
File "/home/yoh/proj/datalad/datalad-master/datalad/utils.py", line 521, in rmtemp
rmtree(f, *args, **kwargs)
File "/home/yoh/proj/datalad/datalad-master/datalad/utils.py", line 434, in rmtree
_rmtree(path, *args, **kwargs)
File "/home/yoh/proj/datalad/datalad-master/datalad/utils.py", line 1882, in wrapped
return f(*args, **kwargs)
File "/home/yoh/proj/datalad/datalad-master/datalad/utils.py", line 1922, in _rmtree
return shutil.rmtree(*args, **kwargs)
File "/home/yoh/proj/datalad/datalad-master/venvs/dev3/lib/python3.7/shutil.py", line 494, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/home/yoh/proj/datalad/datalad-master/venvs/dev3/lib/python3.7/shutil.py", line 432, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/home/yoh/proj/datalad/datalad-master/venvs/dev3/lib/python3.7/shutil.py", line 432, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/home/yoh/proj/datalad/datalad-master/venvs/dev3/lib/python3.7/shutil.py", line 432, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/home/yoh/proj/datalad/datalad-master/venvs/dev3/lib/python3.7/shutil.py", line 452, in _rmtree_safe_fd
onerror(os.unlink, fullname, sys.exc_info())
File "/home/yoh/proj/datalad/datalad-master/venvs/dev3/lib/python3.7/shutil.py", line 450, in _rmtree_safe_fd
os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '.nfs0000000000fa337f00000013' |
|
hm, but not again! right after initial attempt, started to redo and failed to replicate... heh heh |
|
oh, and there is already in the stack: which comes from |
…ibly avoid clinging on open files
Codecov Report
@@ Coverage Diff @@
## master #4157 +/- ##
=======================================
Coverage 87.56% 87.56%
=======================================
Files 279 279
Lines 39131 39131
=======================================
Hits 34267 34267
Misses 4864 4864Continue to review full report at Codecov.
|
|
adb7d92 didn't help Windows, so I guess someone needs to figure it out properly :-/ |
|
Windows tests are generally in a bad place right now. Will dig on a more global scale ... |
Merged
kyleam
added a commit
to kyleam/datalad
that referenced
this issue
Sep 25, 2020
09d3fd8 (ENH: Prefer structured over rendered info, 2020-02-16, dataladgh-4157) switched datalad.cmdline.main over to using to_str() for rendering a caught CommandError. None of the CommandError subclasses have a to_str() method, though, so the value returned by CommandError.to_str() is displayed rather than the value returned by the child's __str__() method. For example, now an OutOfSpaceError is shown as "OutOfSpaceError: " rather than "OutOfSpaceError: needs N more". Likewise, calling `datalad get --source=DO-NOT-EXIST` no longer shows the remote name, just RemoteNotAvailableError: 'annex get' Convert the __str__() methods of all subclasses to to_str(). The message above becomes Remote 'DO-NOT-EXIST' is not available. Command failed: RemoteNotAvailableError: 'annex get' This commit just intends to restore the displayed information that was lost with 09d3fd8. However, the rendering of these CommandError subclasses could arguably be improved (e.g., the "Command failed: RemoteNotAvailableError: 'annex get'" stacking is confusing and could probably be dropped altogether).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
CommandError(msg=) was abused to obfuscate information on
cwd etc. Now these are dedicated properties, and a comprehensive
str representation is rendered at the latest possible stage,
avoiding intermediate overhead.
Turns
into