New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BF: run_procedure: do not hardcode python executable (fixes gh-3623) #3624
Conversation
datalad/interface/run_procedure.py
Outdated
@@ -189,7 +190,7 @@ def _guess_exec(script_file): | |||
'state': state} | |||
elif script_file.endswith('.py'): | |||
return {'type': u'python_script', | |||
'template': u'python "{script}" "{ds}" {args}', | |||
'template': u'%s "{script}" "{ds}" {args}' % sys.executable, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I suppose we should shlex_quote() sys.executable to guard against poorly named virtual environments with spaces.
This is outside the scope of this PR, but the quoting in these templates looks problematic. I think we should drop the quotes around the placeholders and pass each value through shlex_quote().
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point, thx!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kyleam I think I took care of this. I will open a new issue about
This is outside the scope of this PR, but the quoting in these templates looks problematic. I think we should drop the quotes around the placeholders and pass each value through shlex_quote().
and will try to PR a fix.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Windows is not happy. There is no shlex for windows it seems, and https://stackoverflow.com/questions/33560364/python-windows-parsing-command-lines-with-shlex suggests that doing it correctly is a project on its own. I'd say
from datalad.utils import on_windows
sys.executable if on_windows else shlex_quote(sys.executable)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Windows is not happy. There is no shlex for windows it seems
Eh, that means we need to deal with this in other spots that use shlex_quote.
Thanks for the updates, @adswa. This would not be a straightforward test for the test suite (and I'm not suggesting you add one), but I thought I should at least see that it works locally if I create a virtual environment with spaces. But I couldn't even get to that point because other things upstream (some outside of DataLad) were failing.
Anyway, outside of windows, shlex_quote at least gives us a better chance of working with weird names and doesn't hurt for non-weird names, so it seems worth keeping.
I'm going to rebase this branch onto 0.11.x because the sys.executable change is worth having there.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm going to rebase this branch onto 0.11.x because the sys.executable change is worth having there.
Done, with a minor spacing tweak to the last commit:
1: 97acf5f04 = 1: dacb63257 BF: run_procedure: do not hardcode python executable (fixes gh-3623)
2: 94391e1f6 = 2: b48290564 shlex quote sys.executable
3: dc3628311 ! 3: ece364ecc no shlexquote on windows
@@ datalad/interface/run_procedure.py: def _guess_exec(script_file):
'state': state}
else:
return {'type': None, 'template': None, 'state': None}
-
--
- @build_doc
- class RunProcedure(Interface):
- """Run prepared procedures (DataLad scripts) on a dataset
Codecov Report
@@ Coverage Diff @@
## 0.11.x #3624 +/- ##
==========================================
- Coverage 81.23% 70.44% -10.8%
==========================================
Files 256 273 +17
Lines 33819 35511 +1692
==========================================
- Hits 27473 25015 -2458
- Misses 6346 10496 +4150
Continue to review full report at Codecov.
|
Thx @adswa and welcome! |
…-3623) Stay within the same Python version to avoid crash or undesired behavior when PY2 is called from PY3 with missing or outdated PY2 versions of datalad.
Hey, @kyleam this fix is needed for a 0.12 deployment tomorrow. Would be good to be able to pull that from its final resting place in master. |
You'll be able to. I'll merge 0.11.x into master. |
Perfect, thx! |
This test fails on AppVeyor [0]. run_procedure() uses a no-op shlex_quote() on Windows [1], so this is expected. [0]: https://ci.appveyor.com/project/mih/datalad/builds/27022010/job/jq3lr977ht5vrmpj#L1109 [1]: datalad#3624 (comment)
run_procedure() can't use shlex_quote() on Windows [0], so an argument with spaces will not be handled correctly [1]. We should probably teach the windows branch of maybe_shlex_quote() to handle simple cases like a name with spaces. [0]: datalad#3624 (comment) [1]: https://ci.appveyor.com/project/mih/datalad/builds/27022362/job/xoum5f43c135bfgd#L1125
0.11.7 (Sep 02, 2019) -- python2-we-still-love-you-but-... Primarily bugfixes with some optimizations and refactorings. Fixes - [addurls][] - now provides better handling when the URL file isn't in the expected format. ([#3579][]) - always considered a relative file for the URL file argument as relative to the current working directory, which goes against the convention used by other commands of taking relative paths as relative to the dataset argument. ([#3582][]) - [run-procedure][] - hard coded "python" when formatting the command for non-executable procedures ending with ".py". `sys.executable` is now used. ([#3624][]) - failed if arguments needed more complicated quoting than simply surrounding the value with double quotes. This has been resolved for systems that support `shlex.quote`, but note that on Windows values are left unquoted. ([#3626][]) - [siblings][] now displays an informative error message if a local path is given to `--url` but `--name` isn't specified. ([#3555][]) - [sshrun][], the command DataLad uses for `GIT_SSH_COMMAND`, didn't support all the parameters that Git expects it to. ([#3616][]) - Fixed a number of Unicode py2-compatibility issues. ([#3597][]) Enhancements and new features - The [annotate-paths][] helper now caches subdatasets it has seen to avoid unnecessary calls. ([#3570][]) - A repeated configuration query has been dropped from the handling of `--proc-pre` and `--proc-post`. ([#3576][]) - Calls to `git annex find` now use `--in=.` instead of the alias `--in=here` to take advantage of an optimization that git-annex (as of the current release, 7.20190730) applies only to the former. ([#3574][]) - [addurls][] now suggests close matches when the URL or file format contains an unknown field. ([#3594][]) - Shared logic used in the setup.py files of Datalad and its extensions has been moved to modules in the _datalad_build_support/ directory. ([#3600][]) - Get ready for upcoming git-annex dropping support for direct mode ([#3631][]) * tag '0.11.7': (87 commits) DOC: Added an entry to changelogn on merged 3631 ENH: finalizing changelog for 0.11.7 TST: Update tests for a git-annex without direct mode TST: utils: Add decorator that skips when direct mode is unsupported ENH: annexrepo: Refuse to initialize in direct mode if unsupported ENH: annexrepo: Add check_direct_mode_support method BF+TST: Avoid leaking patched git-annex version TST+RF: test_annexrepo: Split up a test CHANGELOG.md: Second batch for 0.11.7 TST: run_procedure: Mark test_spaces() as known Windows failure TST: run_procedure: Mark test_quoting as known windows failure TST: run_procedure: Test more arguments that need quoting BF(py2): run_procedure: Avoid encoding error in log message TST: add run_procedure test with spaces in file name TST/RF: non-hardcoded Python executable RF: newline at end of file RF: helper instead of conditional RF: remove superfluous imports BF/TST: remove quoting ENH: replace conditionals with helper function ...
0.11.7 (Sep 02, 2019) -- python2-we-still-love-you-but-... Primarily bugfixes with some optimizations and refactorings. Fixes - [addurls][] - now provides better handling when the URL file isn't in the expected format. ([#3579][]) - always considered a relative file for the URL file argument as relative to the current working directory, which goes against the convention used by other commands of taking relative paths as relative to the dataset argument. ([#3582][]) - [run-procedure][] - hard coded "python" when formatting the command for non-executable procedures ending with ".py". `sys.executable` is now used. ([#3624][]) - failed if arguments needed more complicated quoting than simply surrounding the value with double quotes. This has been resolved for systems that support `shlex.quote`, but note that on Windows values are left unquoted. ([#3626][]) - [siblings][] now displays an informative error message if a local path is given to `--url` but `--name` isn't specified. ([#3555][]) - [sshrun][], the command DataLad uses for `GIT_SSH_COMMAND`, didn't support all the parameters that Git expects it to. ([#3616][]) - Fixed a number of Unicode py2-compatibility issues. ([#3597][]) - [download-url][] now will create leading directories of the output path if they do not exist ([#3646][]) Enhancements and new features - The [annotate-paths][] helper now caches subdatasets it has seen to avoid unnecessary calls. ([#3570][]) - A repeated configuration query has been dropped from the handling of `--proc-pre` and `--proc-post`. ([#3576][]) - Calls to `git annex find` now use `--in=.` instead of the alias `--in=here` to take advantage of an optimization that git-annex (as of the current release, 7.20190730) applies only to the former. ([#3574][]) - [addurls][] now suggests close matches when the URL or file format contains an unknown field. ([#3594][]) - Shared logic used in the setup.py files of Datalad and its extensions has been moved to modules in the _datalad_build_support/ directory. ([#3600][]) - Get ready for upcoming git-annex dropping support for direct mode ([#3631][]) * tag '0.11.7': Changelog entry for download-url paths handling ENH: downloaders: Ensure directories for target exist
Stay within the same Python version to avoid crash or undesired behavior
when PY2 is called from PY3 with missing or outdated PY2 versions of
datalad.