BF: run_procedure: do not hardcode python executable (fixes gh-3623) #3624
Conversation
datalad/interface/run_procedure.py
Outdated
| @@ -189,7 +190,7 @@ def _guess_exec(script_file): | |||
| 'state': state} | |||
| elif script_file.endswith('.py'): | |||
| return {'type': u'python_script', | |||
| 'template': u'python "{script}" "{ds}" {args}', | |||
| 'template': u'%s "{script}" "{ds}" {args}' % sys.executable, | |||
There was a problem hiding this comment.
I suppose we should shlex_quote() sys.executable to guard against poorly named virtual environments with spaces.
This is outside the scope of this PR, but the quoting in these templates looks problematic. I think we should drop the quotes around the placeholders and pass each value through shlex_quote().
There was a problem hiding this comment.
@kyleam I think I took care of this. I will open a new issue about
This is outside the scope of this PR, but the quoting in these templates looks problematic. I think we should drop the quotes around the placeholders and pass each value through shlex_quote().
and will try to PR a fix.
There was a problem hiding this comment.
Windows is not happy. There is no shlex for windows it seems, and https://stackoverflow.com/questions/33560364/python-windows-parsing-command-lines-with-shlex suggests that doing it correctly is a project on its own. I'd say
from datalad.utils import on_windows
sys.executable if on_windows else shlex_quote(sys.executable)
There was a problem hiding this comment.
Windows is not happy. There is no shlex for windows it seems
Eh, that means we need to deal with this in other spots that use shlex_quote.
Thanks for the updates, @adswa. This would not be a straightforward test for the test suite (and I'm not suggesting you add one), but I thought I should at least see that it works locally if I create a virtual environment with spaces. But I couldn't even get to that point because other things upstream (some outside of DataLad) were failing.
Anyway, outside of windows, shlex_quote at least gives us a better chance of working with weird names and doesn't hurt for non-weird names, so it seems worth keeping.
I'm going to rebase this branch onto 0.11.x because the sys.executable change is worth having there.
There was a problem hiding this comment.
I'm going to rebase this branch onto 0.11.x because the sys.executable change is worth having there.
Done, with a minor spacing tweak to the last commit:
1: 97acf5f04 = 1: dacb63257 BF: run_procedure: do not hardcode python executable (fixes gh-3623)
2: 94391e1f6 = 2: b48290564 shlex quote sys.executable
3: dc3628311 ! 3: ece364ecc no shlexquote on windows
@@ datalad/interface/run_procedure.py: def _guess_exec(script_file):
'state': state}
else:
return {'type': None, 'template': None, 'state': None}
-
--
- @build_doc
- class RunProcedure(Interface):
- """Run prepared procedures (DataLad scripts) on a dataset
Codecov Report
@@ Coverage Diff @@
## 0.11.x #3624 +/- ##
==========================================
- Coverage 81.23% 70.44% -10.8%
==========================================
Files 256 273 +17
Lines 33819 35511 +1692
==========================================
- Hits 27473 25015 -2458
- Misses 6346 10496 +4150
Continue to review full report at Codecov.
|
mih
changed the title
mih
changed the title
|
Thx @adswa and welcome! |
…-3623) Stay within the same Python version to avoid crash or undesired behavior when PY2 is called from PY3 with missing or outdated PY2 versions of datalad.
|
Hey, @kyleam this fix is needed for a 0.12 deployment tomorrow. Would be good to be able to pull that from its final resting place in master. |
You'll be able to. I'll merge 0.11.x into master. |
|
Perfect, thx! |
This test fails on AppVeyor [0]. run_procedure() uses a no-op shlex_quote() on Windows [1], so this is expected. [0]: https://ci.appveyor.com/project/mih/datalad/builds/27022010/job/jq3lr977ht5vrmpj#L1109 [1]: datalad#3624 (comment)
run_procedure() can't use shlex_quote() on Windows [0], so an argument with spaces will not be handled correctly [1]. We should probably teach the windows branch of maybe_shlex_quote() to handle simple cases like a name with spaces. [0]: datalad#3624 (comment) [1]: https://ci.appveyor.com/project/mih/datalad/builds/27022362/job/xoum5f43c135bfgd#L1125
0.11.7 (Sep 02, 2019) -- python2-we-still-love-you-but-...
Primarily bugfixes with some optimizations and refactorings.
Fixes
- [addurls][]
- now provides better handling when the URL file isn't in the
expected format. ([#3579][])
- always considered a relative file for the URL file argument as
relative to the current working directory, which goes against the
convention used by other commands of taking relative paths as
relative to the dataset argument. ([#3582][])
- [run-procedure][]
- hard coded "python" when formatting the command for non-executable
procedures ending with ".py". `sys.executable` is now used.
([#3624][])
- failed if arguments needed more complicated quoting than simply
surrounding the value with double quotes. This has been resolved
for systems that support `shlex.quote`, but note that on Windows
values are left unquoted. ([#3626][])
- [siblings][] now displays an informative error message if a local
path is given to `--url` but `--name` isn't specified. ([#3555][])
- [sshrun][], the command DataLad uses for `GIT_SSH_COMMAND`, didn't
support all the parameters that Git expects it to. ([#3616][])
- Fixed a number of Unicode py2-compatibility issues. ([#3597][])
Enhancements and new features
- The [annotate-paths][] helper now caches subdatasets it has seen to
avoid unnecessary calls. ([#3570][])
- A repeated configuration query has been dropped from the handling of
`--proc-pre` and `--proc-post`. ([#3576][])
- Calls to `git annex find` now use `--in=.` instead of the alias
`--in=here` to take advantage of an optimization that git-annex (as
of the current release, 7.20190730) applies only to the
former. ([#3574][])
- [addurls][] now suggests close matches when the URL or file format
contains an unknown field. ([#3594][])
- Shared logic used in the setup.py files of Datalad and its
extensions has been moved to modules in the _datalad_build_support/
directory. ([#3600][])
- Get ready for upcoming git-annex dropping support for direct mode
([#3631][])
* tag '0.11.7': (87 commits)
DOC: Added an entry to changelogn on merged 3631
ENH: finalizing changelog for 0.11.7
TST: Update tests for a git-annex without direct mode
TST: utils: Add decorator that skips when direct mode is unsupported
ENH: annexrepo: Refuse to initialize in direct mode if unsupported
ENH: annexrepo: Add check_direct_mode_support method
BF+TST: Avoid leaking patched git-annex version
TST+RF: test_annexrepo: Split up a test
CHANGELOG.md: Second batch for 0.11.7
TST: run_procedure: Mark test_spaces() as known Windows failure
TST: run_procedure: Mark test_quoting as known windows failure
TST: run_procedure: Test more arguments that need quoting
BF(py2): run_procedure: Avoid encoding error in log message
TST: add run_procedure test with spaces in file name
TST/RF: non-hardcoded Python executable
RF: newline at end of file
RF: helper instead of conditional
RF: remove superfluous imports
BF/TST: remove quoting
ENH: replace conditionals with helper function
...
0.11.7 (Sep 02, 2019) -- python2-we-still-love-you-but-...
Primarily bugfixes with some optimizations and refactorings.
Fixes
- [addurls][]
- now provides better handling when the URL file isn't in the
expected format. ([#3579][])
- always considered a relative file for the URL file argument as
relative to the current working directory, which goes against the
convention used by other commands of taking relative paths as
relative to the dataset argument. ([#3582][])
- [run-procedure][]
- hard coded "python" when formatting the command for non-executable
procedures ending with ".py". `sys.executable` is now used.
([#3624][])
- failed if arguments needed more complicated quoting than simply
surrounding the value with double quotes. This has been resolved
for systems that support `shlex.quote`, but note that on Windows
values are left unquoted. ([#3626][])
- [siblings][] now displays an informative error message if a local
path is given to `--url` but `--name` isn't specified. ([#3555][])
- [sshrun][], the command DataLad uses for `GIT_SSH_COMMAND`, didn't
support all the parameters that Git expects it to. ([#3616][])
- Fixed a number of Unicode py2-compatibility issues. ([#3597][])
- [download-url][] now will create leading directories of the output path
if they do not exist ([#3646][])
Enhancements and new features
- The [annotate-paths][] helper now caches subdatasets it has seen to
avoid unnecessary calls. ([#3570][])
- A repeated configuration query has been dropped from the handling of
`--proc-pre` and `--proc-post`. ([#3576][])
- Calls to `git annex find` now use `--in=.` instead of the alias
`--in=here` to take advantage of an optimization that git-annex (as
of the current release, 7.20190730) applies only to the
former. ([#3574][])
- [addurls][] now suggests close matches when the URL or file format
contains an unknown field. ([#3594][])
- Shared logic used in the setup.py files of Datalad and its
extensions has been moved to modules in the _datalad_build_support/
directory. ([#3600][])
- Get ready for upcoming git-annex dropping support for direct mode
([#3631][])
* tag '0.11.7':
Changelog entry for download-url paths handling
ENH: downloaders: Ensure directories for target exist
Stay within the same Python version to avoid crash or undesired behavior
when PY2 is called from PY3 with missing or outdated PY2 versions of
datalad.