ENH: pass values through shlex_quote, fixes #3625

[adswa]

In this PR I have

• removed quoted placeholders from the template in _guess_exec in run_procedure.py and
• instead shlex_quoted() the values supplied in __call__, (if we are not on Windows)

However, locally, this crashes two tests:

======================================================================
FAIL: datalad.core.local.tests.test_create.test_create_withprocedure
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/adina/env/handbook/lib/python3.7/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/adina/repos/datalad/datalad/tests/utils.py", line 607, in newfunc
    return t(*(arg + (filename,)), **kw)
  File "/home/adina/repos/datalad/datalad/core/local/tests/test_create.py", line 406, in test_create_withprocedure
    eq_(ds.config['datalad.metadata.nativetype'], ('xmp', 'datacite'))
AssertionError: '"xmp" "datacite"' != ('xmp', 'datacite')

======================================================================
FAIL: datalad.interface.tests.test_run_procedure.test_configs
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/adina/env/handbook/lib/python3.7/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/adina/repos/datalad/datalad/tests/utils.py", line 895, in newfunc
    return func(*args, **kwargs)
  File "/home/adina/repos/datalad/datalad/tests/utils.py", line 434, in newfunc
    return t(*(arg + (d,)), **kw)
  File "/home/adina/repos/datalad/datalad/interface/tests/test_run_procedure.py", line 228, in test_configs
    ok_file_has_content(op.join(ds.path, 'fromproc.txt'), 'some_arg\n')
  File "/home/adina/repos/datalad/datalad/tests/utils.py", line 417, in ok_file_has_content
    assert_equal(content, file_content, **kwargs)
AssertionError: 'some_arg\n' != '"some_arg"\n'
- some_arg
+ "some_arg"
? +        +

because multiple arguments are now returned as a single string. I lack the judgment to decide whether this is indeed problematic, and at which point I can do adjustments.

Can someone give me a pointer?

(I put the commits on top of those in #3624 to prevent merge conflicts, let me know if that's okay)

[mih]

shelx_quote() would need to happen on each a inside the format call AFAICS:

u' '.join(u'"{}"'.format(a if on_windows else shlex_quote(a)) for a in args) if args else '')

[adswa]

oh, thx!

[mih]

i think the entire u'"{}".fornat can go and a or quoted a passed directly.

[kyleam]

because multiple arguments are now returned as a single string. I lack the judgment to decide whether this is indeed problematic, and at which point I can do adjustments.

Can someone give me a pointer?

You explained it pretty well yourself :] As @mih suggests, you should map shlex_quote over each argument.

(I put the commits on top of those in #3624 to prevent merge conflicts, let me know if that's okay)

That's good.

• instead shlex_quoted() the values supplied in __call__, (if we are not on Windows)

I suppose every shlex_quote() call should get the "not on windows" condition, so it seems like we should add a helper (probably to datalad.utils) that handles this.

Also, while I personally don't like our "BF, ENH" subject convention, it is pretty consistently used in this code base, so please prepend those :]

[codecov]

Codecov Report

Merging #3626 into 0.11.x will decrease coverage by 0.12%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           0.11.x    #3626      +/-   ##
==========================================
- Coverage   81.44%   81.31%   -0.13%     
==========================================
  Files         256      256              
  Lines       33823    33851      +28     
==========================================
- Hits        27547    27527      -20     
- Misses       6276     6324      +48

Impacted Files	Coverage Δ
datalad/utils.py	64.87% <100%> (-0.1%)	⬇️
datalad/interface/run_procedure.py	68.32% <100%> (-19.95%)	⬇️
datalad/interface/tests/test_run_procedure.py	100% <100%> (ø)	⬆️
datalad/interface/unlock.py	64.19% <0%> (-13.59%)	⬇️
datalad/downloaders/tests/test_http.py	88.86% <0%> (-2.23%)	⬇️
datalad/support/annexrepo.py	58.52% <0%> (-0.52%)	⬇️
datalad/interface/run.py	70.64% <0%> (+1.49%)	⬆️
datalad/cmdline/main.py	66.37% <0%> (+4.36%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update de6dbb0...d805fbe. Read the comment docs.

[kyleam]

Thanks for working on this.

I left a few comments. The only things that I think really need to be addressed are

• the str issue

• there is a failing test on the sym link test run

  https://travis-ci.org/datalad/datalad/jobs/577417054#L3410

  That looks to be because the test uses a custom call format and puts quotes around the values.

This should also get a regression test, but I can add that on top if you'd like.

[kyleam]

Thanks for working on this.

I left a few comments. The only things that I think really need to be addressed are

• the str issue

• there is a failing test on the sym link test run

  https://travis-ci.org/datalad/datalad/jobs/577417054#L3410

  That looks to be because the test uses a custom call format and puts quotes around the values.

This should also get a regression test, but I can add that on top if you'd like.

[kyleam]

These changes are OK as is, but just a note that essentially the same change is happening three places. This is a good indication that the common bits could have been pulled out into a variable and then the change could have been made to just one spot.

[kyleam]

The changes to all these arguments could be simplified by adding a shlex_quote helper that returns the value as is on windows. Even if you prefer not to add a general helper right now, you could add a local one. Something like (untested)

diff --git a/datalad/interface/run_procedure.py b/datalad/interface/run_procedure.py
index 9d2f8e4a3..048f8690f 100644
--- a/datalad/interface/run_procedure.py
+++ b/datalad/interface/run_procedure.py
@@ -448,11 +448,11 @@ def __call__(
             raise ValueError("No idea how to execute procedure %s. "
                              "Missing 'execute' permissions?" % procedure_file)
 
+        maybe_shlex_quote = lambda x: x if on_windows else shlex_quote
         cmd = ex['template'].format(
-            script=procedure_file if on_windows else shlex_quote(procedure_file),
-            ds='' if not ds else (ds.path if on_windows else shlex_quote(ds.path)),
-            args=(u' '.join(shlex_quote(a) for a in args) if args else '') if not on_windows
-                    else u' '.join(str(a) for a in args) if args else '')
+            script=maybe_shlex_quote(procedure_file),
+            ds=maybe_shlex_quote(ds.path) if ds else '',
+            args=u' '.join(maybe_shlex_quote(a) for a in args))
         lgr.debug('Attempt to run procedure {} as: {}'.format(
             name,
             cmd))

In addition to moving the condition to one place, this takes advantage of ' '.join([]) => '' to drop an unnecessary check for the empty args case.

And one comment that isn't stylistic/subjective: the str() call in u' '.join(str(a) for a in args) can cause unicode issues on python 2. six.text_type is one way you can use unicode/str only py2/py3, but you actually don't need it here because you can just do away with the list comprehension and give args directly to join(). (Taking a quick glance, there would still be upstream issues with unicode in py2, but the code here wouldn't be contributing to the issue.)

[kyleam]

Also, while I personally don't like our "BF, ENH" subject convention, it is pretty consistently used in this code base, so please prepend those :]

adswa changed the title [...]

Oops, sorry, I should've made it clear that I was talking about the commit message subjects.

[mih]

FTR: The test might be fixed like this:

diff --git a/datalad/interface/tests/test_run_procedure.py b/datalad/interface/tests/test_run_procedure.py
index d9a76919e..4a17b7765 100644
--- a/datalad/interface/tests/test_run_procedure.py
+++ b/datalad/interface/tests/test_run_procedure.py
@@ -238,7 +238,7 @@ def test_configs(path):
     # for run:
     ds.config.add(
         'datalad.procedures.datalad_test_proc.call-format',
-        'python "{script}" "{ds}" {{mysub}} {args}',
+        'python {script} {ds} {{mysub}} {args}',
         where='dataset'
     )
     ds.config.add(
@@ -258,7 +258,7 @@ def test_configs(path):
     # config on dataset level:
     ds.config.add(
         'datalad.procedures.datalad_test_proc.call-format',
-        'python "{script}" "{ds}" local {args}',
+        'python {script} {ds} local {args}',
         where='local'
     )
     ds.unlock("fromproc.txt")

I could not test it though, because the current PR is against 0.11, rebase wasn't straightforward and with all the extensions my dev environment cannot easily handle this. Sorry.

[adswa]

@mih this makes it work for me, I've pushed those changes. Thanks!

[mih]

I personally prefer line endings all the way to the last line, but we are not agreement within the project already.

[adswa]

done.

[mih]

This is no longer needed with maybe_shlex_quote

[mih]

It is not used consistently in the code base (yet), but we now do (for any import with more than a single item)

from datalad.utils import (
    on_windows,
    maybe_shlex_quot,
)

i.e. one per line, each line (incl last one has a trailing comma) -- this minimizes the diffs and make them quicker to grasp.

That being said on_windows is no longer needed, see below.

[adswa]

Ah yes, I missed this. I removed all superfluous imports now.

[mih]

Can also be maybe_shlex_quote(sys.executable) now

[adswa]

good point, thx.

[mih]

Tests did not run (apt issue). Restarted.

[mih]

Ah, sorry, didn't spot this one until now.

[mih]

Ah, sorry, didn't spot this one until now.

[mih]

Here and above we should also use sys.executable not python.

[adswa]

oh right. Thanks!

[adswa]

Hi @kyleam, I have tried to write a regression test, but that ended up being a test with args that contain spaces -- which is a test that I think cannot fail currently (although there currently is no test that explicitly gives args with spaces, so I pushed it anyway). I wasn't able to come up with any test that can trigger a failure in the case of quoted placeholders... If you have an idea how to write a test for this, you can let me know or add one yourself, if you like.

[kyleam]

Hi @kyleam, I have tried to write a regression test, but that ended up being a test with args that contain spaces

Thanks!

-- which is a test that I think cannot fail currently

Right, it passes without the changes from this PR (i.e. on top of 0.11.x).

(although there currently is no test that explicitly gives args with spaces, so I pushed it anyway).

Sounds good.

I wasn't able to come up with any test that can trigger a failure in the case of quoted placeholders... If you have an idea how to write a test for this, you can let me know or add one yourself, if you like.

I've pushed a test. It should fail on AppVeyor, but I'd like to see that before marking it as a known windows failure. Feel free to mark it once the test run is over. (edit: already failed and already marked)

[kyleam]

Assuming the currently running tests pass, I consider this is good to go. Thanks @adswa.

[kyleam]

Assuming the currently running tests pass, I consider this is good to go. Thanks @adswa.

[kyleam]

test_spaces() fails on AppVeyor. While run_procedure's previous approach of simply surrounding the argument with quotes is incorrect, it would have worked on windows for the simple case of a file name with spaces (I think, though of course we didn't have this test). Perhaps we should make the windows branch of maybe_shlex_quote() smarter so that it can handle simple cases like this, but how we handle shlex_quote on windows needs more work and thought in general. For the purposes of this PR I think we should switch that test's "skip if indirect" check to a full "skip on windows".

[mih]

Thx @adswa and @kyleam !