New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

quote removal in subprocess mode does not behave as expected #621

Closed
riccardomurri opened this Issue Jan 12, 2016 · 7 comments

Comments

Projects
None yet
3 participants
@riccardomurri
Copy link

riccardomurri commented Jan 12, 2016

Please have a look at the following two examples::

    xonsh$ /bin/echo 'foo'
    foo

    xonsh$ /bin/echo 'foo'bar
    'foo'bar

In the first case quotes are stripped from the second argument; in the
second, quotes are kept.

This is not consistent with what other shells do, and makes it quite
inconvenient to pass option arguments containing spaces. For example,
the following would include the quotes in the commit message::

    svn commit --message='A short description here.'

@scopatz scopatz added the question label Jan 12, 2016

@scopatz scopatz added this to the v0.3 milestone Jan 12, 2016

@scopatz

This comment has been minimized.

Copy link
Member

scopatz commented Jan 12, 2016

Hi @riccardomurri, you are correct that we are not consistent with how other shells treat string. Xonsh makes a willful departure from what other shells do, in part because what other shells do is not self-consistent. Rather, xonsh's literal string syntax sticks as close as possible to Python string behaviour, which we believe to be cleaner and more consistent.

The only departure from Python strings in subproc mode is that literal strings are globbed and have their environment variables expanded, as one would expect from other shells.

The examples that you give of echo and svn are not entirely consistent themselves. It is generally the responsibility of each program to parse its own arguments. So if svn is including quotes this is svn fault. As an demonstration, consider the following Bash script which simply prints its arguments:

temp.sh

#!/bin/bash
echo $*

Now let's run this with multiple arguments.

scopatz@localhost ~ $ chmod 755 temp.sh 

# see bash keeps the quotes
scopatz@localhost ~ $ ./temp.sh --message='hello mom'
--message='hello mom'  

# bash keeps the quotes even when running though bash directly
scopatz@localhost ~ $ bash temp.sh --message='hello mom'
--message='hello mom'

# same thing with double quotes
scopatz@localhost ~ $ bash temp.sh --message="hello mom"
--message="hello mom"

# Let's add a 'foo'bar argument, and see that the quotes remain.
scopatz@localhost ~ $ bash temp.sh 'foo'bar --message="hello mom"
'foo'bar --message="hello mom"

So bash is doing something strange here with echo and a literal string + characters that I don't think is worth replicating.

As per getting the SVN example to work in a natural way, recall that arguments are whitespace separated. Thus, I would leave off the = and you have two distinct tokens:

$ svn commit --message 'A short description here.'

Alternatively, if you truly must have the equals sign and want xonsh to group the internal whitespace, I believe that you can put the whole token in quotes.

$ svn commit '--message=A short description here.'

This is probably less satisfying, but works because your original command is actually equivalent to:

$ svn commit "--message='A short description here.'"

In general for xonsh, if a subprocess argument does not start and end with quote characters or other known syntax like $(), $[], or @() then that argument is interpreted as being wrapped in Python-like quotes.

I hope this helps clear things up.

@riccardomurri

This comment has been minimized.

Copy link

riccardomurri commented Jan 13, 2016

Hello @scopatz, thank for looking into this!

Actually, I cannot replicate the "temp.sh" example you showed. (Did
you run that from windows?) My bash (and zsh too) removes the quotes,
both single and double, wherever they are in the argument being
processed, which is consistent with what the bash man page
states
::

rmurri@xenia:~$ cat /tmp/test.sh
#! /bin/sh

echo $*

rmurri@xenia:~$ /tmp/test.sh foo'bar'
foobar

rmurri@xenia:~$ /tmp/test.sh --message='foo'
--message=foo

rmurri@xenia:~$ /tmp/test.sh --message="foo"
--message=foo

I understand Xonsh does not need to retain compatibility; actually, I
would go as far as to argue that, since Xonsh does not do
word-splitting after variable substitution, it does not need quote
removal at all::

    xonsh$ def cnt(args, stdin=None):
               print(len(args))

    xonsh$ aliases['cnt'] = cnt

    # three separate arguments
    xonsh$ cnt a b c
    3

    # when coming from a var, this is one argument instead
    xonsh$ $var='a b c'
    xonsh$ cnt $var
    1

Instead sh/bash needs quote removal to control whether variables are
expanded into one or several arguments:

    bash$ cnt () { echo $#; }

    # three args
    bash$ cnt a b c
    3

    # still three args because of word splitting
    bash$ var='a b c'
    bash$ cnt $var
    3
    bash$ cnt "$var"
    1

Still, if you keep quote removal, I think it should behave like
sh/bash does, for the principle of least surprise. (And because
rewriting simple cases of quote removal using Xonsh' @(...) is too
verbose.)

@riccardomurri

This comment has been minimized.

Copy link

riccardomurri commented Jan 13, 2016

I stand corrected: of course we need quote removal in order to have
arguments with embedded spaces.

So, to sum up, there are two styles of quote removal:

  • What Xonsh currently implements (Windows-style?): remove quotes only
    if an argument starts and ends with them.
  • UNIX shell style: remove quotes anywhere in an argument as the last
    step of command-line processing.

Being used to UNIX shells, I would like to see the latter style
implemented, at least as an option.

@scopatz

This comment has been minimized.

Copy link
Member

scopatz commented Jan 15, 2016

Hi @riccardomurri, I think I got the bash behaviour wrong myself. temp.sh does act as you describe if I am in a Bash interactive instance.

On the subject of what to do about it, I would really call the first one "Python-style." My personal opinion is that the way the sh-lang spec is written is rather arcane and not up to modern notions of how strings should work. (Though I am not saying that what we have now is perfect.)

Now, I may be wrong about this, but I believe that adding in optional Unix-sh-style strings would touch the parser. I am a little wary of having option-dependent syntax. This means that xonsh code could not be universally parsed.

On the flipside, adding in the behvaiour for strings you are suggesting would have overlapping syntax some Python literals, namely b"x", r"x", u"x", and in v3.6 now f"x". Whatever would be done would have to avoid these specific cases.

Now, that said, I am open to suggestions - though for something like this I would also like for it to be vetted by @adqm, @gforsyth, and the mailing list. A suggestion backed by a PR would be even more compelling. Here are a couple of thoughts that I had that might be more or less compelling:

  1. Use double quotes. I don't believe that Python uses double quote so foo""bar"" and foo''bar'' may be fair game.
  2. Match a backtick and a single quote, like in LaTeX.
  3. Make ="" and ='' an operator in subproc mode that removes the quote characters and keeps the equals sign. This doesn't cover the foo'bar' case, and therefore not as hard to understand or implement.
  4. Come up with some other crazy syntax that is better than the above.

In any event, I guess I'd like folks to weigh in with their thoughts.

@gforsyth

This comment has been minimized.

Copy link
Collaborator

gforsyth commented Jan 15, 2016

Hey @scopatz,

I agree that Python doesn't use double-up quotes -- are you envisioning something along the lines of

> echo foo""bar""
foo"bar"

?

That doesn't seem too terribly hard, but I'm also at a loss for an actual use-case. @riccardomurri -- did you have a specific idea in mind that led you to this example? If it's the svn case, then my inclination is to go with number 3 above and create the subproc operators.

@scopatz

This comment has been minimized.

Copy link
Member

scopatz commented Jan 15, 2016

Hi @gforsyth, sorry to be clear, I am envisioning double quotes as the following:

$ echo foo""bar""
foobar

Which would mirror Bash's quoting behaviour. However, I don't think that this makes anyone happy because it isn't as concise as the sh-lang syntax for the people that want it and it makes something possible that should not be possible for the people who don't want it (and are generally anti-arcana).

@scopatz

This comment has been minimized.

Copy link
Member

scopatz commented Jan 15, 2016

Also, I am most pro option 3 as well.

@gforsyth gforsyth modified the milestones: v0.4.0, v0.3 May 26, 2016

@gforsyth gforsyth modified the milestones: v0.4.0, v0.5 Sep 9, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment