parse command interpolations at parse time

[malmaud]

It would be nice to have a way to "double interpolate" a variable with respects to nested possible interpolation contexts to make it easier to write quoted code blocks that wants to interpolate a variable inside a string. e.g., so the below code echos "hi". Not sure what the best way to go about that is.

macro example()
 f="hi"
 quote
  run(`echo $$f`)
 end
end

[vtjnash]

the following already works without ambiguity, and doesn't require changing string parsing based on context:

macro example()
 f="hi"
 quote
  run($(`echo $f`))
 end
end

Is there an example where this isn't quite so straightforward?

[malmaud]

What about something like

macro example()
 f="a"
 esc(quote
  g="b"
  run(`echo $f $g`)
 end)
end

You could do instead

macro example()
 f="a"
 esc(quote
  g="b"
  h=$(f)
  run(`echo $h $g`)
 end)
end

but it's a little weird to be forced to create a temporary variable, since I'm not sure there is any other situation in Julia in which replacing a reference to a variable with its known value will change the semantics of the program. While here, changing $h to $(f) in the echo line changes the meaning of the program.

(edit: fix the variable in the second example)

[vtjnash]

That's tricky, but you don't exactly see it until you try to expand the macro. In the first example, f only exists at macro expansion time and g only exists at run time. Assigning the value of f to h seems to be necessary to convert the value into a variable. For strings, it is easy to use string(...) instead of interpolation to avoid the distinction.

julia> macro example()                             
               f="a"
               quote
                g="b"
                run(Cmd(ByteString["echo",$(f),g]))
               end                                       
              end

julia> @example
a b

julia> macroexpand(quote @example end)
quote  # none, line 1:
    begin  # none, line 4:
        g#4 = "b" # line 5:
        run(Cmd(ByteString["echo","a",g#4]))
    end
end

(edit: I forgot Cmd() is the analog to string(), so I fixed the example)

[JeffBezanson]

$($f) would work if we moved command expression expansion to the parse stage like we did for string interpolation.

The variable-to-value substitution you mention would be:

  h=$(f)
  run(`echo $h $g`)

to

  run(`echo $($f) $g`)

You can't drop the extra $. This doesn't work now, but would with the change I described.

[vtjnash]

this would be a new feature and thus not appropriate for the v0.4.x bugfix series. i'm going to tentatively move this to v0.5, please comment if you think that is not appropriate.

[StefanKarpinski]

Since we're not doing #12139 we can do this.

[JeffBezanson]

From triage: can live without this for 1.0.

[StefanKarpinski]

Removing from milestone, noting that this may change post 1.0, which won't affect normal code but could affect macros that deal with commands.

[simeonschaub]

Would changing the parsing of `a$(b)c` from (macrocall (core @cmd) "a$(b)c") to (macrocall (core @cmd) (string "a" b "c")) be too breaking for a 1.x release? That would be a slightly different approach than #13199. I think this would definitely make for some more intuitive behavior and would also, for example, make #37007 redundant.

[c42f]

This is an interesting approach particularly because it separates the detailed shell parsing rules from the rules for interpolating values. I suppose this means the parser changes will be much more minimal and completely consistent with the rules for string interpolation which is welcome.

I think this would be an excellent thing, provided it's sound! But I can think of at least one case where interpolation into normal strings vs interpolation into cmd strings behaves quite differently:

julia> `hi '$a'`
`hi '$a'`

julia> "hi '$a'"
ERROR: UndefVarError: a not defined
Stacktrace:
 [1] top-level scope at REPL[5]:1

Looking at the source of shell_parse this seems to be the only case. So in principle this one difference could be dealt with in the parser. It's a bit of an unfortunate difference though.

[StefanKarpinski]

That is quite intentional as it mirrors what the shell does. You often use single quotes in a shell command when you need to write some code that uses $ and not worry about escaping it. I think that deviating from the shell here would be a major reduction in usability. Of we're not going to cleave very closely to how the shell works, then it would be better to just use an array API for spawning commands.

A more general observation here is that the current arrangement is strictly more flexible: the macros get to decide exactly how and if they do interpolation. Yes, that makes implementing interpolation harder, but that's the trade off. Given that implementing macros like this is generally considered an advanced activity, that feels like the right trade off to me. The main downside is that user string expressions can't allow nested quotation like normal strings can; I haven't found that limitation to be especially bothersome—if you're putting quotes in the things you're interpolating into quotes, I think you're better off doing it in separate expressions anyway.

[robsmith11]

I just ran into the quoted Cmd vs String substitution inconsistency. My use case was looping over some variables to repeatedly call a giant ugly curl command with nested quoting generated from Chrome dev tools. It was a pain to break it up into an array, so would have been nice to have variable substitution working as I had expected (how it works for regular Strings).

[c42f]

Why was this closed as completed? If we're not going to fix this we should close it as "won't fix"?

Clearly fixing this would be breaking. But it's legitimately a rather annoying and surprising inconsistency.