Command line -c "normal..." stays in normal mode for subsequent -c commands

[tplunket]

I use this little batch file for sticking my company's SVN commit template onto the clipboard:

@echo off
call vim -R "%~0" -c "normal }jvG$""*y" -c quit
goto :eof

BB:
JIRA:
ChangeSummary:
Details:
Platform:

This works in the gvim74.exe installation and fails for the gvim74-1024.exe installation. It appears as though normal mode persists beyond the -c argument because if I change it to -c hello (so it's obvious what's going on) and remove the -R the editor starts up and I'm left with this in the buffer:

@echo off
call vim "%~0" -c "normal }jvG$""*y" -c hello
goto :eof
hello
BB:
JIRA:
ChangeSummary:
Details:
Platform:

The -c hello is interpreted as a continuation of the normal mode command.

[brammool]

tplunket wrote:

I use this little batch file for sticking my company's SVN commit
template onto the clipboard:

@echo off
call vim -R "%~0" -c "normal }jvG$""*y" -c quit
goto :eof

BB:
JIRA:
ChangeSummary:
Details:
Platform:

This works in the gvim74.exe installation and fails for the
gvim74-1024.exe installation. It appears as though normal mode
persists beyond the -c argument because if I change it to -c hello
(so it's obvious what's going on) and remove the -R the editor
starts up and I'm left with this in the buffer:

@echo off
call vim "%~0" -c "normal }jvG$""*y" -c hello
goto :eof
hello
BB:
JIRA:
ChangeSummary:
Details:
Platform:

The -c hello is interpreted as a continuation of the normal mode command.

Isn't this a problem with the quoting of the argument?
I very much doubt this happens inside Vim.

ARTHUR: You are indeed brave Sir knight, but the fight is mine.
BLACK KNIGHT: Had enough?
ARTHUR: You stupid bastard. You havn't got any arms left.
"Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD

/// Bram Moolenaar -- Bram@Moolenaar.net -- http://www.Moolenaar.net \
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \
\ an exciting new programming language -- http://www.Zimbu.org ///
\ help me help AIDS victims -- http://ICCF-Holland.org ///

[tplunket]

(For the record: I'm calling the Vim executable directly in the below commands rather than using the vim.bat helper script.)

It seems as though Vim is doing something incorrect with the quote. Escaping a double quote inside a quoted string only requires doubling it:

>python -c "import sys; print sys.argv[1]" "hello "" world"
hello " world

Again, this batch file worked fine with the package in the gvim74.exe installer that's on vim.org, and it does not work with the gvim74-1024.exe installed app.

If I run this command:

>vim test.bat -c "execute 'normal }jvG$""*y'" -c quit

then I get the error:

Error detected while processing command line:
E121: Undefined variable: c
E15: Invalid expression: 'normal }jvG$"*y' -c quit

We can see here clearly that it's trying to execute the entire line after the quote, but that the quote was passed through correctly.

The most stripped down command line that causes this error is

vim -c "execute 'normal ""*Y'" -c quit

(If I remove the execute bit I don't get the error but Vim remains running.)

And just as a test to see if the shell is in fact separating the arguments the way we think it should,

>python -c "import sys;print '\n'.join(sys.argv[1:])" -c "execute 'normal ""*Y'" -c quit
-c
execute 'normal "*Y'
-c
quit

All of these tests run with the executable installed by the vim74.exe installer and work correctly. This is a new failure, a regression since its release.

[tplunket]

I spent some time looking at the code but it quickly became apparent that I'll need to step through it to see what's going on. However I was struck by the thought that maybe something was doing quote processing inside of vim so I tried to double my ""*Y attempts.

This works as expected in 7.4.0 and fails with an error in 7.4.1023:

vim +"execute 'normal ""*Y""*Y'" +quit

The error in .1023 is:

Error detected while processing command line:
E121: Undefined variable: quit
E15: Invalid expression: 'normal "*Y*Y' +quit

So it's munching an additional quote mark for some reason.

Again, Python shows that the arguments are passed as desired by the shell:

>python -c "import sys;print '\n'.join(sys.argv[1:])" +"execute 'normal ""*Y""*Y'" +quit
+execute 'normal "*Y"*Y'
+quit

[k-takata]

I think 7.4.432 is the cause of this difference.

Before 7.4.432, the command line was parsed differently when using gvim.exe or
vim.exe, or even built with VC or MinGW. After 7.4.432, the command line is
parsed by CommandLineToArgvW() and it behaves same when using gvim.exe or
vim.exe, or built with VC or MinGW.
(Your batch file might not work when using gvim.exe 7.4.000 instead of vim.exe.)

So the current quotation behavior is based on CommandLineToArgvW():
https://msdn.microsoft.com/en-us/library/windows/desktop/bb776391.aspx

If you want to pass a double quotation inside a pair of double quotations,
either of the following might work:

1. Escape with a backslash.
   This should work gvim/vim, before/after 7.4.432, built with VC/MinGW.
   E.g. vim -c "echo 'foo\"bar'"
2. Use triple double-quotations. (Maybe less portable.)
   E.g. vim -c "echo 'foo"""bar'"

BTW, it is not well documented that how the CommandLineToArgvW() parses double
quotations inside double quotations. :-(
It seems that 3n quotations inside a pair of quotations produce n quotations.

More detail by my analysis:

1. A " starts quotation.
2. Another " or "" ends quotation. If the quotation ends with "", a "
   is produced at the end of the quoted string.

E.g.:

    "foo"           -> [foo]
    "foo""          -> [foo"]
    "foo"bar        -> [foobar]
    "foo" bar       -> [foo], [bar]
    "foo""bar       -> [foo"bar]
    "foo"" bar      -> [foo"], [bar]
    "foo"""bar"     -> [foo"bar]

[tplunket]

Thanks, @k-takata, for researching this. It was previously safe to send quotes through the vim.bat helper script that gets installed but with the change to command line processing now it seems the command processor gets tripped up by the escaping that CommandLineToArgvW requires/expects. It would appear, then, that if someone uses quotes that they cannot use them through those helper scripts, they have to have the exe on their path.

[brammool]

Ken Takata wrote:

I think 7.4.432 is the cause of this difference.

Before 7.4.432, the command line was parsed differently when using gvim.exe or
vim.exe, or even built with VC or MinGW. After 7.4.432, the command line is
parsed by CommandLineToArgvW() and it behaves same when using gvim.exe or
vim.exe, or built with VC or MinGW.
(Your batch file might not work when using gvim.exe 7.4.000 instead of vim.exe.)

So the current quotation behavior is based on CommandLineToArgvW():
https://msdn.microsoft.com/en-us/library/windows/desktop/bb776391.aspx

If you want to pass a double quotation inside a pair of double quotations,
either of the following might work:

1. Escape with a backslash.
   This should work gvim/vim, before/after 7.4.432, built with VC/MinGW.
   E.g. vim -c "echo 'foo\"bar'"
2. Use triple double-quotations. (Maybe less portable.)
   E.g. vim -c "echo 'foo"""bar'"

BTW, it is not well documented that how the CommandLineToArgvW() parses double
quotations inside double quotations. :-(
It seems that 3n quotations inside a pair of quotations produce n quotations.

More detail by my analysis:

1. A " starts quotation.
2. Another " or "" ends quotation. If the quotation ends with "", a "
   is produced at the end of the quoted string.

E.g.:

    "foo"           -> [foo]
    "foo""          -> [foo"]
    "foo"bar        -> [foobar]
    "foo" bar       -> [foo], [bar]
    "foo""bar       -> [foo"bar]
    "foo"" bar      -> [foo"], [bar]
    "foo"""bar"     -> [foo"bar]

Thanks, I'll add that to the help.

The fastest way to get an engineer to solve a problem is to declare that the
problem is unsolvable. No engineer can walk away from an unsolvable problem
until it's solved.
(Scott Adams - The Dilbert principle)

/// Bram Moolenaar -- Bram@Moolenaar.net -- http://www.Moolenaar.net \
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \
\ an exciting new programming language -- http://www.Zimbu.org ///
\ help me help AIDS victims -- http://ICCF-Holland.org ///