8282008: Incorrect handling of quoted arguments in ProcessBuilder

[omikhaltsova]

This fix made equal processing of strings such as ""C:\Program Files\Git\"" before and after JDK-8250568.

For example, it's needed to execute the following command on Windows:
C:\Windows\SysWOW64\WScript.exe "MyVB.vbs" "C:\Program Files\Git\" "Test"
it's equal to:
new ProcessBuilder("C:\\Windows\\SysWOW64\\WScript.exe", "MyVB.vbs", "\"C:\\Program Files\\Git\\\"", "Test").start();

While processing, the 3rd argument ""C:\Program Files\Git\"" treated as unquoted due to the condition added in JDK-8250568.

    private static String unQuote(String str) {
    .. 
       if (str.endsWith("\\\"")) {
            return str;    // not properly quoted, treat as unquoted
        }
    ..
    }

that leads to the additional surrounding by quotes in ProcessImpl::createCommandLine(..) because needsEscaping(..) returns true due to the space inside the string argument.
As a result the native function CreateProcessW (src/java.base/windows/native/libjava/ProcessImpl_md.c) gets the incorrectly quoted argument:

pcmd = C:\Windows\SysWOW64\WScript.exe MyVB.vbs ""C:\Program Files\Git\"" Test
(jdk.lang.Process.allowAmbiguousCommands = true)
pcmd = "C:\Windows\SysWOW64\WScript.exe" MyVB.vbs "\"C:\Program Files\Git\\\"" Test
(jdk.lang.Process.allowAmbiguousCommands = false)

Obviously, a string ending with "\\\"" must not be started with "\"" to treat as unquoted overwise it’s should be treated as properly quoted.

---

Progress

• Change must not contain extraneous whitespace
• Commit message must refer to an issue
• Change must be properly reviewed

Issue

• JDK-8282008: Incorrect handling of quoted arguments in ProcessBuilder

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/7504/head:pull/7504
$ git checkout pull/7504

Update a local copy of the PR:
$ git checkout pull/7504
$ git pull https://git.openjdk.java.net/jdk pull/7504/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 7504

View PR using the GUI difftool:
$ git pr show -t 7504

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/7504.diff

[bridgekeeper]

👋 Welcome back omikhaltcova! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

@omikhaltsova The following label will be automatically applied to this pull request:

• core-libs

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 03: Full - Incremental (8cbdfe77)
• 02: Full - Incremental (0eceecac)
• 01: Full - Incremental (6a2c82f9)
• 00: Full (721d4023)

[RogerRiggs]

(Edited)
The fix does not correctly recognize the escaping of the final double-quote.
The PR should include a test. Can you write a standalone test in jdk/test/java/lang/ProcessBuilder/... to confirm the fix.
Possibly it could be added to Basic.java but that file is pretty large and doesn't seem like the correct place to add.
It may be sufficient to invoke echo with that 3rd argument and verify the output.

[RogerRiggs]

Actually, there's a bit more to this than first meets the eye.

"A double quote mark preceded by a backslash (\") is interpreted as a literal double quote mark (")."
According to: https://docs.microsoft.com/en-us/cpp/cpp/main-function-command-line-args

That was the reason for the change in JDK-8250568.

So the application supplied quotes combined with the trailing file separator results in unbalanced quotes.

Without the application supplied quotes, the implementation quotes the string (because of the embedded space) and doubles up the backslash so it does not escape the final quote.

[RogerRiggs]

@omikhaltsova Please take another look at the comment above. The fix incorrectly allows a final double-quote to be escaped, resulting in unbalanced quotes and possibly allowing an argument to be joined with the next.
The recommendation is for the application to NOT add quotes to arguments and allow ProcessBuilder to do the necessary quoting.

[omikhaltsova]

Roger, thanks for your comments! But in this case how it is possible to present a path ending with '\' and including a space inside?

[RogerRiggs]

ProcessBuilder handles the quoting of arguments with spaces.
In your QuotedArguments.java, just remove the quotes from the first argument to CheckCase:

                new CheckCase("C:\\Program Files\\Git\\", "C:\\Program Files\\Git\\", "true"),
                new CheckCase("C:\\Program Files\\Git\\", "C:\\Program Files\\Git\\", "false")
        };

That worked for me using openjdk version "17.0.2".
Problems with knowing what and if to quote go back a long time.
I'm working on a new JEP to handle more of the cases without application intervention.

[RogerRiggs]

Please close this PR; the proposed change to the application should resolve the issue.
The issue should be closed as "not-an-issue".

[omikhaltsova]

Roger, writing a test via echo was not a good idea obviously for this particular case because of the fact well shown in the doc "4. Everyone Parses Differently", https://daviddeley.com/autohotkey/parameters/parameters.htm#WINCRULESMSEX. The task is more complicated than it seems at the first glance. The same command line correctly parsed in an app written in C/C++ might be incorrectly parsed in a VBS app etc.

The suggestion not to use the path-argument surroundings with '\"' doesn't fix the issue in case of VBS. It leads to a resulting path-argument ending with the doubled backslash in a VBS app according to the rules "10.1 The WSH Command Line Parameter Parsing Rules", https://daviddeley.com/autohotkey/parameters/parameters.htm#WSH.

Below there are some experiments with an app attached to JDK-8282008:

NO FIX

1. new ProcessBuilder("C:\\Windows\\SysWOW64\\WScript.exe", "MyVB.vbs", "\"C:\\Program Files\\Git\\\"", "Test").start();

1.1 allowAmbiguousCommands = false
   arg[0] = \C:\Program 
   arg[1] = Files\Git1\\\  
CreateProcessW: pcmd = "C:\Windows\SysWOW64\WScript.exe" MyVB.vbs "\"C:\Program Files\Git1\\\"" Test   

1.2 allowAmbiguousCommands = true
   arg[0] = C:\Program 
   arg[1] = Files\Git1\  
CreateProcessW: pcmd = C:\Windows\SysWOW64\WScript.exe MyVB.vbs ""C:\Program Files\Git1\"" Test   

2. new ProcessBuilder("C:\\Windows\\SysWOW64\\WScript.exe", "MyVB.vbs", "C:\\Program Files\\Git\\", "Test").start();

2.1 allowAmbiguousCommands = false
   arg[0] = C:\Program Files\Git1\\ 
   arg[1] = Test  
CreateProcessW: pcmd = "C:\Windows\SysWOW64\WScript.exe" MyVB.vbs "C:\Program Files\Git1\\" Test   

2.2 allowAmbiguousCommands = true
   arg[0] = C:\Program Files\Git1\\ 
   arg[1] = Test 
CreateProcessW: pcmd = C:\Windows\SysWOW64\WScript.exe MyVB.vbs "C:\Program Files\Git1\\" Test   

FIXED (as in pr)

1. new ProcessBuilder("C:\\Windows\\SysWOW64\\WScript.exe", "MyVB.vbs", "\"C:\\Program Files\\Git\\\"", "Test").start();

1.1 allowAmbiguousCommands = false
   arg[0] = C:\Program Files\Git1\ 
   arg[1] = Test 
CreateProcessW: pcmd = "C:\Windows\SysWOW64\WScript.exe" MyVB.vbs "C:\Program Files\Git1\" Test   

1.2 allowAmbiguousCommands = true
 arg[0] = C:\Program Files\Git1\ 
 arg[1] = Test 
CreateProcessW: pcmd = C:\Windows\SysWOW64\WScript.exe MyVB.vbs "C:\Program Files\Git1\" Test  

Reading the code of unQuote() in terms of logic: "no beginning or ending quote, or too short => not quoted", - and it seems that if all these conditions are just opposite (starting and ending quotes and long enough) then a string should be treated as quoted but it's not. One exception was added and it's strange that it's applied even in case of paired quotes. Is it truly necessary for the security fix to skip (=to treat as unquoted) a string starting and ending with '\"' in case of a "\\\"" tail?

This proposed fix returns back possibility (as was previously) to use surrounding with '\"' as an argument mark-up that allows to pass correctly a path with a space inside in case of VBS. Roger, would you be so kind to take a look at this small fix once again, please, and to pay attention to VBS parsing arguments problem?!

[mlbridge]

Mailing list message from Raffaello Giulietti on core-libs-dev:

Hi,

as far as I know, on Windows every program can obtain the lpCommandLine
argument, used in the call of CreateProcess() from its parent, by
calling GetCommandLine() and parse that string as it sees fit. This is
in stark contrast with how Unix-like systems create and execute
programs, where the system call execve(2) accepts an array of arguments,
not a single command line.

There are no fixed rules on how to parse the command line, as witnessed
by the different strategies implemented in the C/C++ runtime (which
splits the command line according to the rules outlined in [1] to
populate the argv[] array in main()), the cmd.exe shell, the wscript.exe
runtime, etc.

Consequently, there are no fixed rules on how to encode a command line
(specifically, the lpCommandLine argument to CreateProcess()) because it
really is up to the invoked program to parse it, whether explicitly or
implicitly, according to its own, directly or indirectly implemented
rules. Even a C/C++ console program could ignore the result that the
runtime automatically provides in argv[] and parse the command line
directly, as obtained by GetCommandLine().

Without knowing the parsing rules of the target program, it is not
possible to encode a command line correctly for CreateProcess(). I doubt
there's a "common denominator" which would cover most cases encountered
in practice.

The best we can hope is to implement encoders (and decoders) for
specific, widely used runtimes. (BTW, for the C/C++ runtime I prepared
an implementation mentioned in [2].)

Greetings
Raffaello

----

[1]
https://docs.microsoft.com/en-us/cpp/c-language/parsing-c-command-line-arguments
[2]
https://mail.openjdk.java.net/pipermail/core-libs-dev/2022-February/086105.html

On 2022-02-24 20:18, Olga Mikhaltsova wrote:

[mkartashev]

It is apparent there is no one "correct" way to quote, but one of the key features of the Java ecosystem has been its backwards compatibility. In that light, this change allows our clients to continue doing what they did the way they did it without the need for modification of their Java code or their (maybe even foreign) native code. FWIW.

Separately from the above, I wonder if this change would make the fix less controversial?

-        if (str.endsWith("\\\"")) {
+        if (str.endsWith("\\\"") && !str.endsWith("\\\\\"")) {

This way we verify that the end quote is really just an escaped quote, while correctly identifying escaped backslash as having nothing to do with the following quote.

[mkartashev]

Actually, this change should be made even more generic because the string might end with any even number of the backslash characters followed by a free-standing quote, in which case additional quoting should not be required.

[RogerRiggs]

(I'm still working on a more nuanced fix that works with .exe, .cmd, and with allowAmbiguousCommands both true and false).

The suggested workaround was to remove the application quotes and let ProcessBuilder do the quoting.
That resulted in an extra backslash "\" at the end of a file path. In my investigation, the extra "\" doesn't prevent the
string from being correctly used as a directory path in either VisualBasic or cmd.exe.
So I'm curious, in the original application that uncovered this problem, what is/was reported as the error?
Was the original application retested with the workaround?

The case of the backslash at the end of an argument occurs mainly in a directory path.
Yes, the argument is different, but does it make a difference that matters in the context in which it appears.

[mkartashev]

@RogerRiggs
Our use case was something like this java -classpath "C:\Program Files\MySQL\JDBC\\" .... More specifically, while this works after JDK-8250568 (the string ends with ")

    ProcessBuilder("java.exe", "-classpath", "\"C:\\New folder\"", "Test", "test");

this doesn't and, as I understand, shouldn't (the string ends with \"):

    ProcessBuilder("java.exe", "-classpath", "\"C:\\New folder\\\"", "Test", "test");

and produces errors like these

Error: Could not find or load main class folder"" Test test
Caused by: java.lang.ClassNotFoundException: folder"" Test test

However, the following still doesn't work, but, I believe, should (the string ends with \\"):

    ProcessBuilder("java.exe", "-classpath", "\"C:\\New folder\\\\\"", "Test", "test");

[RogerRiggs]

Thanks for the example, though my question was in the case in which the extra quotes were not included by the app.
For example,

     ProcessBuilder("java.exe", "-classpath", "C:\\New folder\\", "Test", "test");

I thought the troublesome case was specific to VB/WScript being invoked with ".exe" quoting but WSCRIPT interpreting the command line string using simple quoting with no escapes.

[RogerRiggs]

As an alternative fix, please take a look at Draft PR: #7709.

In the default handling of arguments, the check for what is quoted is reverted to prior to 8255068. First and last quotes are sufficient to identify a "quoted" string. The check for a backslash ("\") is removed.
This original check is sufficient for jdk.lang.Process.allowAmbiguousCommands = true.

For the case where the system property jdk.lang.Process.allowAmbiguousCommands = false
and the argument has first and last quotes, a backslash ("\") before the final quote must not allow the quote to interpreted as a literal quote and merge the following argument. The backslashes will doubled to prevent the interpretation of the quote as a literal. This is the correct encoding if the command uses the ".exe" encoding, when reparsing the arguments the doubled backslashes are reduced to the original contents.
When the command is using the simpler parsing that does not support literal quotes, the backslash before the quote is typically is a trailing backslash on a file path and in that case the additional backslash is redundant and has no effect on the interpretation of the argument as a directory path.

The PR includes a test of the 12 combinations of invoking an "java"/.exe program, a .cmd script, and a Visual Basic script (which uses the .exe rules but different command line parser); with and without application quotes and compares the actual results with the expected arguments.

[mkartashev]

@RogerRiggs I believe your patch fixes the use case(s) we are interested in. Would be good to see it merged into master.

[omikhaltsova]

@RogerRiggs Sorry for the delay! I also checked, the test-case with VBS, that raised this issue, successfully workes with your patch. It would be great to have it asap.

[omikhaltsova]

Closed due to an alternative fix #7709.

[RogerRiggs]

Thanks for the followup and confirmation. I'll move #7709 to review from draft.