Implement accommodations for Windows CLIs, notably batch files and misexec-style programs as part of experimental feature PSNativeCommandArgumentPassing

[mklement0]

Summary of the new feature/enhancement

A Guide to this Proposal:

• The rest of this initial post details the motivation and the proposed implementation.

• Easy-to-grasp examples of the proposed accommodations are in this comment.

• Complementary examples of what won't work unless we implement these accommodations are in another comment.

---

As requested by @TravisEz13 in #14692 (comment), following a suggestion from @iSazonov in #14692 (comment):

^{The following is adapted from #14747 (comment), which contains some additional information about native argument-passing on Windows.}

PR #14692 introduces experimental feature PSNativeCommandArgumentPassing that will address parameter-passing woes when calling native programs with respect to embedded quoting and empty-string arguments, taking advantage of System.Diagnostics.ProcessStartInfo.ArgumentList, which:

• on Unix-like platforms: fully solves all problems.

• on Windows: solves the problem only for those programs that adhere to the quoting and escaping conventions used by Microsoft's C/C++ runtime.

While this is a great step in the right direction, it leaves out many Windows CLIs that do not play by these rules:

• The most prominent exception is cmd.exe - and therefore calls to batch files: they accept only "" as an escaped ", not the \" required by the C/C++ convention); while Microsoft compiler-generated executables also support "", there are third-party programs that support only \")

• An additional problem is that batch files unfortunately and inappropriately parse their arguments as if they had been passed from inside cmd.exe, which causes something like .\foo.cmd http://example.org?foo&bar to break due to & being misinterpreted as a statement separator. Using "http://example.org?foo&bar", i.e. quoting from PowerShell doesn't help, because PowerShell - justifiably - omits the quotes when it rebuilds the process command line behind the scenes, given that value contains neither spaces nor embedded " chars.

  • This is especially problematic given that the CLIs of many high-profile environments (e.g., az.cmd for Azure, and the wrapper batch files that npm (Node.js's package manager) creates for (Java)script-based utilities that come with packages) use batch files as their CLI entry points, so that something like
  • az ... 'http://example.org?foo&bar' predictably fails.

• Calling cmd.exe /c "<command-line>" or cmd.exe /k "<command-line>" directly with a single-argument command line to be executed through a happy accident actually currently works as intended, without a workaround - and that behavior must be retained.

• Many programs are particular about partial quoting of arguments, notably msiexec.exe with property arguments such as PROP="VALUE WITH SPACES"; purely syntactically, "PROP=VALUE WITH SPACES" (which is what PowerShell currently sends) should be equivalent (and if you let the C/C++ runtime / CLR parse it, is - the resulting verbatim string is PROP=VALUE WITH SPACES in both cases), but in practice it is not.

  • PowerShell should not pay attention to the original quoting on the PowerShell command line in an attempt to emulate it when re-encoding behind the scenes; no such quoting may be present to begin with (e.g., PROP=$someValuePossiblyWithSpaces), and users generally shouldn't have to worry about such intricacies - see below.

• Finally, calls to the WSH (Windows Script Host) CLIs cscript.exe (console) and wscript.exe - either directly or via associated script file types, notably .vbs (VBScript) and .js (JScript), behave poorly with \"-escaped embedded " characters; while the problem cannot be fully solved, ""-escaping results in better behavior: see below for details.

It's impossible for PowerShell to fully solve these problems, but it makes sense to make accommodations for these exceptions, as long as they are based on general rules (rather than individual exceptions) that are easy to conceptualize and document.

I believe it is vital to make these accommodations as part of the PSNativeCommandArgumentPassing experimental feature implemented in PR #14692 in order to solve the vast majority of quoting headaches once and for all.
They are detailed below.

For the remaining, edge cases there is:

• --% for console applications, or, preferably, because it has fewer limitations and enables use of PowerShell variable values and expressions via string interpolation, cmd /c "<cmd.exe command line>".
• Start-Process for GUI-subsystem applications with a CLI such as msiexec, which allows you to fully control the process command line by passing a single string to -ArgumentList (in a pinch you can also use it with console applications, but you lose stream integration).

Proposed technical implementation details

After PowerShell's own parsing, once the array of verbatim arguments - stripped of $nulls - to pass on is available:

• On Unix-like platform:

  • Pass that array to .ArgumentList - that is all that is ever needed.

• On Windows:

  • Except for the cases detailed below, also pass that array to .ArgumentList - behind the scenes; .NET then performs the necessary re-encoding based on the C/C++ conventions for us, and any conventional CLI should interpret the result correctly.

  • The following exceptions may apply independently or in combination, and they require manual re-encoding by PowerShell (with assignment to .Arguments, as currently):

    • A current behavior that must be retained - i.e. no escaping of embedded " must be performed - is the very specific case of cmd.exe being called directly, with either the /c or the /k option followed by a single argument (with spaces) representing a cmd.exe command line in full.

      • See here for details, including the proposal for an optional additional accommodation that would make sense, namely to robustly support passing the command line following /c or /k as multiple arguments, by transforming it into a single-argument, double-quoted-overall form.
      • Implementation-wise, we'd get this additional accommodation almost for free, because we need it behind the scenes for calling batch files anyway, so as to support reliable exit-code reporting - see next point.

    • If the target command is a batch file:

      • use "" (rather than \") to escape embedded verbatim " (and ensure enclosure in syntactic "...", even if the value has no spaces)
      • "..."-enclose any argument that contain no spaces (such arguments are normally not quoted) but contain any of the following cmd.exe metacharacters: & | < > ^ , ; (while , and ; have no impact on arguments pass-through with %*, they serve as argument separators in intra-batch file argument parsing; this also applies to =, but, unfortunately, passing something like FOO=bar as "FOO=bar" conflicts with the accommodation for msiexec-style CLIs below).
      • Additionally, for reliable exit-code reporting, call the batch file via cmd /c "<batch-file> ... & exit" rather than directly; see below for the detailed rationale.
        • This means:
          • Make cmd.exe the executable.
          • Pass a single argument string /c "<batch-file> ... & exit", where <batch-file> path may need to be double-quoted and ... represents the space-joined list of the arguments quoted based on the rules above. Again, no escaping of any " characters ending up in the overall "..." string passed to /c need or must be performed.
          • Note that on Windows versions before Windows 10, such a call fails if the batch-file path itself must be double-quoted (typically, due to containing spaces); the workaround is to determine the short (8.3) form of that path - which by definition doesn't require double-quoting - and use that instead.

    • Irrespective of the target executable, if any of the arguments have the form of a misexec-style partial-quoting argument, apply double-quoting only to the "value" part (the part after the separator):

      • Specifically, if an argument (a) matches regex ^([/-]\w+[=:]|\w+=)(.+)$, and (b) the part after = or : requires double-quoting (either due to containing spaces and/or an embedded " and/or, in the case of a batch file containing cmd.exe metacharacters), leave the part up to and including = or : unquoted, and double-quote only the remaining part.
      • Examples:
        • The following PowerShell arguments:
          • FOO='bar none', -foo:$value (with $value containing verbatim bar none), /foo:bar` none (and even quoted-in-full variants 'FOO=bar none', ....)
        • would end up in the .Arguments command line as follows:
          • FOO="bar none", -foo:"bar none", /foo:"bar none"

    • If the target executable is a WSH CLI - cscript.exe or wscript.exe - or the filename extension is one of the following WSH-associated extensions listed by default in $env:PATHEXT (which makes them directly executable): .vbs .vbe .js .jse .wsf .wsh, use "" rather than \" to escape " characters embedded in arguments; again see below for details.

---

Again, these are reasonable accommodations to make, which:

• allow users to focus solely on PowerShell's syntax
• should make the vast majority of calls just work.
• are easy to conceptualize and document - the proposed tracing should help too.

I invite everyone to scrutinize these accommodations to see if they're complete, overzealous, ...

This is a chance to finally cure all native quoting / argument-passing headaches - even if only by opt-in.

(To experiment with the proposed behaviors up front (based on my personal implementation that sits on top of the current behavior), you can use Install-Module Native and prepend ie to command lines; if the proposed changes are implemented, such a stopgap will no longer be necessary, although it can still help on earlier versions.)

[iSazonov]

I'm delighted that @JamesWTruher implemented #14692!

I hope we will not stop half way and finally close this story by fully implementing the proposal.

[mklement0]

As it turns out, there's another accommodation worth making for batch files with respect to exit codes (now implemented in v1.3.1 of the Native module):

• Direct invocation of batch files doesn't report their exit code, as reflected in $LASTEXITCODE, reliably; e.g.:

# Create a (temporary) batch file that provokes an error with an unsupported whoami.exe option, 
# and exits with `exit /b` *without an explicit argument* with the intent to *pass whoami.exe's exit code through*.
'@echo off & whoami -nosuch 2>NUL || exit /b' | Set-Content test.cmd

# Invoke the batch file and report the exit code.
PS> .\test.cmd; $LASTEXITCODE
0   # !! whoami.exe's exit code, 1, was NOT passed through.

• The workaround is to use the following invocation in lieu of .\test.cmd: cmd /c ".\test.cmd & exit", as detailed in this Stack Overflow post.

Note that inside a cmd.exe session, %ERRORLEVEL% is set to 1 after calling the batch file.

To be clear: The problem lies with cmd.exe, but it is yet another accommodation we can make to improve the robustness of native-program calls.