Fuzz the escape function, add escaping for interpolation characters (…

…#152)

* Fuzz `escape` function on unix

Update the fuzz target to test both the `quote()` and `escape()`
functions (instead of just `quote()`). Fuzzing the `escape()` function
is roughly identical to fuzzing the `quote()` function, except that the
provided buffer is stripped of whitespace before echoing to ensure all
characters in the string are part of the first argument.

* Refactor index.fuzz.cjs to reduce duplication and fix Windows args

- Define `WHITESPACE_REGEX` to avoid rewriting the regular expression to
  (really) capture all whitespace.
- Update the error message for unexpected output so it's different for
  the two tested functions.
- Define the `options` for `quote()` and `escape()` prior to the resp.
  fuzz function as the value is the same for both.
-  Fix `prepareArg` for Windows cmd.exe when quoted=false

* Update unix escaping based on fuzzing

Update escaping for unix platforms based on the existing fuzz corpus.
The unix escape logic now expects a boolean value indicating whether
interpolation is enabled and if so will escape more characters. This is
needed when the `escape()` API functions is used because the result may
be used in a context where interpolation is enabled. Hence, main assumes
interpolation is enabled unless specified otherwise.

* Escape ")" for unix with interpolation

This escape relates to the escaping of "(" in that shells (tested Zsh) on
unix with interpolation interpret parenthesis in a special way.

* Escape "<" and ">" for unix with interpolation

These characters are used for redirecting input/output streams so should
be escaped.

* Escape "#" at the start of args for unix with interpolation

The "#" character has a special meaning at the start of an argument.

* Escape "|" in args for unix with interpolation

The "|" character means a logical or.

* Escape ";" in args for unix with interpolation

The ";" character denotes the end of a singular command, after which a
new command starts.

* Escape "&" in args for unix with interpolation

The "&" character means a logical and.

* Escape "*" and "?" in args for unix with interpolation

The characters  "*" and "?" can be used for string expansion if
there are no quotes.

In Zsh these characters are always expanded (if unsuccessful the
command will error). On bash and dash these characters will only
be expanded if possible (otherwise they will appear literally). As
the context in which Shescape operates is unknown, for both shell
styles the characters always need to be escaped.

* Escape leading "~" in args for unix with interpolation

The "~" character has a special meaning (the home directory) either:
1. as the only character in the argument (bash, dash)
2. at the start of an argument (Zsh)
Both cases can be efficiently escaped by prefix "~" with a backslash,
this works regardless of whether or not the argument is more than one
string in bash and dash.

* Escape "[" and "]" for Zsh with interpolation

Specifically in Zsh, "[" and "]" can be used for string expansion if
there are no quotes. So, the err on the safe side, they will always be
escaped when `interpolation` is true.

* Escape leading "=" for Zsh with interpolation

The "=" character has a special meaning at the start of an argument in
Zsh.

* Escape "{" and "}" for Zsh with interpolation

Specifically in Zsh, "{" and "}" can be used for string expansion if
there are no quotes. So, the err on the safe side, they will always be
escaped when `interpolation` is true.

* Fuzz `escape` function on PowerShell

Run the fuzz target with the existing corpus on PowerShell and fix all
problems that are uncovered.

* Escape single quote variants in PowerShell on Windows

Escape the following characters for PowerShell on Windows because it
will interpret these as regular single quotes.

- U+2018 (Left Single Quotation Mark)
- U+2019 (Right Single Quotation Mark)
- U+201B (Single High-Reversed-9 Quotation Mark)
- U+201A (Single Low-9 Quotation Mark)

* Escape "<" and ">" for Windows PowerShell with interpolation

These characters have special meaning when they appear at the beginning
of an argument. In the case of ">", it also has this meaning when it is
prefixed by "1", "2", "3", "4", "5", "6", or "*".

* Escape "@" for Windows PowerShell with interpolation

This character has special meaning when it appears at the beginning of
an argument.

* Escape "]" for Windows PowerShell with interpolation

This character has a special meaning when it appears at the beginning of
an argument.

* Escape "," for Windows PowerShell with interpolation

This character is used to separate commands on PowerShell.

* Escape leading "-" in PowerShell on Windows

If "-" is the first character in an argument in PowerShell on Windows it
has a special meaning, so it's always escaped.

* Escape leading ":" in PowerShell on Windows

If ":" is the first character in an argument in PowerShell on Windows it
has a special meaning, so it's always escaped if it is the first
character.

* Fuzz `escape`  function on cmd.exe

Run the fuzz target with the existing corpus on cmd.exe and fix all
problems that are uncovered.

* Escape "^" for Windows cmd.exe with interpolation

This character is used for escaping, so it must be escaped itself. As it
is used for escaping, it should be escaped first to prevent escaping the
"^" instances inserted to escape other characters.

* Update unit test suites for changes to unix.js and win.js

* Update unit test suites for changes to main.js

* Update TypeScript type definitions

* Update documentation for `escape` and `escapeAll`

* Update CHANGELOG

CHANGELOG.md

@@ -7,6 +7,8 @@ Versioning].
 
 ## [Unreleased]
 
+- Add escaping for Unix interpolation characters to `escape`/`escapeAll`.
+- Add escaping for Zsh wildcard characters to `escape`/`escapeAll`.
 - Update TypeScript type definitions.
 - Update type information in the documentation.
 

README.md

@@ -199,6 +199,11 @@ _dangerous_ characters.
 Calling `escape()` directly is not recommended unless you know what you're
 doing.
 
+The `options.interpolation` value should be set to `true` if using this function
+with the `exec` function, or when using `fork`, `spawn`, `execFile`, or similar,
+and setting `{ shell: true }` in the call options. If in doubt, set it to `true`
+explicitly.
+
 #### Example
 
 ```js
@@ -212,11 +217,12 @@ console.log(safeArg);
 
 #### Input-output
 
-| Input           | Type                | Required | Description                  |
-| --------------- | ------------------- | -------- | ---------------------------- |
-| `arg`           | `string`            | Yes      | The argument to escape.      |
-| `options`       | `Object`            | No       | The escape options.          |
-| `options.shell` | `string`, `boolean` | No       | The shell that will be used. |
+| Input                   | Type                | Required | Description                  |
+| ----------------------- | ------------------- | -------- | ---------------------------- |
+| `arg`                   | `string`            | Yes      | The argument to escape.      |
+| `options`               | `Object`            | No       | The escape options.          |
+| `options.interpolation` | `boolean`           | No       | Is interpolation enabled.    |
+| `options.shell`         | `string`, `boolean` | No       | The shell that will be used. |
 
 | Output    | Type     | Description           |
 | --------- | -------- | --------------------- |
@@ -231,6 +237,10 @@ console.log(safeArg);
 The `escapeAll` function takes as input an array of values, the arguments, and
 escapes any _dangerous_ characters in every argument.
 
+The `options.interpolation` value should be set to `true` if using this function
+with `fork`, `spawn`, `execFile`, or similar, and setting `{ shell: true }` in
+the call options. If in doubt, set it to `true` explicitly.
+
 #### Example
 
 ```js
@@ -244,11 +254,12 @@ console.log(safeArgs);
 
 #### Input-output
 
-| Input           | Type                | Required | Description                  |
-| --------------- | ------------------- | -------- | ---------------------------- |
-| `args`          | `string[]`          | Yes      | The arguments to escape.     |
-| `options`       | `Object`            | No       | The escape options.          |
-| `options.shell` | `string`, `boolean` | No       | The shell that will be used. |
+| Input                   | Type                | Required | Description                  |
+| ----------------------- | ------------------- | -------- | ---------------------------- |
+| `args`                  | `string[]`          | Yes      | The arguments to escape.     |
+| `options`               | `Object`            | No       | The escape options.          |
+| `options.interpolation` | `boolean`           | No       | Is interpolation enabled.    |
+| `options.shell`         | `string`, `boolean` | No       | The shell that will be used. |
 
 | Output     | Type       | Description            |
 | ---------- | ---------- | ---------------------- |

index.d.ts

@@ -4,14 +4,19 @@
  * @author Eric Cornelissen <ericornelissen@gmail.com>
  */
 
-interface Options {
+interface EscapeOptions {
+  readonly interpolation?: boolean;
   readonly shell?: boolean | string;
 }
 
-export function escape(arg: string, options?: Options): string;
+interface QuoteOptions {
+  readonly shell?: boolean | string;
+}
+
+export function escape(arg: string, options?: EscapeOptions): string;
 
-export function escapeAll(arg: string[], options?: Options): string[];
+export function escapeAll(arg: string[], options?: EscapeOptions): string[];
 
-export function quote(arg: string, options?: Options): string;
+export function quote(arg: string, options?: QuoteOptions): string;
 
-export function quoteAll(arg: string[], options?: Options): string[];
+export function quoteAll(arg: string[], options?: QuoteOptions): string[];

index.js

@@ -24,16 +24,23 @@ import * as main from "./src/main.js";
  *
  * @param {string} arg The argument to escape.
  * @param {Object} [options] The escape options.
+ * @param {string} [options.interpolation=false] Is interpolation enabled.
  * @param {string} [options.shell] The shell to escape the argument for.
  * @returns {string} The escaped argument.
  * @throws {TypeError} The argument is not stringable.
  * @since 0.1.0
  */
 export function escape(arg, options = {}) {
-  const shell = options.shell;
+  const { interpolation, shell } = options;
   const env = process.env;
   const platform = os.platform();
-  return main.escapeShellArgByPlatform(arg, platform, env, shell);
+  return main.escapeShellArgByPlatform(
+    arg,
+    platform,
+    env,
+    shell,
+    interpolation
+  );
 }
 
 /**
@@ -45,6 +52,7 @@ export function escape(arg, options = {}) {
  *
  * @param {string[]} args The arguments to escape.
  * @param {Object} [options] The escape options.
+ * @param {string} [options.interpolation=false] Is interpolation enabled.
  * @param {string} [options.shell] The shell to escape the arguments for.
  * @returns {string[]} The escaped arguments.
  * @throws {TypeError} One of the arguments is not stringable.
@@ -53,12 +61,18 @@ export function escape(arg, options = {}) {
 export function escapeAll(args, options = {}) {
   if (!Array.isArray(args)) args = [args];
 
-  const shell = options.shell;
+  const { interpolation, shell } = options;
   const env = process.env;
   const platform = os.platform();
   const result = [];
   for (const arg of args) {
-    const safeArg = main.escapeShellArgByPlatform(arg, platform, env, shell);
+    const safeArg = main.escapeShellArgByPlatform(
+      arg,
+      platform,
+      env,
+      shell,
+      interpolation
+    );
     result.push(safeArg);
   }
 

src/main.js

@@ -53,10 +53,17 @@ function getShell(platform, env, shell) {
  * @param {string} platform The platform to escape the argument for.
  * @param {Object} env The environment variables.
  * @param {string} [shell] The shell to escape the argument for, if any.
+ * @param {boolean} [interpolation=false] Is interpolation enabled.
  * @returns {string} The escaped argument.
  * @throws {TypeError} The argument is not stringable.
  */
-export function escapeShellArgByPlatform(arg, platform, env, shell) {
+export function escapeShellArgByPlatform(
+  arg,
+  platform,
+  env,
+  shell,
+  interpolation = false
+) {
   if (!isStringable(arg)) {
     throw new TypeError(typeError);
   }
@@ -65,9 +72,9 @@ export function escapeShellArgByPlatform(arg, platform, env, shell) {
   const argAsString = arg.toString();
   switch (platform) {
     case win32:
-      return win.escapeShellArg(argAsString, shell);
+      return win.escapeShellArg(argAsString, shell, interpolation);
     default:
-      return unix.escapeShellArg(argAsString, shell);
+      return unix.escapeShellArg(argAsString, shell, interpolation);
   }
 }
 
@@ -85,7 +92,7 @@ export function escapeShellArgByPlatform(arg, platform, env, shell) {
  * @throws {TypeError} The argument is not stringable.
  */
 export function quoteShellArgByPlatform(arg, platform, env, shell) {
-  const safeArg = escapeShellArgByPlatform(arg, platform, env, shell);
+  const safeArg = escapeShellArgByPlatform(arg, platform, env, shell, false);
   switch (platform) {
     case win32:
       return `"${safeArg}"`;

src/unix.js

@@ -6,17 +6,58 @@
 
 import { shellRequiredError } from "./constants.js";
 
+/**
+ * Escape a shell argument when string interpolation is *disabled* (e.g. when
+ * the argument is surrounded by single quotes in bash-family shells).
+ *
+ * @param {string} arg The argument to escape.
+ * @returns {string} The escaped argument.
+ */
+function escapeShellArgNoInterpolation(arg) {
+  return arg.replace(/\u{0}/gu, "").replace(/'/g, `'\\''`);
+}
+
+/**
+ * Escape a shell argument when string interpolation is *enabled* (e.g. when
+ * the argument is surrounded by double quotes in bash-family shells).
+ *
+ * @param {string} arg The argument to escape.
+ * @param {string} shell The shell to escape the argument for.
+ * @returns {string} The escaped argument.
+ */
+function escapeShellArgWithInterpolation(arg, shell) {
+  let result = arg
+    .replace(/\u{0}/gu, "")
+    .replace(/\\/g, "\\\\")
+    .replace(/^(~|#)/g, "\\$1")
+    .replace(/(\*|\?)/gu, "\\$1")
+    .replace(/(\$|\;|\&|\|)/g, "\\$1")
+    .replace(/(\(|\)|\<|\>)/g, "\\$1")
+    .replace(/("|'|`)/g, "\\$1");
+
+  if (shell.endsWith("zsh")) {
+    result = result.replace(/^=/gu, "\\=").replace(/(\[|\]|\{|\})/g, "\\$1");
+  }
+
+  return result;
+}
+
 /**
  * Escape a shell argument.
  *
  * @param {string} arg The argument to escape.
  * @param {string} shell The shell to escape the argument for.
+ * @param {boolean} interpolation Is interpolation enabled.
  * @returns {string} The escaped argument.
  */
-export function escapeShellArg(arg, shell) {
+export function escapeShellArg(arg, shell, interpolation) {
   if (shell === undefined) throw new TypeError(shellRequiredError);
 
-  return arg.replace(/\u{0}/gu, "").replace(/'/g, `'\\''`);
+  if (interpolation) {
+    return escapeShellArgWithInterpolation(arg, shell);
+  } else {
+    return escapeShellArgNoInterpolation(arg);
+  }
 }
 
 /**

src/win.js

@@ -10,40 +10,68 @@ import { regexpPowerShell, shellRequiredError } from "./constants.js";
  * Escape a shell argument for use in CMD.
  *
  * @param {string} arg The argument to escape.
+ * @param {boolean} interpolation Is interpolation enabled.
  * @returns {string} The escaped argument.
  */
-function escapeShellArgsForCmd(arg) {
-  return arg.replace(/\u{0}/gu, "").replace(/"/g, `""`);
+function escapeShellArgsForCmd(arg, interpolation) {
+  let result = arg.replace(/\u{0}/gu, "");
+
+  if (interpolation) {
+    result = result
+      .replace(/\^/g, "^^")
+      .replace(/(<|>)/g, "^$1")
+      .replace(/(")/g, "^$1")
+      .replace(/(\&|\|)/g, "^$1");
+  } else {
+    result = result.replace(/"/g, `""`);
+  }
+
+  return result;
 }
 
 /**
  * Escape a shell argument for use in PowerShell.
  *
  * @param {string} arg The argument to escape.
+ * @param {boolean} interpolation Is interpolation enabled.
  * @returns {string} The escaped argument.
  */
-function escapeShellArgsForPowerShell(arg) {
-  return arg
+function escapeShellArgsForPowerShell(arg, interpolation) {
+  let result = arg
     .replace(/\u{0}/gu, "")
-    .replace(/("|“|”|„)/g, `$1$1`)
     .replace(/`/g, "``")
     .replace(/\$/g, "`$");
+
+  if (interpolation) {
+    result = result
+      .replace(/^((?:\*|[1-6])?)(>)/g, "$1`$2")
+      .replace(/^(<|@|#|-|\:|\])/g, "`$1")
+      .replace(/(,|\;|\&|\|)/g, "`$1")
+      .replace(/(\(|\)|\{|\})/g, "`$1")
+      .replace(/('|’|‘|‛|‚)/g, "`$1")
+      .replace(/("|“|”|„)/g, "`$1");
+  } else {
+    result = result.replace(/("|“|”|„)/g, "$1$1");
+  }
+
+  return result;
 }
 
 /**
  * Escape a shell argument.
  *
  * @param {string} arg The argument to escape.
  * @param {string} shell The shell to escape the argument for.
+ * @param {boolean} interpolation Is interpolation enabled.
  * @returns {string} The escaped argument.
  */
-export function escapeShellArg(arg, shell) {
+export function escapeShellArg(arg, shell, interpolation) {
   if (shell === undefined) throw new TypeError(shellRequiredError);
 
   if (regexpPowerShell.test(shell)) {
-    return escapeShellArgsForPowerShell(arg);
+    return escapeShellArgsForPowerShell(arg, interpolation);
   } else {
-    return escapeShellArgsForCmd(arg);
+    return escapeShellArgsForCmd(arg, interpolation);
   }
 }
 

test/common.js

@@ -6,7 +6,8 @@ export const unixPlatform = "linux";
 
 export const binSh = "/bin/sh";
 export const binBash = "/bin/bash";
-export const unixShells = [undefined, binSh, binBash];
+export const binZsh = "/bin/zsh";
+export const unixShells = [undefined, binSh, binBash, binZsh];
 
 export const unixEnv = {};
 

test/fuzz/corpus/1e37238c1a5f6c736d4b8c23bfc2a79734e73379ecd7ae4b91b5c19b0bce54fb

@@ -0,0 +1 @@
+ L:�\(;\

test/fuzz/corpus/266755dd5469c485f454d8323853c5de1afc8237ad3148e87da136f57fe98817

@@ -0,0 +1 @@
+��$L:�>

test/fuzz/corpus/3b60b38594678116b32f8ce69cc2adc067cf4cb47db7b0542807eb710e1b4382

@@ -0,0 +1 @@
+ �G��Q�L:�\(?}

test/fuzz/corpus/3ed325d1d68d7671458659f3ee7aa2f023ab307cb4c6b0a1a50970bc1cfa9ed4

@@ -0,0 +1 @@
+:X|]5

test/fuzz/corpus/720fb50e541a9b37787812a1aa27063f35abf5f0ad7a2b25226679955e24453b

@@ -0,0 +1 @@
+[9�\

test/fuzz/corpus/7ace431cb61584cb9b8dc7ec08cf38ac0a2d649660be86d349fb43108b542fa4

@@ -0,0 +1 @@
+~

test/fuzz/corpus/7af61f03a9bc5bac6b46555a60182ff1eb0ea509758aa0ce12611e7d629b2d8b

@@ -0,0 +1 @@
+@L:^�I�\(?\

test/fuzz/corpus/934fa52640ce75ee7ac58f1bf0772bc97735ae6d42ca5fc47173e5002b8f299a

@@ -0,0 +1 @@
+#��(�
"�3Ww��

test/fuzz/corpus/958c9df7f1fc53b0ba36c3042b416e212c4f5a3c5b845d486d4fd9c306462bee

@@ -0,0 +1 @@
+!*(?\ ��\

test/fuzz/corpus/a95606bd226e5874b431b611f302846fe02611a0bc9d6b333bbdc4da6f4256e8

@@ -0,0 +1 @@
+package.j?on

test/fuzz/corpus/cdbcae15105d6b781e620813c79c7e868740d4e9cc53ce6f5fcbbc12387adf4b

@@ -0,0 +1 @@
+*

test/fuzz/corpus/cfa73e24623488703e1e1a5f9002f63ba74d624cc22063ce60e4e586918079ee

@@ -0,0 +1 @@
+*>W�3
w�3
w��

test/fuzz/corpus/cfab074bdcae4ce4a18ea61959b5ce04024e7e31360b06324e263efd7af0e7e1

@@ -0,0 +1 @@
+p*

test/fuzz/corpus/e87ed9941d75ac1b46da9d20915705a75e9f5ca6bed48123b2d46a0b80df7f2d

@@ -0,0 +1 @@
+]5

test/fuzz/corpus/e9ce945769b352dbf4550f08e15ac066e8ce84f2c2befbcbe8c5188888ca8e48

@@ -0,0 +1 @@
+ L:�\(?\

test/fuzz/corpus/fac29151619609f07c3b649a22c353abe89a6ea1709d7e79f27e62abad201e07

@@ -0,0 +1 @@
+@L:�\(?\