Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Try to make argument with quotes for external command better #6420

Merged
merged 2 commits into from
Aug 26, 2022

Conversation

WindSoilder
Copy link
Collaborator

@WindSoilder WindSoilder commented Aug 26, 2022

Description

Fixes: #6399

Detail

In general, if one external arg is quoted, nu won't remove inner extra quotes.

Else, if one external arg is not quoted, nu will try to remove inner extra quotes, which is implemented in #5846

Summarized:

  1. ^echo --aa="bbbb" ==> --aa=bbbb
  2. ^echo "--aa='bbbb'" ==> --aa='bbbb'
  3. ^echo '--aa="bbb"' ==> --aa="bbb"

For complex argument mentioned in #6399 (comment):

let dump_command = "PGPASSWORD='db_secret' pg_dump -Fc -h 'db.host' -p '$db.port' -U postgres -d 'db_name' > '/tmp/dump_name'"

If we want to use $dump_command as an argument, we can't use bare $dump_command, because it's intepreted as:

PGPASSWORD='db_secret' pg_dump -Fc -h 'db.host' -p '$db.port' -U postgres -d 'db_name' > '/tmp/dump_name'

We need to surrand $dump_command with ", and using them with String interpolation.

^echo $""($dump_command)""

Ok...have to admit it's a bit annoying

or just

^echo "PGPASSWORD='db_secret' pg_dump -Fc -h 'db.host' -p '$db.port' -U postgres -d 'db_name' > '/tmp/dump_name'"

Tests

Make sure you've done the following:

  • Add tests that cover your changes, either in the command examples, the crate/tests folder, or in the /tests folder.
  • Try to think about corner cases and various ways how your changes could break. Cover them with tests.
  • If adding tests is not possible, please document in the PR body a minimal example with steps on how to reproduce so one can verify your change works.

Make sure you've run and fixed any issues with these commands:

  • cargo fmt --all -- --check to check standard code formatting (cargo fmt --all applies these changes)
  • cargo clippy --workspace --features=extra -- -D warnings -D clippy::unwrap_used -A clippy::needless_collect to check that you're using the standard code style
  • cargo test --workspace --features=extra to check that all the tests pass

@WindSoilder WindSoilder changed the title Try to make argument with quotes for external command works better Try to make argument with quotes for external command better Aug 26, 2022
@fdncred
Copy link
Collaborator

fdncred commented Aug 26, 2022

let's try this to see if it helps folks with quoting.

@fdncred fdncred merged commit fbae137 into nushell:main Aug 26, 2022
@WindSoilder WindSoilder deleted the string_quote branch August 27, 2022 00:18
dheater pushed a commit to dheater/nushell that referenced this pull request Sep 2, 2022
@fdncred
Copy link
Collaborator

fdncred commented Sep 12, 2022

@WindSoilder do you know if these changes were reverted somewhere? I can't get this stuff working right anymore.

https://gist.github.com/fdncred/c55a7a83f41bb6b3aa48381dd44ed4f8
like this is broken
image
and this is broken
image

@fdncred
Copy link
Collaborator

fdncred commented Sep 12, 2022

oof, this test doesn't work any longer, in Windows:

^echo '--aa="bbb"'
--aa=\bbb"

@WindSoilder
Copy link
Collaborator Author

WindSoilder commented Sep 12, 2022

oof, this test doesn't work any longer:

^echo '--aa="bbb"'
--aa=\bbb"

This works for me in the latest main.

let selected_log_lines = (
git log [--format=format:($log_fmt_str) --date=short --color=always] |
fzf [--ansi --multi --tiebreak=index --preview='git show --color=always --stat --patch {1}']
)

For something like this, because we pass list to fzf, I think we need to quote the whole --preview='gitxxx'], like this:

let selected_log_lines = (
git log [--format=format:($log_fmt_str) --date=short --color=always] |
fzf [--ansi --multi --tiebreak=index --preview='git show --color=always --stat --patch {1}']
)

@fdncred
Copy link
Collaborator

fdncred commented Sep 12, 2022

I was testing on windows. This works on Mac but not Windows.

^echo '--aa="bbb"'

neither of the selected_log_lines examples work on my Mac on the latest main.

@WindSoilder
Copy link
Collaborator Author

neither of the selected_log_lines examples work on my Mac on the latest main.

I'm not sure if it's introduced by this pr, I'll install fzf and try on mac today.

@WindSoilder
Copy link
Collaborator Author

WindSoilder commented Sep 13, 2022

Hi @fdncred , there are two ways to work around this:

  1. pass list to fzf:
git log [--format=format:($log_fmt_str) --date=short --color=always] | 
fzf [--ansi --multi --tiebreak=index  "--preview=git show --color=always --stat --patch {1}"]

We need to "--preview=git show --color=always --stat --patch {1}", rather than --preview="git show --color=always --stat --patch {1}"

If we only quote from git show part, it will show a string git show --color=always --stat --patch {1}, because the quote " is also passed to fzf command.

The rule is commented here:

Value::List { vals, .. } => {
// turn all the strings in the array into params.
// Example: one_arg may be something like ["ls" "-a"]
// convert it to "ls" "-a"
for v in vals {
spanned_args.push(value_as_spanned(v)?);
// for arguments in list, it's always treated as a whole arguments
arg_keep_raw.push(true);
}
}

2. don't pass list to fzf:

git log [--format=format:($log_fmt_str) --date=short --color=always] | 
fzf --ansi --multi --tiebreak=index  --preview='git show --color=always --stat --patch {1}'

Could you please check if it works for you?

@WindSoilder
Copy link
Collaborator Author

And,

oof, this test doesn't work any longer:

^echo '--aa="bbb"'
--aa=\bbb"

Here is an issue mentioned about this #6465, I'm sorry my computer is broken, I can't verify if it was introduced by this pr.

@fdncred
Copy link
Collaborator

fdncred commented Sep 13, 2022

Could you please check if it works for you?

Both of your solutions work in Mac. I'll test in Windows in a little while.

@fdncred
Copy link
Collaborator

fdncred commented Sep 13, 2022

Both of those work on Windows too. I'd still like to fix this problem though.

This does not work on windows

^echo '--aa="bbb"'
--aa=\bbb"

@evelant
Copy link

evelant commented Sep 14, 2022

As a new nu user I was bitten by this. Very unexpected behavior. I was just trying to use ripgrep and this happened:

rg -e "npm_package_[\"'`]" -. --no-ignore --no-messages -t js -t ts -t json -M 100 ~/
regex parse error:
    npm_package_[\"'`]

From my perspective as a newbie that looks like "escape sequences don't work". Maybe a better solution than trying to cleverly handle quotes could be to add a custom string delimiter option so users can pick a delimiter that won't conflict with whatever they need to put in a string. That or have add string delimiter option that's very unlikely to conflict with anything such as ,,,string with "all sorts" of 'quotes', and `no conflicts`.,,,

IanManske pushed a commit that referenced this pull request May 23, 2024
This PR is a complete rewrite of `run_external.rs`. The main goal of the
rewrite is improving readability, but it also fixes some bugs related to
argument handling and the PATH variable (fixes
#6011).

I'll discuss some technical details to make reviewing easier.

## Argument handling

Quoting arguments for external commands is hard. Like, *really* hard.
We've had more than a dozen issues and PRs dedicated to quoting
arguments (see Appendix) but the current implementation is still buggy.

Here's a demonstration of the buggy behavior:

```nu
let foo = "'bar'"
^touch $foo            # This creates a file named `bar`, but it should be `'bar'`
^touch ...[ "'bar'" ]  # Same
```

I'll describe how this PR deals with argument handling.

First, we'll introduce the concept of **bare strings**. Bare strings are
**string literals** that are either **unquoted** or **quoted by
backticks** [^1]. Strings within a list literal are NOT considered bare
strings, even if they are unquoted or quoted by backticks.

When a bare string is used as an argument to external process, we need
to perform tilde-expansion, glob-expansion, and inner-quotes-removal, in
that order. "Inner-quotes-removal" means transforming from
`--option="value"` into `--option=value`.

## `.bat` files and CMD built-ins

On Windows, `.bat` files and `.cmd` files are considered executable, but
they need `CMD.exe` as the interpreter. The Rust standard library
supports running `.bat` files directly and will spawn `CMD.exe` under
the hood (see
[documentation](https://doc.rust-lang.org/std/process/index.html#windows-argument-splitting)).
However, other extensions are not supported [^2].

Nushell also supports a selected number of CMD built-ins. The problem
with CMD is that it uses a different set of quoting rules. Correctly
quoting for CMD requires using
[Command::raw_arg()](https://doc.rust-lang.org/std/os/windows/process/trait.CommandExt.html#tymethod.raw_arg)
and manually quoting CMD special characters, on top of quoting from the
Nushell side. ~~I decided that this is too complex and chose to reject
special characters in CMD built-ins instead [^3]. Hopefully this will
not affact real-world use cases.~~ I've implemented escaping that works
reasonably well.

## `which-support` feature

The `which` crate is now a hard dependency of `nu-command`, making the
`which-support` feature essentially useless. The `which` crate is
already a hard dependency of `nu-cli`, and we should consider removing
the `which-support` feature entirely.

## Appendix

Here's a list of quoting-related issues and PRs in rough chronological
order.

* #4609
* #4631
* #4601
  * #5846
* #5978
  * #6014
* #6154
  * #6161
* #6399
  * #6420
  * #6426
* #6465
* #6559
  * #6560

[^1]: The idea that backtick-quoted strings act like bare strings was
introduced by Kubouch and briefly mentioned in [the language
reference](https://www.nushell.sh/lang-guide/chapters/strings_and_text.html#backtick-quotes).

[^2]: The documentation also said "running .bat scripts in this way may
be removed in the future and so should not be relied upon", which is
another reason to move away from this. But again, quoting for CMD is
hard.

[^3]: If anyone wants to try, the best resource I found on the topic is
[this](https://daviddeley.com/autohotkey/parameters/parameters.htm).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

External command parameters get mangled
3 participants