exec-env/exec-file: support "--" to separate command to run

[ancoron]

An issue I am facing regularly is that I need to create an actual command that is being executed with exec-env or exec-file with variables. Due to the need to put the whole command line into a single argument it becomes very and error-prone due to quoting issues.

A simple example:

my_url="${api_baseuri}/${resource_path}"
sops exec-env test.env 'curl -s --user "${API_USER}:${API_PASSWORD}" "'"${my_url}"'"'

If sops could introduce special support for the -- (double-dash) argument, we could say that everything after it becomes a part of the command to execute and it could look like this instead:

sops exec-env test.env -- curl -s --user '${API_USER}:${API_PASSWORD}' "${my_url}"

For manual/interactive command building, this would also allow to auto-complete file names in the shell we're working in.

Of course, the user would still need to ensure that variable expansion doesn't happen in the shell (using single quotes instead of double-quotes), but that has less error potential than quoting (at least for me).

[felixfontein]

Hmm, this is not so trivial to implement. Simply passing on the array of strings from the command line arguments on as a new command is simple, but that doesn't allow you to access further shell features (like ${API_USER}). But if you want to pass what has been handed in through another shell, we need to figure out how to handle other things that shell might interpret, such as ! or spcaes in arguments. For example, what should sops exec-env test.env -- foo "bar baz" "b'a'm" execute? The command foo with three parameters bar, baz, and b'a'm? Or the command foo with two arguments bar baz and b'a'm? And do you want to quote the single quotes in b'a'm? Or remove them as a shell would do?

[ancoron]

@felixfontein Thanx for the feedback and I agree, that behavior needs to be well-defined to clarify expectations on the user side.

Picking up your example:

sops exec-env test.env -- foo "bar baz" "b'a'm"

...should internally be creating a command with the following parts:

1. command/executable: foo
2. command arguments:
   • bar baz
   • b'a'm

To sops itself, the argument should present itself as follows:

1. exec-env
2. test.env
3. --
4. foo
5. bar baz
6. b'a'm

So its barely a pass-through of arguments with the exception of the first argument after -- to represent the command/executable.

If I had a shell script where I needed to produce an actual command line, I'd have something like this:

my_url="${api_baseuri}/${resource_path}"

curl_opts=(-s --fail-with-body --user '${API_USER}:${API_PASSWORD}')

# ...add other options depending on script behavior...

sops exec-env test.env -- curl "${curl_opts[@]}" "${my_url}"

Here, the "${curl_opts[@]}" is of course expanded by the shell into the individual arguments as presented in the array and due to single-quoting the user option value, it will expand neither API_USER nor API_PASSWORD, which then shall be handled by sops.

At least that's my take on this issue.

[uasi]

@ancoron @felixfontein Why not just make sops execute the given arguments, and use an actual shell if we need to expand variables, like sops exec-env test.env -- bash -c 'curl -s --user "${API_USER}:${API_PASSWORD}" "${my_url}"' ?

edit: Nevermind, it seems sops exec-env already runs the given command using shell. It would still be useful to support multiple arguments though.

[ancoron]

@uasi The main point of this issue is to support -- in the first place with a well-defined argument handling that doesn't get in the way.

In the description I've presented briefly what the main issue with SOPS is when it comes to dynamic command building as part of scripting.

In your example I'd still need to:

1. export variable my_url so the sops sees it
2. trust that sops exec-env will actually pass it down to the command
3. somehow assume that test.env does NOT contain variable my_url

The major issue is with variable expansion and control of WHERE it gets expanded without losing readability of the SOPS invocation code.

[joshbode]

My quick hack to work around not being able to pass multiple arguments to sops exec-env (until it is supported) is this wrapper function:

_sops_exec() {
  local FILE=$1 COMMAND=$2 ARGS
  shift 2

  ARGS=$(printf " %q" "$@")  # %q quotes the arguments properly
  sops exec-env "${FILE}" "${COMMAND}${ARGS}"
}
alias sops-exec=_sops_exec

which can be used like:

$ sops-exec ~/secrets.env foo --bar --baz x y z

[hraban]

Proposed fix in #1626

PS @joshbode fyi bash has that functionality built-in using the @Q modifier:

$ foo="Joe's   tavern"
$ echo "${foo@Q}"
'Joe'\''s   tavern'
$ echo ${foo@Q}
'Joe'\''s tavern'
$ args "${foo@Q}"
["'Joe'\\''s   tavern'"]
$ args ${foo@Q}
["'Joe'\\''s", "tavern'"]