tool: add command line variables

[bagder]

Support command line variables. Set variables with --variable name=content or --variable name@file (where "file" can be stdin if set to a single dash (-)).

Variable contents can be expanded in option parameters using {{name}}" if the option name is prefixed with --expand-. This gets the contents of the variable name inserted, or a blank if the name does not exist as a variable. Insert {{ verbatim in the string by prefixing it with a backslash, like \{{.

You an access and expand environment variables by first importing them. You can select to either require the environment variable to be set or you can provide a default value in case it is not already set. Plain --variable %name imports the variable called name but exits with an error if that environment variable is not already set. To provide a default value if it is not set, use --variable %name=content or --variable %name@content.

Example: get the USER environment variable into the URL. Fail if USER is not set:

--variable '%USER'
--expand-url = "https://example.com/api/{{USER}}/method"

When expanding variables, curl supports a set of functions that can make the variable contents more convenient to use. It can trim leading and trailing white space with trim, it can output the contents as a JSON quoted string with json and it can URL encode the string with urlencode. You apply the function(s) to a variable expansion, add them colon separated to the right side of the variable. They are then executed in a left to right order.

Example: get the contents of a file called $HOME/.secret into a variable called "fix". Make sure that the content is trimmed and percent-encoded sent as POST data. if HOME is not set, use "dummy" as default:

--variable %HOME=dummy
--expand-variable fix@{{HOME}}/.secret
--expand-data "{{fix:trim:urlencode}}"
https://example.com/

Documented. Eight new test cases.

• Tests
• Documentation
• torture test safe
• build + tests work on all platforms
• embedded null-byte in content safe
• positive user feedback

[jay]

I think 'encode' or 'urlencode' is easier to understand than 'percent' or %. --expand-data "{{fix:urlencode}}", also wouldn't it be easier if it could all be done in one step

edit:

Percent-encodes the entire contents.

ah, I didn't see that. Why would you percent encode the entire thing? Is there really a need for this type of functionality?

[emanuele6]

@jay percent urlencodes only the value of the variable when expanding it, you use it when you want to a variable as a path component of a URL for example

--variable "foo=$foo" --expand-url 'https://example.org/{{foo:percent}}'

If you don't use :percent you will expand the value of {{foo}} directly potentially adding queries, fragments, slashes, etc

You mostly use bare {{foo}} for passing full values like --expand-user-agent {{foo}} or --expand-url {{foo}}, or for --expand-data '@{{foo}}/file.txt' where you don't need to prepare the value for an injection. Then :percent when you need to inject into a URL, :json when you need inject a compent into a JSON payload, etc

[bagder]

To consider: expanding a variable that contains a null byte. What to do with that null byte if neither json nor percent are applied? Should we cut the content at the null? Seems illogical. Should we replace/encode it with something that can be displayed? If so, with what? A period?

[emanuele6]

Most UNIX shells (with few exceptions, e.g. ksh93 deletes everything after the first NUL) will remove all NUL bytes when reading the output of a command for $(), so "$(printf 'a\0b\0\0\0cdef')" => abcdef; that could be another option, I think it would be less surprising than adding dots to the value.
Bash also prints a warning when a NUL byte is removed this way (at most one warning per expansion) to let the user know it is happening.

Another option is to just fail with a fatal error if a user tries to expand a variable that contains NUL without using a function that escapes, which I honestly don't think is that horrible. Afterall, this can only happen if the user uses something like --variable foo@file (it cannot happen for environment variables or variables set with =), and they uses {{foo}} without :json/:percent which I think would be pretty niche; then if {{foo}} contains NUL in that case, the input is most definitely malformed.

[bagder]

After having slept on it, I believe I now favor the "fail with a fatal error" option. This will be the most helpful thing to users as it makes it immediately apparent what happens, as removing or replacing the null bytes will lead to more hard-to-debug errors to puzzle users.

If we want a replace option, we could instead implement a dedicated function for that purpose.

[eloydegen]

I think the documentation should include some note about the security risks. If an attacker is able to modify an environment variable it might be possible to exfiltrate data which might otherwise not be possible.

[bagder]

I think the documentation should include some note about the security risks. If an attacker is able to modify an environment variable it might be possible to exfiltrate data which might otherwise not be possible.

Providing clear documentation is of course valuable and a priority. Do you have any proposed wording?

An attacker that can change environment variables for curl is a serious attack already before this, since curl will acknowledge several variables outside of this new feature.

Also, this new feature is only expanding environment variables that is explicitly asked for, which is a reason we add this --expand- prefix thing.

[bagder]

Me and @emanuele6 brainstormed a bit more on IRC.

1. What if you want to force an environment variable to be set or
2. What if you want a default value in case the environment variable is not set?

We came up with this tweak to support both those things, that I think fits rather nicely:

• remove {{%name}} support, get back to a single name space where we need to "import" environment variables to be able to expand them

Import environment variables like this:

• --variable %name imports an environment variable, fails with an error if not set. It can then be expanded as normal with {{name}}.
• --variable %name=default imports an environment variable, or sets it to the default value. The default can then also be read from a file with --variable %name@default
  - name can not be the same name as an existing variable

[emanuele6]

• name can not be the same name as an existing variable

Personally I think you should be allowed to redefine a variable.

While --variable foo=bar --variable foo=baz is likely a mistake, it is possible that the curlrc is defining with common names like proxy, host or ua, and if a script tries to use --variable proxy= unknowingly, it will fail unexpectedly.

Since variables are only used by the user for --expand- options, and not by curl internally, I think it should be fine to allow redefining them.

[bagder]

I think it should be fine to allow redefining them.

It works for me. I'll amend.

[bagder]

thanks @jay!