tool: add "variable" support

Add support for command line variables. Set variables with --variable
name=content or --variable name@file (where "file" can be stdin if set
to a single dash (-)).

Variable content is expanded in option parameters using "{{name}}"
(without the quotes) if the option name is prefixed with
"--expand-". This gets the contents of the variable "name" inserted, or
a blank if the name does not exist as a variable. Insert "{{" verbatim
in the string by prefixing it with a backslash, like "\\{{".

Import an environment variable with --variable %name. It makes curl exit
with an error if the environment variable is not set. It can also rather
get a default value if the variable does not exist, using =content or
@file like shown above.

Example: get the USER environment variable into the URL:

 --variable %USER
 --expand-url = "https://example.com/api/{{USER}}/method"

When expanding variables, curl supports a set of functions that can make
the variable contents more convenient to use. It can trim leading and
trailing white space with "trim", output the contents as a JSON quoted
string with "json", URL encode it with "url" and base 64 encode it with
"b64". To apply functions to a variable expansion, add them colon
separated to the right side of the variable. They are then performed in
a left to right order.

Example: get the contents of a file called $HOME/.secret into a variable
called "fix". Make sure that the content is trimmed and percent-encoded
sent as POST data:

  --variable %HOME=/home/default
  --expand-variable fix@{{HOME}}/.secret
  --expand-data "{{fix:trim:url}}"
  https://example.com/

Documented. Many new test cases.

Co-brainstormed-by: Emanuele Torre
Assisted-by: Jat Satiro
Closes #11346

Author: bagder

docs/cmdline-opts/Makefile.inc

@@ -275,6 +275,7 @@ DPAGES = \
   use-ascii.d \
   user-agent.d \
   user.d \
+  variable.d \
   verbose.d \
   version.d \
   write-out.d \

docs/cmdline-opts/config.d

@@ -21,12 +21,12 @@ if so, the colon or equals characters can be used as separators. If the option
 is specified with one or two dashes, there can be no colon or equals character
 between the option and its parameter.
 
-If the parameter contains whitespace (or starts with : or =), the parameter
-must be enclosed within quotes. Within double quotes, the following escape
-sequences are available: \\\\, \\", \\t, \\n, \\r and \\v. A backslash
-preceding any other letter is ignored.
+If the parameter contains whitespace or starts with a colon (:) or equals sign
+(=), it must be specified enclosed within double quotes (\&"). Within double
+quotes the following escape sequences are available: \\\\, \\", \\t, \\n, \\r
+and \\v. A backslash preceding any other letter is ignored.
 
-If the first column of a config line is a '#' character, the rest of the line
+If the first non-blank column of a config line is a '#' character, that line
 will be treated as a comment.
 
 Only write one option per physical line in the config file. A single line is

docs/cmdline-opts/page-header

@@ -97,6 +97,48 @@ that getting many files from the same server do not use multiple connects /
 handshakes. This improves speed. Connection re-use can only be done for URLs
 specified for a single command line invocation and cannot be performed between
 separate curl runs.
+.SH VARIABLES
+Starting in curl 8.3.0, curl supports command line variables. Set variables
+with --variable name=content or --variable name@file (where "file" can be
+stdin if set to a single dash (-)).
+
+Variable contents can expanded in option parameters using "{{name}}" (without
+the quotes) if the option name is prefixed with "--expand-". This gets the
+contents of the variable "name" inserted, or a blank if the name does not
+exist as a variable. Insert "{{" verbatim in the string by prefixing it with a
+backslash, like "\\{{".
+
+You an access and expand environment variables by first importing them. You
+can select to either require the environment variable to be set or you can
+provide a default value in case it is not already set. Plain --variable %name
+imports the variable called 'name' but exits with an error if that environment
+variable is not alreadty set. To provide a default value if it is not set, use
+--variable %name=content or --variable %name@content.
+
+Example. Get the USER environment variable into the URL, fail if USER is not
+set:
+
+ --variable '%USER'
+ --expand-url = "https://example.com/api/{{USER}}/method"
+
+When expanding variables, curl supports a set of functions that can make the
+variable contents more convenient to use. It can trim leading and trailing
+white space with "trim", it can output the contents as a JSON quoted string
+with "json" and it can URL encode the string with "urlencode". You apply
+function to a variable expansion, add them colon separated to the right side
+of the variable. Variable content holding null bytes that are not encoded when
+expanded, will cause error.
+
+Exmaple: get the contents of a file called $HOME/.secret into a variable
+called "fix". Make sure that the content is trimmed and percent-encoded sent
+as POST data:
+
+  --variable %HOME
+  --expand-variable fix@{{HOME}}/.secret
+  --expand-data "{{fix:trim:urlencode}}"
+  https://example.com/
+
+Command line variables and expansions were added in in 8.3.0.
 .SH OUTPUT
 If not told otherwise, curl writes the received data to stdout. It can be
 instructed to instead save that data into a local file, using the --output or

docs/cmdline-opts/variable.d

@@ -0,0 +1,50 @@
+c: Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al.
+SPDX-License-Identifier: curl
+Long: variable
+Arg: <[%]name=text/@file>
+Help: Set variable
+Category: curl
+Example: --variable name=smith $URL
+Added: 8.3.0
+See-also: config
+Multi: append
+---
+Set a variable with "name=content" or "name@file" (where "file" can be stdin
+if set to a single dash (-)). The name is a case sensitive identifier that
+must consist of no other letters than a-z, A-Z, 0-9 or underscore. The
+specified content is then associated with this identifier.
+
+The name must be unique within a command line invoke, setting the same
+variable name again will be ignored.
+
+The contents of a variable can be referenced in a later command line option
+when that option name is prefixed with "--expand-", and the name is used as
+"{{name}}" (without the quotes).
+
+--variable can import environment variables into the name space. Opt to either
+require the environment variable to be set or provide a default value for the
+variable in case it is not already set.
+
+--variable %name imports the variable called 'name' but exits with an error if
+that environment variable is not alreadty set. To provide a default value if
+the environment variable is not set, use --variable %name=content or
+--variable %name@content. Note that on some systems - but not all -
+environment variables are case insensitive.
+
+When expanding variables, curl supports a set of functions that can make the
+variable contents more convenient to use. You apply a function to a variable
+expansion by adding a colon and then list the desired functions in a
+comma-separted list that is evaluated in a left-to-right order. Variable
+content holding null bytes that are not encoded when expanded, will cause
+error.
+
+These are functions that can help you get the value inserted more
+conveniently.
+
+"trim" removes all leading and trailing white space.
+
+"json" outputs the content using JSON string quoting rules.
+
+"url" shows the content URL (percent) encoded.
+
+"b64" expands the variable base64 encoded

docs/options-in-versions

@@ -261,6 +261,7 @@
 --use-ascii (-B)                     5.0
 --user (-u)                          4.0
 --user-agent (-A)                    4.5.1
+--variable                           8.3.0
 --verbose (-v)                       4.0
 --version (-V)                       4.0
 --write-out (-w)                     6.5

lib/base64.c

@@ -32,13 +32,14 @@
   !defined(CURL_DISABLE_POP3) || \
   !defined(CURL_DISABLE_IMAP) || \
   !defined(CURL_DISABLE_DOH) || defined(USE_SSL)
-
-#include "urldata.h" /* for the Curl_easy definition */
+#include "curl/curl.h"
 #include "warnless.h"
 #include "curl_base64.h"
 
 /* The last 2 #include files should be in this order */
+#ifdef BUILDING_LIBCURL
 #include "curl_memory.h"
+#endif
 #include "memdebug.h"
 
 /* ---- Base64 Encoding/Decoding Table --- */

lib/curl_base64.h

@@ -24,11 +24,18 @@
  *
  ***************************************************************************/
 
+#ifndef BUILDING_LIBCURL
+/* this renames functions so that the tool code can use the same code
+   without getting symbol collisions */
+#define Curl_base64_encode(a,b,c,d) curlx_base64_encode(a,b,c,d)
+#define Curl_base64url_encode(a,b,c,d) curlx_base64url_encode(a,b,c,d)
+#define Curl_base64_decode(a,b,c) curlx_base64_decode(a,b,c)
+#endif
+
 CURLcode Curl_base64_encode(const char *inputbuff, size_t insize,
                             char **outptr, size_t *outlen);
 CURLcode Curl_base64url_encode(const char *inputbuff, size_t insize,
                                char **outptr, size_t *outlen);
 CURLcode Curl_base64_decode(const char *src,
                             unsigned char **outptr, size_t *outlen);
-
 #endif /* HEADER_CURL_BASE64_H */

projects/generate.bat

@@ -217,6 +217,7 @@ rem
       call :element %1 lib "curl_multibyte.c" %3
       call :element %1 lib "version_win32.c" %3
       call :element %1 lib "dynbuf.c" %3
+      call :element %1 lib "base64.c" %3
     ) else if "!var!" == "CURL_SRC_X_H_FILES" (
       call :element %1 lib "config-win32.h" %3
       call :element %1 lib "curl_setup.h" %3
@@ -228,6 +229,7 @@ rem
       call :element %1 lib "curl_multibyte.h" %3
       call :element %1 lib "version_win32.h" %3
       call :element %1 lib "dynbuf.h" %3
+      call :element %1 lib "curl_base64.h" %3
     ) else if "!var!" == "CURL_LIB_C_FILES" (
       for /f "delims=" %%c in ('dir /b ..\lib\*.c') do call :element %1 lib "%%c" %3
     ) else if "!var!" == "CURL_LIB_H_FILES" (

src/CMakeLists.txt

@@ -63,7 +63,7 @@ endif()
 
 # CURL_CFILES, CURLX_CFILES, CURL_HFILES come from Makefile.inc
 if(BUILD_STATIC_CURL)
-  set(CURLX_CFILES ../lib/dynbuf.c)
+  set(CURLX_CFILES ../lib/dynbuf.c ../lib/base64.c)
 endif()
 
 add_executable(

src/Makefile.inc

@@ -32,13 +32,14 @@
 # libcurl has sources that provide functions named curlx_* that aren't part of
 # the official API, but we re-use the code here to avoid duplication.
 CURLX_CFILES = \
+  ../lib/base64.c \
+  ../lib/curl_multibyte.c \
+  ../lib/dynbuf.c \
+  ../lib/nonblock.c \
   ../lib/strtoofft.c \
   ../lib/timediff.c \
-  ../lib/nonblock.c \
-  ../lib/warnless.c \
-  ../lib/curl_multibyte.c \
   ../lib/version_win32.c \
-  ../lib/dynbuf.c
+  ../lib/warnless.c
 
 CURLX_HFILES = \
   ../lib/curl_setup.h \
@@ -91,7 +92,8 @@ CURL_CFILES = \
   tool_vms.c \
   tool_writeout.c \
   tool_writeout_json.c \
-  tool_xattr.c
+  tool_xattr.c \
+  var.c
 
 CURL_HFILES = \
   slist_wc.h \
@@ -135,7 +137,8 @@ CURL_HFILES = \
   tool_vms.h \
   tool_writeout.h \
   tool_writeout_json.h \
-  tool_xattr.h
+  tool_xattr.h \
+  var.h
 
 CURL_RCFILES = curl.rc
 

src/tool_cfgable.h

@@ -26,6 +26,7 @@
 #include "tool_setup.h"
 #include "tool_sdecls.h"
 #include "tool_urlglob.h"
+#include "var.h"
 
 struct GlobalConfig;
 
@@ -322,6 +323,7 @@ struct GlobalConfig {
   unsigned short parallel_max; /* MAX_PARALLEL is the maximum */
   bool parallel_connect;
   char *help_category;            /* The help category, if set */
+  struct var *variables;
   struct OperationConfig *first;
   struct OperationConfig *current;
   struct OperationConfig *last;   /* Always last in the struct */