curl CLI v8.11.1 fails to offer HTTP Basic auth specified in .netrc when invoked with --netrc-optional

[bzanin-wdc]

I did this

Given an executable test script with the following contents named test-curl which will ruthlessly shove aside any existing ~/.netrc file..:

#!/usr/bin/env bash
if ((BASH_VERSINFO < 4)); then
	printf '%s\n' 'we need at least bash 4.x (arrays) to function'
	exit 1
fi
case=${case:-'fail'}
  # change this value to 'pass' to demonstrate expected behaviour,
  # or set and export the environment variable "$case" to 'pass' or 'fail'
[[ ! "$case" == @(pass|fail) ]] && exit 1
  # no unsanitized input ever
printf '%s\n' 'machine api.github.com' 'password fake' > .netrc.case-pass
printf '%s\n' 'machine api.github.com' 'password fake' '' 'machine example.com' > .netrc.case-fail
mv -v .netrc ".netrc.bak.$$.$EPOCHSECONDS"
ln -v .netrc.case-"${case}" .netrc
opts_curl=( \
  '--url'    'https://api.github.com/meta'
    # Just a convenient endpoint that will answer whether or not it is
    # given a valid Authorization: HTTP header.
  '--output' '/dev/null'
    # Not really necessary, but arguably good form?
  '--disable'
    # ignore any existing ~/.curlrc file
  '--verbose'
    # make sure we can look for the Authorization: header we care about
    # in the stdout/stderr generated by curl
  '--head'
    # don't actually ask GitHub to send us the file contents since we
    # only care about the headers.  Is it worth saving like 147KB of
    # network data transfer?  Arguably.  Is it worth being polite about
    # requesting the transfer of data we won't use?  Yes, it is.
  '--netrc-optional'
    # key invocation flag for replicating this error case.  Specifying
    # a netrc file explicitly with --netrc .netrc.case-pass does not
    # replicate the failure behaviour.
)
curl "${opts_curl[@]}" 2>&1 | grep -E '^> Authorization:'

...then invoking this script with case='pass' ./test-curl replicates the expected and desired behaviour, whereas invoking it with case='fail' ./test-curl replicates the failure case.

I expected the following

Desired outcome with case='pass', with the Authorization: Basic ... header being sent with the HTTP request:

renamed '.netrc' -> '.netrc.bak.52898.1734455278'
'.netrc' => '.netrc.case-pass'
> Authorization: Basic OmZha2U=

Surprising outcome with case='fail', with the Authorization: Basic ... header absent from the HTTP request:

renamed '.netrc' -> '.netrc.bak.53651.1734455299'
'.netrc' => '.netrc.case-fail'

curl/libcurl version

curl 8.11.1 (aarch64-apple-darwin23.6.0) libcurl/8.11.1 OpenSSL/3.4.0 (SecureTransport) zlib/1.2.12 brotli/1.1.0 zstd/1.5.6 AppleIDN libssh2/1.11.1 nghttp2/1.64.0 librtmp/2.3
Release-Date: 2024-12-11
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

operating system

Darwin P2XYX6PHVX 23.6.0 Darwin Kernel Version 23.6.0: Wed Jul 31 20:49:39 PDT 2024; root:xnu-10063.141.1.700.5~1/RELEASE_ARM64_T6000 arm64 arm Darwin

[bagder]

I can reproduce in a new curl test case.

[bagder]

Is this a regression, do you know?

[gnomon-]

I do believe it is a regression, yes: .netrc files with multiple entries pulled in via --netrc-optional work in curl versions prior to 8.x, e.g. 7.88.1 as packaged in Debian Bookworm.

[bagder]

I just checked. I broke this in e9b9bba. It worked fine in 8.11.0.

[dfandrich]

Given that https://curl.se/docs/CVE-2024-11053.html recommends applying e9b9bba to fix a vulnerability, that advisory should probably be updated to recommend applying 9fce2c5 as well to avoid introducing a new regression.