Test 1700 fails inexplicably in 8.13.0 without the openssl tool present

[graywolf]

I did this

I tried to update my curl package to 8.13.0, however during running the test suite I got the following error:

***************************************** 
test 1700...[HTTP/2 GET with Upgrade:]

 1700: stdout FAILED:
--- log/check-expected	2025-04-02 16:56:06.689423971 +0000
+++ log/check-generated	2025-04-02 16:56:06.689423971 +0000
@@ -1,22 +0,0 @@
-HTTP/1.1 101 Switching Protocols[CR][LF]
-Connection: Upgrade[CR][LF]
-Upgrade: h2c[CR][LF]
-[CR][LF]
-HTTP/2 200 [CR][LF]
-date: Tue, 09 Nov 2010 14:49:00 GMT[CR][LF]
-last-modified: Tue, 13 Jun 2000 12:10:00 GMT[CR][LF]
-etag: "21025-dc7-39462498"[CR][LF]
-accept-ranges: bytes[CR][LF]
-content-length: 6[CR][LF]
-content-type: text/html[CR][LF]
-funny-head: yesyes[CR][LF]
-via: 1.1 nghttpx[CR][LF]
-[CR][LF]
--foo-[LF]
-HTTP/2 200 [CR][LF]
-date: Tue, 09 Nov 2010 14:49:00 GMT[CR][LF]
-content-length: 6[CR][LF]
-content-type: text/html[CR][LF]
-via: 1.1 nghttpx[CR][LF]
-[CR][LF]
--maa-[LF]

 - abort tests

I have tried bisecting but did not have much luck due to skipping some commit due to tests failing for other reasons.

There are only 'skip'ped commits left to test.
The first bad commit could be any of:
3b6c7142f6e86dc575a97fb9c9f678e5e5e1db5c
8cc05992a8575b713054ffb5e4a33c648ab9d9e6
8836e65967cd60c30f7b5f1d39f8019756d24e70
e95f509c66abdd88ae02e3243cdc217f19c4a330
a910f5ba6a97a84a37a610c04d888336786d1128
c8b0f0c9ad78eafc6c8f0005113de346ee797c21
0c1ad21f978c8f5acf3d0c1708d83a93635d9df3
7cb079ad1bcb90b8f6c8f3735a2694d21af08ab9
44341e736a3e2f7a2b25a774be3a9796e81abab9
ad99067d9295346be7b41215eccaf3839db0c60c
7be2c421bf2e9f71d2883e30cb98bae6a3b5e97d
We cannot bisect more!

Topologically first commit is 44341e736a3e2f7a2b25a774be3a9796e81abab9, and 44341e736a3e2f7a2b25a774be3a9796e81abab9~1 does build (and test) fine.

I would be happy to provide more information if you need any.

Full build output attached.

sy4ihiiy32ax3mfpyg5z785ak5r4gm-curl-full-8.12.1.drv.gz

I expected the following

I expected the tests to pass.

curl/libcurl version

curl 8.13.0

operating system

Linux host 6.12.20 #1 SMP PREEMPT_DYNAMIC 1 x86_64 GNU/Linux

Distribution: GNU Guix

[vszakats]

In these commit some may be safely discarded, leaving:

• make signal handler signal-safe: e95f509
• obey IOV_MAX in HTTP/3: a910f5b
• HTTPSRR and c-ares: 7cb079a
• generate certs dynamically and switch to EC-256: 0c1ad21 44341e7 7be2c42

Possibly the last one may be related if the nghttpx server doesn't like the new test certs.
1700 is definitely tested in CI on macOS, but I'm not sure about Linux. If it's this, I'd expect
more than just this single test failing, but hard to say.

Any details of your build configuration, env?

[graywolf]

I have extracted some information from the build log above.

The configure script shows the following:

configure: Configured to build curl/libcurl:

  Host setup:       x86_64-unknown-linux-gnu
  Install prefix:   /gnu/store/ygbrlrkn9zz2lxli8ini9ajqn02a4cjh-curl-full-8.12.1
  Compiler:         gcc
   CFLAGS:          -Werror-implicit-function-declaration -O2 -Wno-system-headers
   CFLAGS extras:   
   CPPFLAGS:        -D_GNU_SOURCE -isystem /gnu/store/cmzi8a17f44fvb55s77jd7d4r678w093-p11-kit-0.24.1/include/p11-kit-1 -isystem /gnu/store/cmzi8a17f44fvb55s77jd7d4r678w093-p11-kit-0.24.1/include/p11-kit-1
   LDFLAGS:         -L/gnu/store/1prv14v6jfnzzg7szm57690b7fr6sx33-zlib-1.3/lib -L/gnu/store/xm75fjkb61153yqlb547j0ff5ijkmwfc-brotli-1.0.9/lib -L/gnu/store/m05g4pzw906bg2pydbl74vrnvkmi9rbj-zstd-1.5.2-lib/lib -L/gnu/store/9mkcil1rl450r84hn1hcbny5pi5js8ig-gnutls-3.8.3/lib -L/gnu/store/6cmhhd01xc4r87k705m26ijd3l1klvf4-libpsl-0.21.1/lib -L/gnu/store/7347jb8c1c2h43bcrlnbmi3j0fkd101f-libssh2-1.10.0/lib -L/gnu/store/miq5g6nvnn992a3l0vwq97n5l4grxvyj-rtmpdump-2.4-0.c5f04a5/lib -L/gnu/store/9mkcil1rl450r84hn1hcbny5pi5js8ig-gnutls-3.8.3/lib -L/gnu/store/al613p11xv5w1xmnqn7ykw0x6d4b0539-nettle-3.9.1/lib -L/gnu/store/7xizylh3gi6sj23nz19q6xhvx2d50wvr-libidn2-2.3.4/lib -L/gnu/store/n8yhw99s1hizvj8gfay3jvv6nf0m4zq7-nghttp2-1.58.0-lib/lib -L/gnu/store/6fdym0rlnv5p485wpv7jjdlrfa0bangc-ngtcp2-1.11.0/lib -L/gnu/store/6fdym0rlnv5p485wpv7jjdlrfa0bangc-ngtcp2-1.11.0/lib -L/gnu/store/g5lixlsmp5hpn5j02yrmcajfq7lsbxn6-nghttp3-1.8.0/lib
     curl-config:   -L/gnu/store/xm75fjkb61153yqlb547j0ff5ijkmwfc-brotli-1.0.9/lib -L/gnu/store/m05g4pzw906bg2pydbl74vrnvkmi9rbj-zstd-1.5.2-lib/lib -L/gnu/store/9mkcil1rl450r84hn1hcbny5pi5js8ig-gnutls-3.8.3/lib -L/gnu/store/6cmhhd01xc4r87k705m26ijd3l1klvf4-libpsl-0.21.1/lib -L/gnu/store/7347jb8c1c2h43bcrlnbmi3j0fkd101f-libssh2-1.10.0/lib -L/gnu/store/miq5g6nvnn992a3l0vwq97n5l4grxvyj-rtmpdump-2.4-0.c5f04a5/lib -L/gnu/store/9mkcil1rl450r84hn1hcbny5pi5js8ig-gnutls-3.8.3/lib -L/gnu/store/al613p11xv5w1xmnqn7ykw0x6d4b0539-nettle-3.9.1/lib -L/gnu/store/7xizylh3gi6sj23nz19q6xhvx2d50wvr-libidn2-2.3.4/lib -L/gnu/store/n8yhw99s1hizvj8gfay3jvv6nf0m4zq7-nghttp2-1.58.0-lib/lib -L/gnu/store/6fdym0rlnv5p485wpv7jjdlrfa0bangc-ngtcp2-1.11.0/lib -L/gnu/store/6fdym0rlnv5p485wpv7jjdlrfa0bangc-ngtcp2-1.11.0/lib -L/gnu/store/g5lixlsmp5hpn5j02yrmcajfq7lsbxn6-nghttp3-1.8.0/lib
   LIBS:            -lnghttp3 -lngtcp2_crypto_gnutls -lngtcp2 -lnghttp2 -lidn2 -lrtmp -lz -lgmp -lgnutls -lhogweed -lnettle -lssh2 -lssh2 -lgsasl -lpsl -lnettle -lgnutls -L/gnu/store/b9kfblvwd0xx5jr8zzvz4ypa0936jh6v-mit-krb5-1.20/lib -lgssapi_krb5 -lldap -llber -lzstd -lzstd -lbrotlidec -lbrotlidec -lz

  curl version:     8.13.0-DEV
  SSL:              enabled (GnuTLS)
  SSH:              enabled (libssh2)
  zlib:             enabled
  brotli:           enabled (libbrotlidec)
  zstd:             enabled (libzstd)
  GSS-API:          enabled (MIT Kerberos/Heimdal)
  GSASL:            enabled
  TLS-SRP:          enabled
  resolver:         POSIX threaded
  IPv6:             enabled
  Unix sockets:     enabled
  IDN:              enabled (libidn2)
  Build docs:       enabled (--disable-docs)
  Build libcurl:    Shared=yes, Static=no
  Built-in manual:  enabled
  --libcurl option: enabled (--disable-libcurl-option)
  Verbose errors:   enabled (--disable-verbose)
  Code coverage:    disabled
  SSPI:             no      (--enable-sspi)
  ca cert bundle:   no
  ca cert path:     no
  ca cert embed:    no
  ca fallback:      no
  LDAP:             enabled (OpenLDAP)
  LDAPS:            enabled
  IPFS/IPNS:        enabled
  RTSP:             enabled
  RTMP:             enabled (librtmp)
  PSL:              enabled
  Alt-svc:          enabled (--disable-alt-svc)
  Headers API:      enabled (--disable-headers-api)
  HSTS:             enabled (--disable-hsts)
  HTTP1:            enabled (internal)
  HTTP2:            enabled (nghttp2)
  HTTP3:            enabled (ngtcp2 + nghttp3)
  ECH:              no      (--enable-ech)
  HTTPS RR:         no      (--enable-httpsrr)
  SSLS-EXPORT:      no      (--enable-ssls-export)
  Protocols:        dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
  Features:         alt-svc AsynchDNS brotli gsasl GSS-API HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

Few things that seem worth pointing out. I am using GnuTLS. nghttp2 is in version 1.58.

In general, software on $PATH has the following versions:

autoconf 2.71
automake 1.16.5
bash-minimal 5.1.16
binutils 2.41
brotli 1.0.9
bzip2 1.0.8
coreutils 9.1
diffutils 3.10
file 5.45
findutils 4.9.0
gawk 5.3.0
gcc 11.4.0
glibc 2.39
gnutls 3.8.3
grep 3.11
groff 1.22.4
gsasl 2.2.0
gzip 1.13
ld-wrapper 0
libgcrypt 1.11.0
libgpg-error 1.51
libidn 1.41
libidn2 2.3.4
libpsl 0.21.1
libtasn1 4.19.0
libtool 2.4.7
m4 1.4.19
make 4.4.1
mit-krb5 1.20
nettle 3.9.1
nghttp2 1.58.0
openldap 2.6.4
p11-kit 0.24.1
patch 2.7.6
perl 5.36.0
pkg-config 0.29.2
python-minimal 3.10.7
python-minimal-wrapper 3.10.7
rtmpdump-2.4 0.c5f04a5
sed 4.8
tar 1.34
xz 5.4.5
zstd 1.5.2

I will try to revert some of the commits above to see if it helps.

[vszakats]

Might be unrelated by I cannot spot openssl on the list of installed software.
openssl (1.0.2+) is required to generate the test certs.

[bagder]

I tried to update my curl package to 8.13.0, however during running the test suite I got the following error:

Is this the only test failure you get? Judging by the output it seems curl does not even get the HTTP/1 status line back in test 1700, which would rather indicate some infra problem rather than a curl bug.

[graywolf]

As far as I can tell from the build log, this is the only test failure. However when I excluded 1700, 1701 failed instead.

[bagder]

@vszakats you seem to be right. From his build log:

make[1]: Entering directory '/tmp/guix-build-curl-full-8.12.1.drv-0/source/tests/certs'
/gnu/store/v6bivyjbg6bj07s8iqfzdm6hpvypc0p1-perl-5.36.0/bin/perl ./genserv.pl test test-localhost.prm test-localhost.nn.prm test-localhost0h.prm test-localhost-san-first.prm test-localhost-san-last.prm
Can't exec "command": No such file or directory at ./genserv.pl line 51.
Can't exec "openssl": No such file or directory at ./genserv.pl line 52.
Can't exec "openssl": No such file or directory at ./genserv.pl line 57.
Can't exec "openssl": No such file or directory at ./genserv.pl line 60.
sh: line 1: openssl: command not found
Can't exec "openssl": No such file or directory at ./genserv.pl line 63.
sh: line 1: openssl: command not found
CA root generated: test 6000 days prime256v1
Can't exec "openssl": No such file or directory at ./genserv.pl line 78.
Can't exec "openssl": No such file or directory at ./genserv.pl line 81.
Can't exec "openssl": No such file or directory at ./genserv.pl line 83.
Can't exec "openssl": No such file or directory at ./genserv.pl line 84.
Can't exec "openssl": No such file or directory at ./genserv.pl line 97.
Use of uninitialized value in print at ./genserv.pl line 104, <$fi> line 1.
Certificate generated: CA=test 300days prime256v1 test-localhost
Can't exec "openssl": No such file or directory at ./genserv.pl line 78.
Can't exec "openssl": No such file or directory at ./genserv.pl line 81.
Can't exec "openssl": No such file or directory at ./genserv.pl line 83.
Can't exec "openssl": No such file or directory at ./genserv.pl line 84.
Can't exec "openssl": No such file or directory at ./genserv.pl line 97.
Use of uninitialized value in print at ./genserv.pl line 104, <$fi> line 1.
Certificate generated: CA=test 300days prime256v1 test-localhost.nn
Can't exec "openssl": No such file or directory at ./genserv.pl line 78.
Can't exec "openssl": No such file or directory at ./genserv.pl line 81.
Can't exec "openssl": No such file or directory at ./genserv.pl line 83.
Can't exec "openssl": No such file or directory at ./genserv.pl line 84.
Can't exec "openssl": No such file or directory at ./genserv.pl line 97.
Use of uninitialized value in print at ./genserv.pl line 104, <$fi> line 1.
Certificate generated: CA=test 300days prime256v1 test-localhost0h
Can't exec "openssl": No such file or directory at ./genserv.pl line 78.
Can't exec "openssl": No such file or directory at ./genserv.pl line 81.
Can't exec "openssl": No such file or directory at ./genserv.pl line 83.
Can't exec "openssl": No such file or directory at ./genserv.pl line 84.
Can't exec "openssl": No such file or directory at ./genserv.pl line 97.
Use of uninitialized value in print at ./genserv.pl line 104, <$fi> line 1.
Certificate generated: CA=test 300days prime256v1 test-localhost-san-first
Can't exec "openssl": No such file or directory at ./genserv.pl line 78.
Can't exec "openssl": No such file or directory at ./genserv.pl line 81.
Can't exec "openssl": No such file or directory at ./genserv.pl line 83.
Can't exec "openssl": No such file or directory at ./genserv.pl line 84.
Can't exec "openssl": No such file or directory at ./genserv.pl line 97.

[graywolf]

Might be unrelated by I cannot spot openssl on the list of installed software. openssl (1.0.2+) is required to generate the test certs.

Oh, this is it! When I add openssl into the build environment, the test start passing.

[graywolf]

I am not sure how I feel about needing two separated crypto packages for the build (gnutls for the linking, openssl for the tests), but it is easy to fix in the package definition. Thank you very much. :)

If you think there is nothing else to do, feel free to close this, but maybe some more descriptive error message when openssl is missing would be nice.

[bagder]

maybe some more descriptive error message when openssl is missing would be nice.

Yeah, we need to improve this so that it is more apparent what happens and what's needed.

[bagder]

I am not sure how I feel about needing two separated crypto packages for the build (gnutls for the linking, openssl for the tests),

I agree it is not ideal, but this for the moment the best setup we have figured out. 8.13.0 is the first release that generates these certificates on demand instead of using pre-generated certs shipped with the release.

[vszakats]

One mitigation is to fail early with an explicit error if genserv.pl doesn't detect
openssl. Not perfect, but may be an improvement. PR → #16929