Repository navigation
curl 7.87.0 test 0313 [CRL test] failing on EL-9 #10135
Description
Activity
OpenSSL/3.0.1 really?
Ideally, the test suite could be made a bit more robust about this sort of thing if it could check that the ports needed for a test were not already in use, and pick different ports if necessary.
It already does.
OpenSSL/3.0.1 really?
Yes, but heavily patched. See the SOURCES directory here:
https://git.rockylinux.org/staging/rpms/openssl/-/tree/r9Ideally, the test suite could be made a bit more robust about this sort of thing if it could check that the ports needed for a test were not already in use, and pick different ports if necessary.
It already does.
Thanks for the hint. I took a look at runtests.pl and found the --seed option. Trying again with --seed=1 gave similar results but with a different port:
startnew: perl -I../../tests ../../tests/httpserver.pl --pidfile ".http_server.pid" --logfile "log/http_server.log" --portfile .http_server.port --ipv4 --port 0 --srcdir "../../tests/data/.." RUN: ../src/curl --max-time 13 --output log/http_verify.out --silent --verbose --globoff "http://127.0.0.1:39779/verifiedserver" 2>log/http_verify.log CMD (0): ../src/curl --max-time 13 --output log/http_verify.out --silent --verbose --globoff "http://127.0.0.1:39779/verifiedserver" 2>log/http_verify.log RUN: HTTP server is on PID 75987 port 39779 * pid http => 75987 75987 startnew: perl -I../../tests ../../tests/secureserver.pl --pidfile ".https_server.pid" --logfile "log/https_stunnel.log" --ipv4 --proto https --certfile "Server-localhost-sv.pem" --stunnel "/usr/bin/stunnel" --srcdir "../../tests" --connect 39779 --accept 24851 RUN: HTTPS server is PID 76003 port 24851 * pid https => 76003 76007 prechecked /usr/bin/perl -e "print 'Test requires default test server host' if ( '127.0.0.1' ne '127.0.0.1' );" test 0313...[CRL test] ../src/curl --output log/curl313.out --include --trace-ascii log/trace313 --trace-time --cacert ../../tests/certs/EdelCurlRoot-ca.crt --crlfile ../../tests/certs/Server-localhost-sv.crl https://localhost:24851/313 >log/stdout313 2>log/stderr313 CMD (8960): ../src/curl --output log/curl313.out --include --trace-ascii log/trace313 --trace-time --cacert ../../tests/certs/EdelCurlRoot-ca.crt --crlfile ../../tests/certs/Server-localhost-sv.crl https://localhost:24851/313 >log/stdout313 2>log/stderr313 curl returned 35, when expecting 60 exit FAILED == Contents of files in the log/ dir after test 313 === Start of file commands.log ../src/curl --output log/curl313.out --include --trace-ascii log/trace313 --trace-time --cacert ../../tests/certs/EdelCurlRoot-ca.crt --crlfile ../../tests/certs/Server-localhost-sv.crl https://localhost:24851/313 >log/stdout313 2>log/stderr313 === End of file commands.log === Start of file ftpserver.cmd Testnum 313 === End of file ftpserver.cmd === Start of file http_server.log 14:26:34.461099 exit_signal_handler: 15 14:26:34.461277 signalled to die 14:26:34.461366 ========> IPv4 sws (port 39419 pid: 75946) exits with signal (15) 14:26:39.521894 Running HTTP IPv4 version on port 39779 14:26:39.521983 Wrote pid 75987 to .http_server.pid 14:26:39.522012 Wrote port 39779 to .http_server.port 14:26:40.521418 ====> Client connect 14:26:40.521479 accept_connection 3 returned 4 14:26:40.521526 accept_connection 3 returned 0 14:26:40.521569 Read 93 bytes 14:26:40.521596 Process 93 bytes request 14:26:40.521682 Got request: GET /verifiedserver HTTP/1.1 14:26:40.521718 Are-we-friendly question received 14:26:40.521771 Wrote request (93 bytes) input to log/server.input 14:26:40.521830 Identifying ourselves as friends 14:26:40.522083 Response sent (56 bytes) and written to log/server.response 14:26:40.522123 special request received, no persistency 14:26:40.522168 ====> Client disconnect 0 === End of file http_server.log === Start of file http_verify.log * Trying 127.0.0.1:39779... * Connected to 127.0.0.1 (127.0.0.1) port 39779 (#0) > GET /verifiedserver HTTP/1.1 > Host: 127.0.0.1:39779 > User-Agent: curl/7.87.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < Content-Length: 17 < { [1 bytes data] * Connection #0 to host 127.0.0.1 left intact === End of file http_verify.log === Start of file http_verify.out WE ROOLZ: 75987 === End of file http_verify.out === Start of file https_stunnel.log 2022.12.22 14:26:40 LOG5[ui]: stunnel 5.62 on x86_64-redhat-linux-gnu platform 2022.12.22 14:26:40 LOG5[ui]: Compiled/running with OpenSSL 3.0.1 14 Dec 2021 2022.12.22 14:26:40 LOG5[ui]: Threading:PTHREAD Sockets:POLL,IPv6 TLS:ENGINE,FIPS,OCSP,PSK,SNI 2022.12.22 14:26:40 LOG5[ui]: Reading configuration from file /builddir/build/BUILD/curl-7.87.0/build-minimal/tests/https_stunnel.conf 2022.12.22 14:26:40 LOG5[ui]: UTF-8 byte order mark not detected 2022.12.22 14:26:40 LOG5[ui]: FIPS mode disabled 2022.12.22 14:26:40 LOG5[ui]: Configuration successful 2022.12.22 14:26:40 LOG5[ui]: Binding service [curltest] to :::24851: Address already in use (98) 2022.12.22 14:26:40 LOG5[cron]: Updating DH parameters 2022.12.22 14:26:41 LOG5[0]: Service [curltest] accepted connection from 127.0.0.1:38218 2022.12.22 14:26:41 LOG3[0]: SSL_accept: ssl/record/rec_layer_s3.c:1584: error:0A00041B:SSL routines::tlsv1 alert decrypt error 2022.12.22 14:26:41 LOG5[0]: Connection reset: 0 byte(s) sent to TLS, 0 byte(s) sent to socket === End of file https_stunnel.log === Start of file server.response HTTP/1.1 200 OK Content-Length: 17 WE ROOLZ: 75987 === End of file server.response === Start of file stderr313 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (35) OpenSSL/3.0.1: error:03000098:digital envelope routines::invalid digest === End of file stderr313 === Start of file trace313 14:26:41.586244 == Info: Trying 127.0.0.1:24851... 14:26:41.785777 == Info: Connected to localhost (127.0.0.1) port 24851 (#0) 14:26:41.791828 == Info: ALPN: offers h2 14:26:41.791851 == Info: ALPN: offers http/1.1 14:26:41.793892 == Info: CAfile: ../../tests/certs/EdelCurlRoot-ca.crt 14:26:41.793915 == Info: CApath: none 14:26:41.794124 == Info: successfully loaded CRL file: 14:26:41.794141 == Info: CRLfile: ../../tests/certs/Server-localhost-sv.crl 14:26:41.794669 == Info: [CONN-0-0][CF-SSL] TLSv1.0 (OUT), TLS header, Certificate Status (22): 14:26:41.794687 => Send SSL data, 5 bytes (0x5) 0000: ..... 14:26:41.794728 == Info: [CONN-0-0][CF-SSL] TLSv1.3 (OUT), TLS handshake, Client hello (1): 14:26:41.794738 => Send SSL data, 512 bytes (0x200) 0000: ......-.+=.s...G....F.HF..z.X`.6...&UK ....o...u.=...f.{.F.3.S2. 0040: ........>.......,.0.........+./...$.(.k.#.'.g.....9.....3.....=. 0080: <.5./.....u.........localhost................................... 00c0: ......h2.http/1.1.........1.....*.(............................. 0100: ............+............-.....3.&.$... .<.Z....d...HNI.....m... 0140: ]b.....L........................................................ 0180: ................................................................ 01c0: ................................................................ 14:26:41.801864 == Info: [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Certificate Status (22): 14:26:41.801894 <= Recv SSL data, 5 bytes (0x5) 0000: ....z 14:26:41.801934 == Info: [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Server hello (2): 14:26:41.801945 <= Recv SSL data, 122 bytes (0x7a) 0000: ...v..=l.B.G:....-............L.cx..*. ....o...u.=...f.{.F.3.S2. 0040: .............+.....3.$... 1....\V......C...RIWr......NI!.g 14:26:41.802666 == Info: [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Finished (20): 14:26:41.802685 <= Recv SSL data, 5 bytes (0x5) 0000: ..... 14:26:41.802710 == Info: [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Supplemental data (23): 14:26:41.802720 <= Recv SSL data, 5 bytes (0x5) 0000: ..... 14:26:41.802745 <= Recv SSL data, 1 bytes (0x1) 0000: . 14:26:41.802786 == Info: [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): 14:26:41.802797 <= Recv SSL data, 10 bytes (0xa) 0000: .......... 14:26:41.802823 == Info: [CONN-0-0][CF-SSL] TLSv1.2 (IN), TLS header, Supplemental data (23): 14:26:41.802832 <= Recv SSL data, 5 bytes (0x5) 0000: ....f 14:26:41.802861 <= Recv SSL data, 1 bytes (0x1) 0000: . 14:26:41.802887 == Info: [CONN-0-0][CF-SSL] TLSv1.3 (IN), TLS handshake, Certificate (11): 14:26:41.802896 <= Recv SSL data, 1109 bytes (0x455) 0000: ...Q...M..H0..D0..,............f0...*.H........0h1.0...U....NN11 0040: 0/..U...(Edel Curl Arctic Illudium Research Cloud1&0$..U....Nort 0080: hern Nowhere Trust Anchor0...221125123246Z..310211123246Z0T1.0.. 00c0: .U....NN110/..U...(Edel Curl Arctic Illudium Research Cloud1.0.. 0100: .U....localhost0.."0...*.H.............0.........h2O..O}2....... 0140: d|-...+.T..r]..9..u.......G.......(&& ,|*...)x-.>o.........v..Xj 0180: 9.....=.GYQ_....B..B..O[V../.c{......>..N...D..N...bj6cP....".#. 01c0: ...\...c.....F...R..{.[......9..P.....`..&Y."..W.1....jZ......q. 0200: ..*..z.....s..r.....7s..^.....fK..\....k.................0...0.. 0240: .U....0...localhost0...U........0...U.%..0...+.......0...U...... 0280: dg......A.b.v...).2A0...U.#..0...R@..O.X)..g.........0...U....0. 02c0: 0C..+........70503..+.....0..'http://test.curl.se/ca/EdelCurlRoo 0300: t.cer08..U...10/0-.+.).'http://test.curl.se/ca/EdelCurlRoot.crl0 0340: ...*.H.............1.c.../.Z.....I...........3v.....T..M.S....Q. 0380: l82..V..."E......,6A..X..~TA..`..T.g..+......(..=...O3....4...3X 03c0: .#..9."..1.GqM.&kP.M._.m<..-..=...>.[...1.......g.4.z..W...>.... 0400: 0VM1.e}j:o&...;...Z...j.,....l.._v......7.....t.ST..B.@..H.../|. 0440: .......q....Ps*"..V.. 14:26:41.805005 == Info: [CONN-0-0][CF-SSL] TLSv1.2 (OUT), TLS header, Unknown (21): 14:26:41.805027 => Send SSL data, 5 bytes (0x5) 0000: ..... 14:26:41.805118 == Info: [CONN-0-0][CF-SSL] TLSv1.3 (OUT), TLS alert, decrypt error (563): 14:26:41.805132 => Send SSL data, 2 bytes (0x2) 0000: .3 14:26:41.805173 == Info: OpenSSL/3.0.1: error:03000098:digital envelope routines::invalid digest 14:26:41.805316 == Info: Closing connection 0 === End of file trace313So, still getting "Address already in use", this time on port 24851. This is weird, the other stunnel-based tests are working OK.
Since curl logs a lot of TLS traffic, I think "Address already in use" is a ghost chase. Part of finding an unused port can involve trying a port that is in use, but then it should continue and try another port. (Unfortunately stunnel does not allow us to make it pick a random unused one by itself.)
The problem is rather this:
curl returned 35, when expecting 6035 being
CURLE_SSL_CONNECT_ERRORand 60CURLE_PEER_FAILED_VERIFICATIONThe error message from OpenSSL in there is also weird:
curl: (35) OpenSSL/3.0.1: error:03000098:digital envelope routines::invalid digest. That's not expected.Looks like a certificate issue. I copied the files
EdelCurlRoot-ca.crt,Server-localhost-sv.crlandServer-localhost-sv.pemfrom thetests/certsdirectory of the curl 7.86.0 release and that made test 0313 pass, at the expense of tests 2034, 2037, 2041, 3000 and 3001 failing for various certificate non-matching issues.I think we would take in any improvements to
tests/certs/Makefile.amand related scripts. These are the ones used to generate the current batch.- added a commit that references this issue
on Dec 23, 2022 this thread is one of the only search hits for "0A00041B."
in my case it had been working before, but I had accidentally reconfigured tpm2tss-engine without --disable-digestsign which causes signatures to fail. curl gave this error "0A00041B" but nginx showed signature failures on the client side which eventually led me to the solution: reconfigure tpm-tss-engine with --disable-digestsign
I did this
Updated my curl packaging from 7.86.0 to 7.87.0 and built for current Fedora and RHEL releases (RHEL builds on Rocky Linux, an RHEL clone).
I expected the following
All builds completing successfully. And they did, except for the build on Rocky Linux 9.
curl/libcurl version
7.87.0
[curl -V output]
Don't get as far as a built curl package but here are the system characteristics from the start of the test run:
operating system
I am building on a Fedora 37 (x86_64) host using a chroot-based build system targeting Rocky Linux 9 (x86_64).
Here is the build output from the failing test:
The equivalent test in 7.86.0 still passes in the same build environment. At first I suspected that this would be related to the more strict crypto policies in RHEL9 that disallow SHA1 signatures etc., but the certificates in this test look to be using SHA256, which should be fine. Then, whilst pasting in the build log above I noticed this in
https_stunnel.log:This is a type of error I've seen many times before, usually when running a second iteration of the test suite for a differently-configured curl build in the same environment, where a test server from the original test run was still running. That's not the case here though, because this is the first test run. Also, I don't see any mention of the use of port 25073 earlier in the build log. That would usually lead me to suspect a service running on that port on the build host but I can't see anything there either.
I suspect that if I could persuade the test to run stunnel on a different port that it might work, but since the port numbers seem to be selected at random and based on the build environment somehow, I don't know how to do that. Any suggestions?
Ideally, the test suite could be made a bit more robust about this sort of thing if it could check that the ports needed for a test were not already in use, and pick different ports if necessary. Clearly that would be racy but better than nothing.