autobuilds: test 1560 [URL API] failing

[MarcelRaad]

I did this

Look at the autobuilds at https://curl.se/dev/builds.html. Since commit 901392c, @mkauf 's NetBSD builds and Kirill Sokolovsky's Linux builds are failing for the newly extended test 1560 with:

=== Start of file stderr1560
URL: -
/curl/tests/libtest/lib1560.c:1058 returned 7 (A memory function failed). URL: 'https://r[C3][A4]ksm[C3][B6]rg[C3][A5]s.se/path?q#frag'
=== End of file stderr1560

I cannot reproduce the error with a curl configured as close as possible to the failing Linux configuration with the Windows Subsystem for Linux (tried with Ubuntu, Debian, Kali, openSUSE Tumbleweed, and Alpine).

I expected the following

Test 1560 succeeding.

curl/libcurl version

NetBSD:

Host setup: armv7eb-unknown-netbsdelf9.3-eabihf
Install prefix: /usr/local
Compiler: gcc
CFLAGS: -Werror-implicit-function-declaration -g -O0 -std=gnu89 -pedantic -Wall -W -Wpointer-arith -Wwrite-strings -Wunused -Wshadow -Winline -Wnested-externs -Wmissing-declarations -Wmissing-prototypes -Wno-long-long -Wbad-function-cast -Wfloat-equal -Wno-multichar -Wsign-compare -Wundef -Wno-format-nonliteral -Wendif-labels -Wstrict-prototypes -Wdeclaration-after-statement -Wold-style-definition -Wstrict-aliasing=3 -Wcast-align -Wtype-limits -Wold-style-declaration -Wmissing-parameter-type -Wempty-body -Wclobbered -Wignored-qualifiers -Wconversion -Wno-sign-conversion -Wvla -ftree-vrp -Wdouble-promotion -Wformat=2 -Warray-bounds=2 -Wshift-negative-value -Wshift-overflow=2 -Wnull-dereference -fdelete-null-pointer-checks -Wduplicated-cond -Wunused-const-variable -Wduplicated-branches -Wrestrict -Walloc-zero -Wformat-overflow=2 -Wformat-truncation=2 -Wimplicit-fallthrough=4 -Wno-system-headers -pthread
CPPFLAGS: -isystem /usr/pkg/include -isystem /usr/pkg/include
LDFLAGS: -L/usr/lib -Wl,-R/usr/lib -L/usr/pkg/lib -Wl,-R/usr/pkg/lib -L/usr/pkg/lib -L/usr/pkg/lib
LIBS: -lnghttp2 -lidn2 -lssl -lcrypto -lssl -lcrypto -lldap -lz

curl version: 7.87.1-DEV
SSL: enabled (OpenSSL)
SSH: no (--with-{libssh,libssh2})
zlib: enabled
brotli: no (--with-brotli)
zstd: no (--with-zstd)
GSS-API: no (--with-gssapi)
GSASL: no (libgsasl not found)
TLS-SRP: enabled
resolver: POSIX threaded
IPv6: enabled
Unix sockets: enabled
IDN: enabled (libidn2)
Build libcurl: Shared=yes, Static=yes
Built-in manual: enabled
--libcurl option: enabled (--disable-libcurl-option)
Verbose errors: enabled (--disable-verbose)
Code coverage: disabled
SSPI: no (--enable-sspi)
ca cert bundle: no
ca cert path: /etc/openssl/certs
ca fallback: no
LDAP: enabled (OpenLDAP)
LDAPS: enabled
RTSP: enabled
RTMP: no (--with-librtmp)
PSL: no (libpsl not found)
Alt-svc: enabled (--disable-alt-svc)
Headers API: enabled (--disable-headers-api)
HSTS: enabled (--disable-hsts)
HTTP1: enabled (internal)
HTTP2: enabled (nghttp2)
HTTP3: no (--with-ngtcp2, --with-quiche --with-msh3)
ECH: no (--enable-ech)
WebSockets: no (--enable-websockets)
Protocols: DICT FILE FTP FTPS GOPHER GOPHERS HTTP HTTPS IMAP IMAPS LDAP LDAPS MQTT POP3 POP3S RTSP SMB SMBS SMTP SMTPS TELNET TFTP
Features: AsynchDNS HSTS HTTP2 HTTPS-proxy IDN IPv6 Largefile NTLM NTLM_WB SSL TLS-SRP UnixSockets alt-svc libz threadsafe

Linux:

Host setup: x86_64-pc-linux-gnu
Install prefix: /usr/local
Compiler: gcc
CFLAGS: -Werror-implicit-function-declaration -g -O0 -std=gnu89 -pedantic -Wall -W -Wpointer-arith -Wwrite-strings -Wunused -Wshadow -Winline -Wnested-externs -Wmissing-declarations -Wmissing-prototypes -Wno-long-long -Wbad-function-cast -Wfloat-equal -Wno-multichar -Wsign-compare -Wundef -Wno-format-nonliteral -Wendif-labels -Wstrict-prototypes -Wdeclaration-after-statement -Wold-style-definition -Wstrict-aliasing=3 -Wcast-align -Wtype-limits -Wold-style-declaration -Wmissing-parameter-type -Wempty-body -Wclobbered -Wignored-qualifiers -Wconversion -Wno-sign-conversion -Wvla -ftree-vrp -Wdouble-promotion -Wformat=2 -Warray-bounds=2 -Wshift-negative-value -Wshift-overflow=2 -Wnull-dereference -fdelete-null-pointer-checks -Wduplicated-cond -Wunused-const-variable -Wduplicated-branches -Wrestrict -Walloc-zero -Wformat-overflow=2 -Wformat-truncation=2 -Wimplicit-fallthrough=4 -Warith-conversion -Wenum-conversion -Wno-system-headers -pthread
CPPFLAGS:
LDFLAGS:
LIBS: -lnghttp2 -lidn2 -lpsl -lssl -lcrypto -lssl -lcrypto -lldap -llber -lzstd -lz

curl version: 7.87.1-DEV
SSL: enabled (OpenSSL)
SSH: no (--with-{libssh,libssh2})
zlib: enabled
brotli: no (--with-brotli)
zstd: enabled (libzstd)
GSS-API: no (--with-gssapi)
GSASL: no (libgsasl not found)
TLS-SRP: enabled
resolver: POSIX threaded
IPv6: enabled
Unix sockets: enabled
IDN: enabled (libidn2)
Build libcurl: Shared=yes, Static=yes
Built-in manual: enabled
--libcurl option: enabled (--disable-libcurl-option)
Verbose errors: enabled (--disable-verbose)
Code coverage: disabled
SSPI: no (--enable-sspi)
ca cert bundle: /etc/ssl/certs/ca-certificates.crt
ca cert path: no
ca fallback: no
LDAP: enabled (OpenLDAP)
LDAPS: enabled
RTSP: enabled
RTMP: no (--with-librtmp)
PSL: enabled
Alt-svc: enabled (--disable-alt-svc)
Headers API: enabled (--disable-headers-api)
HSTS: enabled (--disable-hsts)
HTTP1: enabled (internal)
HTTP2: enabled (nghttp2)
HTTP3: no (--with-ngtcp2, --with-quiche --with-msh3)
ECH: no (--enable-ech)
WebSockets: no (--enable-websockets)
Protocols: DICT FILE FTP FTPS GOPHER GOPHERS HTTP HTTPS IMAP IMAPS LDAP LDAPS MQTT POP3 POP3S RTSP SMB SMBS SMTP SMTPS TELNET TFTP
Features: AsynchDNS HSTS HTTP2 HTTPS-proxy IDN IPv6 Largefile NTLM NTLM_WB PSL SSL TLS-SRP UnixSockets alt-svc libz threadsafe zstd

operating system

NetBSD:

********* System characteristics ********
* curl 7.87.1-DEV (armv7eb-unknown-netbsdelf9.3-eabihf)
* libcurl/7.87.1-DEV OpenSSL/1.1.1k zlib/1.2.10 libidn2/2.3.3 nghttp2/1.49.0
* Features: alt-svc AsynchDNS Debug HSTS HTTP2 HTTPS-proxy IDN IPv6 Largefile libz NTLM NTLM_WB SSL threadsafe TLS-SRP TrackMemory UnixSockets
* Disabled:
* Host: cubieboard
* System: NetBSD cubieboard 9.3 NetBSD 9.3 (GENERIC) #0: Thu Aug 4 15:30:37 UTC 2022 mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/evbarm/compile/GENERIC evbarm
* OS: netbsd
*
*** DISABLES memory tracking when using threaded resolver
*
* Servers: SSL HTTP-IPv6 HTTP-unix FTP-IPv6
* Env:
* Seed: 251056
*****************************************

Linux:

********* System characteristics ********
* curl 7.87.1-DEV (x86_64-pc-linux-gnu)
* libcurl/7.87.1-DEV OpenSSL/1.0.2u zlib/1.2.11 zstd/1.5.0 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.0) nghttp2/1.45.1
* Features: alt-svc AsynchDNS Debug HSTS HTTP2 HTTPS-proxy IDN IPv6 Largefile libz NTLM NTLM_WB PSL SSL threadsafe TLS-SRP TrackMemory UnixSockets zstd
* Disabled:
* Host: scan
* System: Linux scan 4.1.15-gentoo-r1 #2 SMP Thu Feb 18 14:16:46 UTC 2016 x86_64 Intel(R) Xeon(R) CPU X5675 @ 3.07GHz GenuineIntel GNU/Linux
* OS: linux
*
*** DISABLES memory tracking when using threaded resolver
*
* Servers: HTTP-IPv6 HTTP-unix FTP-IPv6
* Env:
* Seed: 231329
*****************************************

[bagder]

We probably need to make sure they use UTF-8 locales for test 1560.

[bagder]

There's still a risk that this fix does not work, since there is no guarantee that systems have UTF8 locales around, but then I'm not really sure how we should go about and detect that and skip the tests on those systems...

[dfandrich]

There are a number of tests that check that already in <precheck>, such as test955.

[bagder]

Ah right. Thanks!

[bagder]

except that test 1560 needs a condition like "using winidn OR !using-IDN OR has a UTF-8 locale" ...

[sharewax]

On my build Oracle Linux 9 I have an issue with this 1560 test.

test 1560...[URL API]
../libtool --mode=execute /usr/bin/valgrind --tool=memcheck --quiet --leak-check=yes --suppressions=../../tests/valgrind.supp --num-callers=16 --log-file=log/valgrind1560 ./libtest/lib1560 - > log/stdout1560 2> log/stderr1560
CMD (768): ../libtool --mode=execute /usr/bin/valgrind --tool=memcheck --quiet --leak-check=yes --suppressions=../../tests/valgrind.supp --num-callers=16 --log-file=log/valgrind1560 ./libtest/lib1560 - > log/stdout1560 2> log/stderr1560
 1560: stdout FAILED:
--- log/check-expected  2023-05-23 08:31:11.619504665 +0000
+++ log/check-generated 2023-05-23 08:31:11.619504665 +0000
@@ -1 +0,0 @@
-success[LF]
== Contents of files in the log/ dir after test 1560
=== Start of file check-expected
 success[LF]
=== End of file check-expected
=== Start of file commands.log
 ../libtool --mode=execute /usr/bin/valgrind --tool=memcheck --quiet --leak-check=yes --suppressions=../../tests/valgrind.supp --num-callers=16 --log-file=log/valgrind1560 ./libtest/lib1560 - > log/stdout1560 2> log/stderr1560
=== End of file commands.log
=== Start of file ftpserver.cmd
 Testnum 1560
=== End of file ftpserver.cmd
=== Start of file stderr1560
 ../libtool: line 4460: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
 ../libtool: line 4460: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory
 ../libtool: line 4460: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): No such file or directory
 URL: -
 ../../../tests/libtest/lib1560.c:1112 returned 7 (A memory function failed). URL: 'https://räksmörgås.se/path?q#frag'
 Get URL
 in: https://räksmörgås.se/path?q#frag
 returned 7 (expected 0)
=== End of file stderr1560

Am I right, that I need some additional locale package requirements for build?
It seems that I need add this one. Thank you

# needed for test1560 to succeed
BuildRequires: glibc-langpack-en