Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HTTP2 frame issue with libcurl and openscap #15941

Closed
Jiri-Stary opened this issue Jan 8, 2025 · 12 comments
Closed

HTTP2 frame issue with libcurl and openscap #15941

Jiri-Stary opened this issue Jan 8, 2025 · 12 comments
Assignees
@icing
Labels
HTTP/2 not-a-curl-bug This is not a bug in curl

Comments

@Jiri-Stary
Copy link

I did this

/usr/bin/oscap info --verbose DEVEL --fetch-remote-resources --profiles /ssg/scap-security-guide-0.1.75/ssg-debian12-ds.xml;

It uses libcurl to fetch the data and it fails on frame issue:

I: oscap: Using environment variables: [oscap(17):oscap(7fd8c7d58390):debug.c:316:oscap_print_env_vars]
I: oscap: OSCAP_CHECK_ENGINE_PLUGIN_DIR='' [oscap(17):oscap(7fd8c7d58390):debug.c:319:oscap_print_env_vars]
I: oscap: OSCAP_CONTAINER_VARS='' [oscap(17):oscap(7fd8c7d58390):debug.c:319:oscap_print_env_vars]
I: oscap: OSCAP_EVALUATION_TARGET='' [oscap(17):oscap(7fd8c7d58390):debug.c:319:oscap_print_env_vars]
I: oscap: OSCAP_FULL_VALIDATION='' [oscap(17):oscap(7fd8c7d58390):debug.c:319:oscap_print_env_vars]
I: oscap: OSCAP_OVAL_COMMAND_OPTIONS='' [oscap(17):oscap(7fd8c7d58390):debug.c:319:oscap_print_env_vars]
I: oscap: OSCAP_PCRE_EXEC_RECURSION_LIMIT='' [oscap(17):oscap(7fd8c7d58390):debug.c:319:oscap_print_env_vars]
I: oscap: OSCAP_PROBE_ROOT='' [oscap(17):oscap(7fd8c7d58390):debug.c:319:oscap_print_env_vars]
I: oscap: SEXP_VALIDATE_DISABLE='' [oscap(17):oscap(7fd8c7d58390):debug.c:319:oscap_print_env_vars]
I: oscap: SOURCE_DATE_EPOCH='' [oscap(17):oscap(7fd8c7d58390):debug.c:319:oscap_print_env_vars]
I: oscap: OSCAP_PROBE_MEMORY_USAGE_RATIO='' [oscap(17):oscap(7fd8c7d58390):debug.c:319:oscap_print_env_vars]
I: oscap: OSCAP_PROBE_MAX_COLLECTED_ITEMS='' [oscap(17):oscap(7fd8c7d58390):debug.c:319:oscap_print_env_vars]
I: oscap: OSCAP_PROBE_IGNORE_PATHS='' [oscap(17):oscap(7fd8c7d58390):debug.c:319:oscap_print_env_vars]
I: oscap: Identified document type: data-stream-collection [oscap(17):oscap(7fd8c7d58390):doc_type.c:96:oscap_determine_document_type_reader]
Downloading: https://www.debian.org/security/oval/oval-definitions-bookworm.xml.bz2 ... D: oscap: == cURL info: Host www.debian.org:443 was resolved.
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: IPv6: 2603:400a:ffff:bb8::801f:3e
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: IPv4: 128.31.0.62
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: Trying [2603:400a:ffff:bb8::801f:3e]:443...
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: Immediate connect fail for 2603:400a:ffff:bb8::801f:3e: Network unreachable
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: Trying 128.31.0.62:443...
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: ALPN: curl offers h2,http/1.1
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: TLSv1.3 (OUT), TLS handshake, Client hello (1):
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: CAfile: /etc/ssl/certs/ca-certificates.crt
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: CApath: /etc/ssl/certs
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: TLSv1.3 (IN), TLS handshake, Server hello (2):
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: TLSv1.3 (IN), TLS handshake, Certificate (11):
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: TLSv1.3 (IN), TLS handshake, CERT verify (15):
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: TLSv1.3 (IN), TLS handshake, Finished (20):
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: TLSv1.3 (OUT), TLS handshake, Finished (20):
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / RSASSA-PSS
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: ALPN: server accepted h2
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: Server certificate:
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: subject: CN=www.debian.org
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: start date: Dec 10 00:38:49 2024 GMT
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: expire date: Mar 10 00:38:48 2025 GMT
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: subjectAltName: host "www.debian.org" matched cert's "www.debian.org"
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: issuer: C=US; O=Let's Encrypt; CN=R11
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: SSL certificate verify ok.
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: Certificate level 0: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: Connected to www.debian.org (128.31.0.62) port 443
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: using HTTP/2
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: [HTTP/2] [1] OPENED stream for https://www.debian.org/security/oval/oval-definitions-bookworm.xml.bz2
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: [HTTP/2] [1] [:method: GET]
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: [HTTP/2] [1] [:scheme: https]
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: [HTTP/2] [1] [:authority: www.debian.org]
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: [HTTP/2] [1] [:path: /security/oval/oval-definitions-bookworm.xml.bz2]
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: [HTTP/2] [1] [accept: /]
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: [HTTP/2] [1] [te: gzip]
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: [HTTP/2] [1] [accept-encoding: deflate, gzip, br, zstd]
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: => cURL header (out): GET /security/oval/oval-definitions-bookworm.xml.bz2 HTTP/2
Host: www.debian.org
Accept: /
Connection: TE
TE: gzip
Accept-Encoding: deflate, gzip, br, zstd
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: Request completely sent off
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
D: oscap: == cURL info: Connection #0 to host www.debian.org left intact
[oscap(17):oscap(7fd8c7d58390):oscap_acquire.c:315:_curl_trace]
error
OpenSCAP Error: Download failed: Stream error in the HTTP/2 framing layer [/home/buildozer/aports/community/openscap/src/openscap-1.3.10/src/common/oscap_acquire.c:405]
Could not extract scap_org.open-scap_cref_ssg-debian12-xccdf.xml with all dependencies from datastream. [/home/buildozer/aports/community/openscap/src/openscap-1.3.10/src/DS/ds_sds_session.c:228]
An error occurred during evaluation

I expected the following

not to have the the frame error on the file

https://www.debian.org/security/oval/oval-definitions-bookworm.xml.bz2

Download failed: Stream error in the HTTP/2 framing layer

curl/libcurl version

#10 [5/8] RUN curl --version
#10 0.157 curl 8.11.1 (x86_64-alpine-linux-musl) libcurl/8.11.1 OpenSSL/3.3.2 zlib/1.3.1 brotli/1.1.0 zstd/1.5.6 c-ares/1.33.1 libidn2/2.3.7 libpsl/0.21.5 nghttp2/1.62.1
#10 0.157 Release-Date: 2024-12-11
#10 0.157 Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp ws wss
#10 0.157 Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTPS-proxy IDN IPv6 Largefile libz NTLM PSL SSL threadsafe TLS-SRP UnixSockets zstd
#10 DONE 0.2s

operating system

alpine linux 3.20
@icing
Copy link
Contributor

icing commented Jan 8, 2025 •
edited
Loading

Looks like the request header te: gzip confused the server.

works> curl https://www.debian.org/security/oval/oval-definitions-bookworm.xml.bz2 -o /dev/null
breaks> curl https://www.debian.org/security/oval/oval-definitions-bookworm.xml.bz2 -o /dev/null -H 'te: gzip'

Does your application add that to requests?

@icing icing self-assigned this Jan 8, 2025
@Jiri-Stary Jiri-Stary mentioned this issue Jan 8, 2025
Open
@bagder bagder added the HTTP/2 label Jan 8, 2025
@icing
Copy link
Contributor

icing commented Jan 8, 2025

To quote from https://www.rfc-editor.org/rfc/rfc9113.html#name-connection-specific-header- headers like 'TE' are not allowed in HTTP/2 and lead to a stream reset. Apache is correct in its behaviour.

The question is now: is the libcurl application wrong in adding the header or should curl, when selecting HTTP/2, automatically strip the header from the request.

@bagder
Copy link
Member

bagder commented Jan 8, 2025

Perhaps it uses CURLOPT_TRANSFER_ENCODING ?

@evgenyz
Copy link

evgenyz commented Jan 8, 2025

So, due to the nature of the content openscap consumes (hundreds of megabytes of usually uncompressed XMLs) it is preferable to force compression in every way possible from the application point of view. On the other hand openscap as an application doesn't really care about the HTTP protocol version and just follows what curl provides.

I can add a check (any examples on how/when to properly do the check?) to avoid setting the header if the protocol is HTTP/2, but it kinda feels weird.

@evgenyz
Copy link

evgenyz commented Jan 8, 2025

Perhaps it uses CURLOPT_TRANSFER_ENCODING ?

No, we explicitly set it (https://github.com/OpenSCAP/openscap/blob/829598a38d7d641b650b688883edbf0c738f6672/src/common/oscap_acquire.c#L360).

@icing
Copy link
Contributor

icing commented Jan 8, 2025

@evgenyz Perhaps you are looking for CURLOPT_ACCEPT_ENCODING which announces compression ways for the content the server may apply. The transfer encoding in HTTP is something else.

@bagder
Copy link
Member

bagder commented Jan 8, 2025

Yeah, transfer encoding is kind of the way it was once meant to be but content encoding is the way everyone uses.

@evgenyz
Copy link

evgenyz commented Jan 8, 2025

Perhaps you are looking for CURLOPT_ACCEPT_ENCODING which announces compression ways for the content the server may apply. The transfer encoding in HTTP is something else.

Yeah, I know that they are different, but TE was used for some weird case as far as I remember and we haven't had any problems with it up until now.

@icing
Copy link
Contributor

icing commented Jan 8, 2025

Perhaps you are looking for CURLOPT_ACCEPT_ENCODING which announces compression ways for the content the server may apply. The transfer encoding in HTTP is something else.

Yeah, I know that they are different, but TE was used for some weird case as far as I remember and we haven't had any problems with it up until now.

You can restrict your requests to HTTP/1.1 only in libcurl. But TE and HTTP/2 together is simply forbidden by the standard.

@evgenyz
Copy link

evgenyz commented Jan 8, 2025

Okay, let me think about this (I really don't remember already why exactly TE is there) and maybe ask you later about options. I'll reopen the original issue in OpenSCAP. Thank you for clarification.

@icing icing added the not-a-curl-bug This is not a bug in curl label Jan 8, 2025
@evgenyz
Copy link

evgenyz commented Jan 8, 2025

The question is now: is the libcurl application wrong in adding the header or should curl, when selecting HTTP/2, automatically strip the header from the request.

Stripping the header kinda makes sense in this situation from my point of view, even if it is an application mistake (with maybe a warning?). Just my 2 cents.

icing added a commit to icing/curl that referenced this issue Jan 8, 2025
@icing
HTTP/2, strip TE request header
68a6713 
The TE request header field is invalid in HTTP/2. Since
clients may not know in advance if a connection negotiates
HTTP/2, automatically strip such a header when h2 is in play.

Add test_01_10 to verify.

refs curl#15941

e Please enter the commit message for your changes. Lines starting
@icing icing mentioned this issue Jan 8, 2025
Closed
@icing
Copy link
Contributor

icing commented Jan 8, 2025

I propose #15943 to add auto-stripping of TE headers in HTTP/2 transfers.

@bagder bagder closed this as completed in 4e15605 Jan 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@icing icing

Labels
HTTP/2 not-a-curl-bug This is not a bug in curl
Milestone
No milestone
Development

No branches or pull requests
4 participants
@icing @bagder @evgenyz @Jiri-Stary