Since 8.1.0 php-curl POST fails with HTTP/2 stream 1 was reset for {url} #11175

mariandomanik · 2023-05-22T09:05:21Z

We are using Sentry Laravel SDK https://docs.sentry.io/platforms/php/guides/laravel/
Since curl 8.1.0 both on prod and local, we were unable to connect to Sentry servers, always resulting with

HTTP/2 stream 1 was reset for {our-sentry-dsn}

This was called from PHP SDK and we were trying to follow the HTTP Exception, but this was the only message we were able to gather, with no HTTP error code and no additional information.

Downgrading to curl 7.86 fixed this issue.

Fails on both MacOSX 13.4 and
5.10.178-162.673.amzn2.x86_64 #1 SMP Mon Apr 24 23:34:06 UTC 2023 x86_64 Linux

The text was updated successfully, but these errors were encountered:

bagder · 2023-05-22T09:18:01Z

This is next to impossible for us to act on. Can you please elaborate on how we can reproduce or you need to provide a lot more information about what exactly is happening!

bagder · 2023-05-22T09:25:45Z

Where does the ...was reset for... message come from? I can't seem to find that in libcurl 8.1.0.

mariandomanik · 2023-05-22T10:19:01Z

I'll try to provide as much info as possible, although only from PHP side. I hope this information is helpful in any way, if not and you have any more suggestions, I'll be happy to help some more:

curl handle in PHP

array:38 [ // vendor/symfony/http-client/Response/CurlResponse.php:327
  "url" => "https://**omitted.ingest.sentry.io/api/**omitted/store/"
  "content_type" => null
  "http_code" => 0
  "header_size" => 0
  "request_size" => 425
  "filetime" => -1
  "ssl_verify_result" => 0
  "redirect_count" => 0
  "total_time" => 0.113623
  "namelookup_time" => 0.003425
  "connect_time" => 0.0415
  "pretransfer_time" => 0.077446
  "size_upload" => 4459.0
  "size_download" => 0.0
  "speed_download" => 0.0
  "speed_upload" => 39243.0
  "download_content_length" => -1.0
  "upload_content_length" => 4459.0
  "starttransfer_time" => 0.077449
  "redirect_time" => 0.0
  "redirect_url" => ""
  "primary_ip" => "34.120.195.249"
  "certinfo" => []
  "primary_port" => 443
  "local_ip" => "192.168.1.36"
  "local_port" => 56567
  "http_version" => 0
  "protocol" => 2
  "ssl_verifyresult" => 0
  "scheme" => "HTTPS"
  "appconnect_time_us" => 76850
  "connect_time_us" => 41500
  "namelookup_time_us" => 3425
  "pretransfer_time_us" => 77446
  "redirect_time_us" => 0
  "starttransfer_time_us" => 77449
  "total_time_us" => 113623
  "effective_method" => "POST"
]

resulting curl error is 56

PHP curl_error() translates it into

"HTTP/2 stream 1 was reset"

stayallive · 2023-05-22T11:04:39Z

Hello, maintainer from the Sentry SDK team here, we are already looking into this and we can confirm this happens on PHP with cURL 8.1.0, however using the cURL CLI directly seems to work fine so I'm inclined to lean that this is something on the PHP side or it's interaction with cURL or some other factor we are not seeing at the moment.

The CLI test I ran:

docker run --rm -ti curlimages/curl:8.1.0 \
  -X POST https://o447951.ingest.sentry.io/api/5572016/store/ \
  -H "Content-Type: application/json" \
  -H "X-Sentry-Auth: Sentry sentry_key=49bf4ae0f80a4871abb813507b2d7cae" \
  -d '{}'

Sentry PHP SDK issue: getsentry/sentry-php#1537.

- refs curl#11157 and curl#11175 where uploads get stuck or lead to RST streams - fixes our h2 send behaviour to continue sending in the nghttp2 session as long as it wants to. This will empty our send buffer as long as the remote stream/connection window allows. - in case the window is exhausted, the data remaining in the send buffer will wait for a WINDOW_UPDATE from the server. Which is a socket event that engages our transfer loop again - the problem in the issue was that we did not exhaust the window, but left data in the sendbuffer and no further socket events did happen. The server was just waiting for us to send more. - relatedly, there was an issue fixed that closing a stream with KEEP_HOLD set kept the transfer from shutting down - as it should have - leading to a timeout.

icing · 2023-05-22T11:30:10Z

My question would be if you see a RST on the stream do to a timeout on the server side? Or does the server reset immediately because it does not like what it sees.

In case of a timeout, please see #11176 where I fixed an issue that led to stalled curl uploads.

cleptric · 2023-05-22T13:36:02Z

Sentry uses GCP LBs in front of the ingestion service, hence it will be crazy difficult to get deeper insights here.
If you can point me to some docs on how I can compile curl with PHP, I'm happy to give your PR a try to see if it fixes our issue.

- refs #11157 and #11175 where uploads get stuck or lead to RST streams - fixes our h2 send behaviour to continue sending in the nghttp2 session as long as it wants to. This will empty our send buffer as long as the remote stream/connection window allows. - in case the window is exhausted, the data remaining in the send buffer will wait for a WINDOW_UPDATE from the server. Which is a socket event that engages our transfer loop again - the problem in the issue was that we did not exhaust the window, but left data in the sendbuffer and no further socket events did happen. The server was just waiting for us to send more. - relatedly, there was an issue fixed that closing a stream with KEEP_HOLD set kept the transfer from shutting down - as it should have - leading to a timeout. Closes #11176

bagder · 2023-05-23T09:05:58Z

Please try 8.1.1 and see if the problem remains!

S3B-4 · 2023-05-23T12:45:42Z

Unfortunately i'm encountering the same problem on my dev machine. Macbook Pro M2 / macOS: 13.2.1 / curl 8.1.0

Sending test event...
There was an error sending the event.
SDK: Failed to send the event to Sentry. Reason: "HTTP/2 stream 1 was reset for "sentry-dsn".".
Please check the error message from the SDK above for further hints about what went wrong.

icing · 2023-05-23T14:54:20Z

@S3B-4 did you test 8.1.0 or 8.1.1? In 8.1.1 we fixed issues in upload handling. That could make a difference.

mariandomanik · 2023-05-23T16:04:21Z

Tried with brew version 8.1.1, same issue

"version" => "8.1.1", "host" => "aarch64-apple-darwin22.4.0",

krowinski · 2023-05-23T16:26:00Z

The problem is that before 8.1 invalid TE header returned data now its returning error

to replicate on >=8.1

curl https://google.com -H "TE: gzip"
curl: (56) HTTP/2 stream 1 was reset

and before it was "ok"

curl https://google.com -H "TE: gzip"
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">

And RFC for HTTP/2 specifies:

The only exception to this is the TE header field, which MAY be present in an HTTP/2 request; when it is, it MUST NOT contain any value other than "trailers"."

https://httpwg.org/specs/rfc9113.html#rfc.section.8.2.2

mariandomanik · 2023-05-23T17:15:15Z

@krowinski

thank you very much for suggestion, can confirm, commenting out the TE headers fixed the issue.
They consisted of 'gzip', 'deflate', 'chunked'

Looks like it was unrelated to the curl itself, more into PHP/Laravel implementation.

I suggest to close this issue and continue resolving somewhere else.

icing · 2023-05-23T17:19:43Z

Thanks @krowinski and @mariandomanik.

Indeed, inserting transfer encoding headers seems ill advised. A case where curl allows one to shoot ones own foot.

Will close the issue.

hktalent · 2023-06-07T02:06:30Z

me too:curl: (18) HTTP/2 stream 1 was reset

Mac OS 13.5 Beta
/usr/local/Cellar/curl/8.1.2/bin/curl --data-binary "@xxx_cn_httpx_json1.json" -sk -XPOST 'https://127.0.0.1:8081/doBlv?t=osint&tags=china,gov,httpx&c=httpx&q=hash'

* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: O=mkcert development certificate; OU=51pwn@123.local
*  start date: Nov  3 05:49:26 2022 GMT
*  expire date: Feb  3 05:49:26 2025 GMT
*  issuer: O=mkcert development CA; OU=51pwn@123.local; CN=mkcert 51pwn@123.local
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* using HTTP/2
* h2 [:method: POST]
* h2 [:scheme: https]
* h2 [:authority: 127.0.0.1:8081]
* h2 [:path: /doBlv?t=osint&tags=china,gov,httpx&c=httpx&q=hash]
* h2 [user-agent: Mozilla/5.0 (Windows NT 6.1; rv:45.0) Gecko/20100101 Firefox/45.0]
* h2 [accept: */*]
* h2 [content-length: 41802357]
* h2 [content-type: application/x-www-form-urlencoded]
* Using Stream ID: 1 (easy handle 0x7fa834815000)
> POST /doBlv?t=osint&tags=china,gov,httpx&c=httpx&q=hash HTTP/2
> Host: 127.0.0.1:8081
> User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:45.0) Gecko/20100101 Firefox/45.0
> Accept: */*
> Content-Length: 41802357
> Content-Type: application/x-www-form-urlencoded
>

icing · 2023-06-07T08:06:27Z

@hktalent not sure what I am looking at. Can you elaborate?

hktalent · 2023-06-07T12:19:25Z

@icing I have checked and confirmed that it is caused by asynchronous reading of post streams on the server side, and it has been resolved

thanks

- refs curl#11157 and curl#11175 where uploads get stuck or lead to RST streams - fixes our h2 send behaviour to continue sending in the nghttp2 session as long as it wants to. This will empty our send buffer as long as the remote stream/connection window allows. - in case the window is exhausted, the data remaining in the send buffer will wait for a WINDOW_UPDATE from the server. Which is a socket event that engages our transfer loop again - the problem in the issue was that we did not exhaust the window, but left data in the sendbuffer and no further socket events did happen. The server was just waiting for us to send more. - relatedly, there was an issue fixed that closing a stream with KEEP_HOLD set kept the transfer from shutting down - as it should have - leading to a timeout. Closes curl#11176

awahhab · 2023-07-23T05:35:38Z

Just for people who may drop by from search results.
Same issue mentioned in resolved in getsentry/sentry-laravel#712

hktalent · 2023-07-23T07:05:59Z

@awahhab
My problem has been solved
This is mainly caused by the backend code not handling the shutdown properly

thank you very much

bagder added the HTTP/2 label May 22, 2023

bagder added the needs-info label May 22, 2023

icing self-assigned this May 22, 2023

icing mentioned this issue May 22, 2023

http/2, unstuck uploads #11176

Closed

This comment was marked as off-topic.

Sign in to view

icing closed this as completed May 23, 2023

cleptric mentioned this issue May 23, 2023

DecoderPlugin - cURL 8.1.0 issue with HTTP/2 php-http/client-common#230

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Since 8.1.0 php-curl POST fails with HTTP/2 stream 1 was reset for {url} #11175

Since 8.1.0 php-curl POST fails with HTTP/2 stream 1 was reset for {url} #11175

mariandomanik commented May 22, 2023

bagder commented May 22, 2023

bagder commented May 22, 2023

mariandomanik commented May 22, 2023 •

edited by bagder

stayallive commented May 22, 2023 •

edited

icing commented May 22, 2023

cleptric commented May 22, 2023

This comment was marked as off-topic.

bagder commented May 23, 2023

S3B-4 commented May 23, 2023 •

edited

icing commented May 23, 2023

mariandomanik commented May 23, 2023

krowinski commented May 23, 2023 •

edited

mariandomanik commented May 23, 2023

icing commented May 23, 2023

hktalent commented Jun 7, 2023

icing commented Jun 7, 2023

hktalent commented Jun 7, 2023

awahhab commented Jul 23, 2023

hktalent commented Jul 23, 2023

Since 8.1.0 php-curl POST fails with HTTP/2 stream 1 was reset for {url} #11175

Since 8.1.0 php-curl POST fails with HTTP/2 stream 1 was reset for {url} #11175

Comments

mariandomanik commented May 22, 2023

bagder commented May 22, 2023

bagder commented May 22, 2023

mariandomanik commented May 22, 2023 • edited by bagder

stayallive commented May 22, 2023 • edited

icing commented May 22, 2023

cleptric commented May 22, 2023

This comment was marked as off-topic.

bagder commented May 23, 2023

S3B-4 commented May 23, 2023 • edited

icing commented May 23, 2023

mariandomanik commented May 23, 2023

krowinski commented May 23, 2023 • edited

mariandomanik commented May 23, 2023

icing commented May 23, 2023

hktalent commented Jun 7, 2023

icing commented Jun 7, 2023

hktalent commented Jun 7, 2023

awahhab commented Jul 23, 2023

hktalent commented Jul 23, 2023

mariandomanik commented May 22, 2023 •

edited by bagder

stayallive commented May 22, 2023 •

edited

S3B-4 commented May 23, 2023 •

edited

krowinski commented May 23, 2023 •

edited