Connection: Close not set when needed - breaks downloads.

#657 breaks downloads towards https://api.wordpress.org/ and many other sites when using Curl 7.29.0 (and perhaps other versions)

Error: RuntimeException: Failed to get url 'https://api.wordpress.org/core/version-check/1.7/?locale=en_US': cURL error 28: Operation timed out after 10000 milliseconds with 807 out of -1 bytes received.

It also causes issues with the REST API in Site Health with the error: REST API response: (http_request_failed) cURL error 28: Operation timed out after 10005 milliseconds with XXX out of XXX bytes received”

It also prevents WordPress plugin and core updates, basically anything that relies on the internal Curl handler in WordPress.

Adding back Connection: Close solves the issue. Please revert #657 ASAP and push the fix to WordPress 6.4

If you are just looking to fix the cURL timeouts issue in WordPress 6.4 Read here:

~~To fix the issue, replace /wp-includes/Requests/src/Transport/Curl.php with the following file: Curl.php.zip or see #838 (comment)~~

UPDATE: No need to do any manual fix, updating the WordPress Core to release 6.4.1 should work just fine now and resolve the issue.

@tomsommer

@tomsommer PR #657 fixes a bug. Reverting it recreates the bug. What makes this bug more important than the previous bug ?

I suggest someone do some research as to why the Curl version which is checked for now (7.22) is not correct/causing problems, if another version should be checked for instead (or a Curl constant) and iterate on the previous fix.

@jrfnl

@jrfnl I agree with that, I suggest the author of #657 should account for that (or the person who merged this)? Right now the change breaks previously working versions of Curl, versions which are bundled with production PHP versions (Remi PHP) and Operating Systems (RHEL7/CentOS 7), which makes this bug more important - IMHO.

@tomsommer

@tomsommer Anyone can work on this, it's open source, so contributions are welcome.

For info, WP 6.4 issues mentioned in the forums in regards to timeouts, REST API problems in Site Health and issues reaching WP plugin/update servers are probably related to this issue (since curl fails on remote endpoints).
https://wordpress.org/support/forum/how-to-and-troubleshooting/
https://wordpress.org/support/forum/installation/

Even if you fix this now the issue prevents any future auto-upgrade to a 6.4.1, since it breaks Curl requests, so the only way for people to update would be manually. The longer you wait, the bigger the problem will become. Just FYI.

We are applying a patch to our 6.4 customers which reverts #657 directly in the 6.4 source, this solves the issue. I recommend others do this as well.

We are applying a patch to our 6.4 customers which reverts #657 directly in the 6.4 source, this solves the issue.

And how will the related websites receive this update? 6.4 is already installed and Updates are not possible?

@tomsommer

Chiming in here. We're also seeing this widespread across our platform with sites on 6.4. Using the patch @tomsommer suggested seems to resolve the issue.

If this indeed only affects curl 7.29, it seems the proper fix would be to add another condition to limit the scope of that header change.

However, I'm concerned how this is even going to get applied to sites now without manual intervention.

@jrfnl

@jrfnl I'm following the PR trail for this change. It looks like the original submitter was having issues with Wordpress updates on a very specific environment (with an IPS installed). I'm curious how this was ever tested, as none were ever provided, and very little information (if any) is available on what exactly was broken with curl to begin with.

I feel like the original PR here should be a straight rollback. There is a very real issue present now that is extremely impactful and easily reproducible, which cannot be said for the original issue that was raised.

I've opened #839 to address this.

extremely impactful

tens of thousands of sites for us. i'm sure many more elsewhere.

Can't update sites. Autoupdating WordPress core is marked now as default... Please, revert it... Problems after updating to WordPress 6.3 and more problems after updating to WordPress 6.4.

Same error here. You can temporary fix it changing line 367 in wp-include/Requests/src/Transport/Curl.php from:

if ($this->version < self::CURL_7_22_0 && !isset($headers['Connection'])) {

to:

if (!isset($headers['Connection'])) {

Can't update sites. Autoupdating WordPress core is marked now as default... Please, revert it... Problems after updating to WordPress 6.3 and more problems after updating to WordPress 6.4.

Ditto.

@gregoriopellegrino

Same here! All my websites locked after updating to Wordpress 6.4. Those one without update, working as normally.

When trying to ejecute some Ajax calls (backups, WP updates)... it gives a timeout. For WP updates the .maintenance file need to be removed manually.

I'm using CentOS7 with Plesk 18.0.56 .

@gregoriopellegrino change solves the problem

Heres the related bug report in trac: https://core.trac.wordpress.org/ticket/59842

Given the impact, this should be hotfixed right away in WP Core (not waiting for a Requests release) and then we need to talk to the WP systems team to see how we can get that hotfix onto the running systems. I assume that this might be something that can be fixed at the server level for api.wordpress.org.

NOT Fixed :( Done everything I was asked to do, STILL hanging and erroring ;(

The WordPress Core 6.4.1 release including this fix is scheduled to be released tomorrow Nov 9 @ 00:00 UTC.

The RC1 is already available for testing: https://make.wordpress.org/core/2023/11/08/wordpress-6-4-1-rc1-is-now-available/

@jrfnl

@jrfnl I agree with that, I suggest the author of #657 should account for that (or the person who merged this)? Right now the change breaks previously working versions of Curl, versions which are bundled with production PHP versions (Remi PHP) and Operating Systems (RHEL7/CentOS 7), which makes this bug more important - IMHO.

My comment isn't helpful, but the following should be considered.

curl version 7.29.0 was released on February 6 2013. The following 78 security problems are known to exist in this version.

https://curl.se/docs/vuln-7.29.0.html

I understand some providers are still running CentOS7 and RHEL7 still in production, but why?

There's a goal to move to PHP 8.1 where most Linux distributions are running curl 7.6+

@jrfnl

@jrfnl I agree with that, I suggest the author of #657 should account for that (or the person who merged this)? Right now the change breaks previously working versions of Curl, versions which are bundled with production PHP versions (Remi PHP) and Operating Systems (RHEL7/CentOS 7), which makes this bug more important - IMHO.

My comment isn't helpful, but the following should be considered.

curl version 7.29.0 was released on February 6 2013. The following 78 security problems are known to exist in this version.

https://curl.se/docs/vuln-7.29.0.html

I understand some providers are still running CentOS7 and RHEL7 still in production, but why?

There's a goal to move to PHP 8.1 where most Linux distributions are running curl 7.6+

Hosting environments can be incredibly difficult to migrate for reasons that are mixed technical and logistical. I'm finally making the move to Almalinux and looking forward to leaving the legacy behind but I'm only hosting about 110 or so sites. I can only imagine the headache for larger providers hosting thousands.

Hosting environments can be incredibly difficult to migrate for reasons that are mixed technical and logistical. I'm finally making the move to Almalinux and looking forward to leaving the legacy behind but I'm only hosting about 110 or so sites. I can only imagine the headache for larger providers hosting thousands.

I agree with your statement. However, although incredibly difficult, it's not impossible and that's the duty of a hosting provider. So either do tough things or innovate and do tougher things in a more efficient fashion. Stick to the 80/20 rule, if you successfully migrate 80% of your clients, and 20% are stuck then deal with those separately but don't force the 80% to suffer from something like this.

We're quick to call out clients who have 10-year-old plugins and passwords but not hosting providers on 10-year-old libraries.

Hosting environments can be incredibly difficult to migrate for reasons that are mixed technical and logistical. I'm finally making the move to Almalinux and looking forward to leaving the legacy behind but I'm only hosting about 110 or so sites. I can only imagine the headache for larger providers hosting thousands.

I agree with your statement. However, although incredibly difficult, it's not impossible and that's the duty of a hosting provider. So either do tough things or innovate and do tougher things in a more efficient fashion. Stick to the 80/20 rule, if you successfully migrate 80% of your clients, and 20% are stuck then deal with those separately but don't force the 80% to suffer from something like this.

We're quick to call out clients who have 10-year-old plugins and passwords but not hosting providers on 10-year-old libraries.

I'm going to chime in here, as I work for one of the hosting providers affected by this.

The code above affected any version of curl that did not default to http2. This included versions all the way up to 7.47.0.
This isnt an issue of PHP version(s), its the core library in the OS. CentOS 7 is still supported, and due to the discontinuation of CentOS as a whole, MANY providers were left without viable upgrade paths.
No official packages exist for upgrading curl on CentOS 7 beyond 7.29. Due to the number of things that depend on curl, its extremely ill-advised to try and build your own packages for it, or even use 3rd party builds.
Many of those vulnerabilities you've listed above have their patches backported to the vendor supplied 7.29 library.
Calling out a client with a handful of sites for not updating is not the same as a hosting provider, who is responsible for hundreds of thousands of sites. One cannot simply ignore the fact that the 20% that would be affected by a migration/upgrade could very well tank a business.
I 100% agree that many of the hosts running EOL or near-EOL OS's need to update. However, thats a long process that isnt likely to happen before EOL. The short of it is, web hosts have to continue to support EOL software for the sake of supporting our customers, thats never going to change.

Theres a lot more intricacies to this that I haven't went over, but its absolutely not a simple/easy task when you're talking about hosting at scale.

Hosting environments can be incredibly difficult to migrate for reasons that are mixed technical and logistical. I'm finally making the move to Almalinux and looking forward to leaving the legacy behind but I'm only hosting about 110 or so sites. I can only imagine the headache for larger providers hosting thousands.

I agree with your statement. However, although incredibly difficult, it's not impossible and that's the duty of a hosting provider. So either do tough things or innovate and do tougher things in a more efficient fashion. Stick to the 80/20 rule, if you successfully migrate 80% of your clients, and 20% are stuck then deal with those separately but don't force the 80% to suffer from something like this.
We're quick to call out clients who have 10-year-old plugins and passwords but not hosting providers on 10-year-old libraries.

I'm going to chime in here, as I work for one of the hosting providers affected by this.
1. The code above affected any version of curl that did not default to http2. This included versions all the way up to 7.47.0.

2. This isnt an issue of PHP version(s), its the core library in the OS. CentOS 7 is still supported, and due to the discontinuation of CentOS as a whole, MANY providers were left without viable upgrade paths.

3. No official packages exist for upgrading curl on CentOS 7 beyond 7.29. Due to the number of things that depend on curl, its extremely ill-advised to try and build your own packages for it, or even use 3rd party builds.

4. Many of those vulnerabilities you've listed above have their patches backported to the vendor supplied 7.29 library.

5. Calling out a client with a handful of sites for not updating is not the same as a hosting provider, who is responsible for hundreds of thousands of sites. One cannot simply ignore the fact that the 20% that would be affected by a migration/upgrade could very well tank a business.

6. I 100% agree that many of the hosts running EOL or near-EOL OS's need to update. However, thats a long process that isnt likely to happen before EOL. The short of it is, web hosts have to continue to support EOL software for the sake of supporting our customers, thats never going to change.
Theres a lot more intricacies to this that I haven't went over, but its absolutely not a simple/easy task when you're talking about hosting at scale.

No patch has been back ported since at least two years.

https://git.centos.org/rpms/curl/releases

Hosting environments can be incredibly difficult to migrate for reasons that are mixed technical and logistical. I'm finally making the move to Almalinux and looking forward to leaving the legacy behind but I'm only hosting about 110 or so sites. I can only imagine the headache for larger providers hosting thousands.

I agree with your statement. However, although incredibly difficult, it's not impossible and that's the duty of a hosting provider. So either do tough things or innovate and do tougher things in a more efficient fashion. Stick to the 80/20 rule, if you successfully migrate 80% of your clients, and 20% are stuck then deal with those separately but don't force the 80% to suffer from something like this.
We're quick to call out clients who have 10-year-old plugins and passwords but not hosting providers on 10-year-old libraries.

I'm going to chime in here, as I work for one of the hosting providers affected by this.
1. The code above affected any version of curl that did not default to http2. This included versions all the way up to 7.47.0.

2. This isnt an issue of PHP version(s), its the core library in the OS. CentOS 7 is still supported, and due to the discontinuation of CentOS as a whole, MANY providers were left without viable upgrade paths.

3. No official packages exist for upgrading curl on CentOS 7 beyond 7.29. Due to the number of things that depend on curl, its extremely ill-advised to try and build your own packages for it, or even use 3rd party builds.

4. Many of those vulnerabilities you've listed above have their patches backported to the vendor supplied 7.29 library.

5. Calling out a client with a handful of sites for not updating is not the same as a hosting provider, who is responsible for hundreds of thousands of sites. One cannot simply ignore the fact that the 20% that would be affected by a migration/upgrade could very well tank a business.

6. I 100% agree that many of the hosts running EOL or near-EOL OS's need to update. However, thats a long process that isnt likely to happen before EOL. The short of it is, web hosts have to continue to support EOL software for the sake of supporting our customers, thats never going to change.
Theres a lot more intricacies to this that I haven't went over, but its absolutely not a simple/easy task when you're talking about hosting at scale.
No patch has been back ported since at least two years.

https://git.centos.org/rpms/curl/releases

Yes, and that includes a large number of the vulnerabilities listed above.

Yes, and that includes a large number of the vulnerabilities listed above.

Did some quick JSON filtering on: https://curl.se/docs/vuln-7.29.0.json

And it looks like 16 vulnerabilities have been released after 2022-04-27, which falls outside the last patch window.

So, yeah...

Guys. Please. This not the place to be smart on other peoples setups. Thanks.

@calvinalkan

@calvinalkan Tom is right, this isn't the place. If you'd like to have a more direct conversation about this, we can certainly take it offline. You can find me on the Wordpress Make Slack as nexflaszlo

I'm going to chime in here, as I work for one of the hosting providers affected by this.

The code above affected any version of curl that did not default to http2. This included versions all the way up to 7.47.0.

CentOS 8 has cURL 7.81 available via Stream? https://centos.pkgs.org/8-stream/centos-baseos-x86_64/curl-7.61.1-18.el8.x86_64.rpm.html but updates stopped in December 2021

This isnt an issue of PHP version(s), its the core library in the OS. CentOS 7 is still supported, and due to the discontinuation of CentOS as a whole, MANY providers were left without viable upgrade paths.

It's not the core issue, PHP versions do play a small role here. How are you providing customers PHP 8.1 on CentOS 7?

Were you not able to upgrade to CentOS 8? I know Stream was killed off in 2021, but it was released in 2019.

Siteground has PHP 7.4 with Curl 7.85.0 right now, which I'm super surprised about.

No official packages exist for upgrading curl on CentOS 7 beyond 7.29. Due to the number of things that depend on curl, its extremely ill-advised to try and build your own packages for it, or even use 3rd party builds.

Totally agree.

Many of those vulnerabilities you've listed above have their patches backported to the vendor supplied 7.29 library.

See Calvin's comments.

Calling out a client with a handful of sites for not updating is not the same as a hosting provider, who is responsible for hundreds of thousands of sites. One cannot simply ignore the fact that the 20% that would be affected by a migration/upgrade could very well tank a business.

I'm afraid I have to disagree; it's a mentality not a management issue. But I'm okay with disagreement; it's healthy rapport.

Who said ignore? There's a tatical approach to this, move the customers you can, deal with the 20% separately with a stated time limit. Learn why those 20% had issues migrating/updating and change up your processes and tools. Next time you attack the same problem that 20% might be 10% or 5%.

I will repeat their name since it's a lived experience, but Siteground migrated quite a few times, and it was decently efficient and error-free. But I'm sure there was some issues, it's inevitable.

Pick your poison, lost 20% or not have this issue for your entire customer base?

I 100% agree that many of the hosts running EOL or near-EOL OS's need to update. However, thats a long process that isnt likely to happen before EOL. The short of it is, web hosts have to continue to support EOL software for the sake of supporting our customers, thats never going to change.

I agree too, it's a long process. But the lifecycle of upgrading and migration and has been around since the 90's. It sucks to deal with, but you have to do it; putting it off makes it worse.

I'm afraid I have to disagree with continuing to support EOL software at this scale, it's bad hygene. It's the same as WordPress supporting PHP 5.6 for years, slowing down the process of supporting PHP 8.1. If a customer chooses to run EOL software, don't punish the rest; provide options. Let the EOL customers live in their nightmare.

Theres a lot more intricacies to this that I haven't went over, but its absolutely not a simple/easy task when you're talking about hosting at scale.

I don't want to discredit your own experiences or your opinion. There will be tough things that we all need to do; at the end of the day, they need to get done. But putting them off to save the few doesn't make sense to me.

I think overall there's lots to learn here for everyone.

@calvinalkan

@calvinalkan Theres a lot to unpack there. Hit me up on Slack and I'd be happy to discuss. I dont want to drag this issue out and muddy the waters so to speak.

I will answer one question here, as it doesn't really require much thought/discussion. We can support PHP up to 8.3 on EL7 via the remi packages that are made available. https://rpms.remirepo.net/enterprise/7/

I'm going to lock the conversation in here at this point, as it went off track and doesn't relate to Requests development. The above discussions should be held in other settings.

In case something new comes up related to the above topic, please open a new issue.

Navigation Menu

Repository navigation

Connection: Close not set when needed - breaks downloads. #838

If you are just looking to fix the cURL timeouts issue in WordPress 6.4 Read here:

19 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Connection: Close not set when needed - breaks downloads. #838

Description

If you are just looking to fix the cURL timeouts issue in WordPress 6.4 Read here:

Activity

jrfnl commented on Nov 8, 2023

tomsommer commented on Nov 8, 2023

jrfnl commented on Nov 8, 2023

tomsommer commented on Nov 8, 2023

La-Geek commented on Nov 8, 2023

laszlof commented on Nov 8, 2023

laszlof commented on Nov 8, 2023

adrian-enspired commented on Nov 8, 2023

portavoznet commented on Nov 8, 2023

gregoriopellegrino commented on Nov 8, 2023

ajtruckle commented on Nov 8, 2023

javiermartingonzalez commented on Nov 8, 2023

laszlof commented on Nov 8, 2023

schlessera commented on Nov 8, 2023

19 remaining items

bbus99172 commented on Nov 8, 2023

schlessera commented on Nov 8, 2023

jordantrizz commented on Nov 9, 2023

itsensellc commented on Nov 9, 2023

jordantrizz commented on Nov 9, 2023

laszlof commented on Nov 9, 2023

calvinalkan commented on Nov 9, 2023

laszlof commented on Nov 9, 2023

calvinalkan commented on Nov 9, 2023

tomsommer commented on Nov 9, 2023

laszlof commented on Nov 9, 2023

jordantrizz commented on Nov 9, 2023

laszlof commented on Nov 9, 2023

schlessera commented on Nov 9, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions