New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jetpack JSON API access crashes PHP-FPM running PHP 7.1 #6629

Closed
michaelnordmeyer opened this Issue Mar 10, 2017 · 55 comments

Comments

Projects
None yet
@michaelnordmeyer
Copy link

michaelnordmeyer commented Mar 10, 2017

Steps to reproduce the issue

  • Install Jetpack as the only plugin, enable JSON API, connect self-hosted blog to WordPress.com
  • Access self-hosted blog using either WordPress.com or the WordPress.com macOS app to display blog posts

What I expected

Blog posts are displayed.

What happened instead

App didn't display any content, only outlines of soon to be displayed blog posts.

Observation

PHP task crashes on every API access.

This behaviour also happens for other related API access.

Environment

  • WordPress 4.7.3
  • Jetpack 4.7
  • Ubuntu Server 16.10 64bit
  • nginx/1.11.9
  • PHP 7.1.2-4+deb.sury.org~yakkety+1
  • MySQL 5.7.17-0ubuntu0.16.10.1

Error logs

These are the nginx error logs intertwined with php7.1-fpm.log lines for better readability. Hostname was replaced by example.com.

2017/03/08 22:03:45 [error] 24445#24445: *122060 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.0.89.76, server: example.com, request: "POST /xmlrpc.php?for=jetpack&token=0XhK4%284EfeBjlEf%2A%23XCat%239%5EL%407FKxHC%3A1%3A1&timestamp=1489010624&nonce=nR7WFrlVUR&body-hash=hurPrlvKTkYR0JyNzGVyo79r4LU%3D&signature=UxRZ3JYIGr698YYsa9kpIHEvOFw%3D HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.1-fpm.sock:", host: "example.com", referrer: "https://example.com/xmlrpc.php?for=jetpack&token=0XhK4%284EfeBjlEf%2A%23XCat%239%5EL%407FKxHC%3A1%3A1&timestamp=1489010624&nonce=nR7WFrlVUR&body-hash=hurPrlvKTkYR0JyNzGVyo79r4LU%3D&signature=UxRZ3JYIGr698YYsa9kpIHEvOFw%3D"

[08-Mar-2017 22:03:45] WARNING: [pool www] child 19742 exited on signal 11 (SIGSEGV) after 23.396837 seconds from start
[08-Mar-2017 22:03:45] NOTICE: [pool www] child 19746 started

2017/03/08 22:03:46 [error] 24445#24445: *122064 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.0.87.110, server: example.com, request: "POST /xmlrpc.php?for=jetpack&token=0XhK4%284EfeBjlEf%2A%23XCat%239%5EL%407FKxHC%3A1%3A1&timestamp=1489010626&nonce=cMBfZx3i1j&body-hash=AY82TQbx4apsl%2FQ7piLrvLuh4Iw%3D&signature=IxgM%2BxOhIH6dav33yzH3K7aWnCM%3D HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.1-fpm.sock:", host: "example.com", referrer: "https://example.com/xmlrpc.php?for=jetpack&token=0XhK4%284EfeBjlEf%2A%23XCat%239%5EL%407FKxHC%3A1%3A1&timestamp=1489010626&nonce=cMBfZx3i1j&body-hash=AY82TQbx4apsl%2FQ7piLrvLuh4Iw%3D&signature=IxgM%2BxOhIH6dav33yzH3K7aWnCM%3D"

[08-Mar-2017 22:03:46] WARNING: [pool www] child 19719 exited on signal 11 (SIGSEGV) after 88.240395 seconds from start
[08-Mar-2017 22:03:46] NOTICE: [pool www] child 19747 started

2017/03/08 22:03:46 [error] 24445#24445: *122065 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.0.87.118, server: example.com, request: "POST /xmlrpc.php?for=jetpack&token=0XhK4%284EfeBjlEf%2A%23XCat%239%5EL%407FKxHC%3A1%3A1&timestamp=1489010626&nonce=cPD7THgNoe&body-hash=I%2Bryd9dbgRyu0RxrIIWYMKCMA%2FI%3D&signature=ACfhZdsezGbDQNO98JfGnO9lbqQ%3D HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.1-fpm.sock:", host: "example.com", referrer: "https://example.com/xmlrpc.php?for=jetpack&token=0XhK4%284EfeBjlEf%2A%23XCat%239%5EL%407FKxHC%3A1%3A1&timestamp=1489010626&nonce=cPD7THgNoe&body-hash=I%2Bryd9dbgRyu0RxrIIWYMKCMA%2FI%3D&signature=ACfhZdsezGbDQNO98JfGnO9lbqQ%3D"

[08-Mar-2017 22:03:46] WARNING: [pool www] child 19746 exited on signal 11 (SIGSEGV) after 1.566387 seconds from start
[08-Mar-2017 22:03:46] NOTICE: [pool www] child 19749 started

2017/03/08 22:03:46 [error] 24445#24445: *122069 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.0.87.138, server: example.com, request: "POST /xmlrpc.php?for=jetpack&token=0XhK4%284EfeBjlEf%2A%23XCat%239%5EL%407FKxHC%3A1%3A1&timestamp=1489010626&nonce=DFmVKPQAwe&body-hash=wHY17ONIuFUDZEK3FDddt11BQa0%3D&signature=ZICaFXHuefFbvbVUqB4vYrVi%2FZ4%3D HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.1-fpm.sock:", host: "example.com", referrer: "https://example.com/xmlrpc.php?for=jetpack&token=0XhK4%284EfeBjlEf%2A%23XCat%239%5EL%407FKxHC%3A1%3A1&timestamp=1489010626&nonce=DFmVKPQAwe&body-hash=wHY17ONIuFUDZEK3FDddt11BQa0%3D&signature=ZICaFXHuefFbvbVUqB4vYrVi%2FZ4%3D"

[08-Mar-2017 22:03:46] WARNING: [pool www] child 19747 exited on signal 11 (SIGSEGV) after 0.199491 seconds from start
[08-Mar-2017 22:03:46] NOTICE: [pool www] child 19751 started
@jeherve

This comment has been minimized.

Copy link
Member

jeherve commented Mar 10, 2017

cc @georgestephanis who's worked on other PHP 7.1 oddities lately.

@Ipstenu

This comment has been minimized.

Copy link

Ipstenu commented Mar 16, 2017

This is affecting me too BTW.

@georgestephanis

This comment has been minimized.

Copy link
Member

georgestephanis commented Mar 20, 2017

@Ipstenu @michaelnordmeyer :

  • Can you get any sort of a backtrace before the segfault?
  • Does it work if you disable opcache via init_set() ?
  • Does it work if you pull back to PHP 7.0?
  • Does it work if you disable the 'related posts' module?
@michaelnordmeyer

This comment has been minimized.

Copy link

michaelnordmeyer commented Mar 20, 2017

@georgestephanis:

  • No, my PHP is not a debug build
  • Yes
  • It worked before switching from PHP 7.0 to 7.1 some weeks ago. If 7.0 would have been installed I would have added test info to the issue in the first place.
  • Only Jetpack with JSON API is enabled, no other Jetpack modules or 3rd-party plugins
@Ipstenu

This comment has been minimized.

Copy link

Ipstenu commented Mar 20, 2017

  • No :(
  • No
  • It works on PHP 7 just fine.
  • That happens to be disabled anyway

A quick and amusing check, on PHP 7.1 http://public-api.wordpress.com/rest/v1/sites/ipstenu.org shows up as a 404, but it works fine on PHP 7.

@georgestephanis

This comment has been minimized.

Copy link
Member

georgestephanis commented Mar 22, 2017

Heard back on an upstream bug report that I believe is related -- I'm going to try and track down what we can do to work around this, but for info for y'all in progress -- https://bugs.php.net/bug.php?id=74213

@camaran

This comment has been minimized.

Copy link

camaran commented Mar 28, 2017

I'm also have this problem, switch to php 7.0.x and work

@michaelnordmeyer

This comment has been minimized.

Copy link

michaelnordmeyer commented Apr 14, 2017

After upgrading the environment today to

  • Jetpack 4.8.1
  • Ubuntu Server 17.04 64bit
  • PHP 7.1.4-1+deb.sury.org~zesty+1

there haven't been any crashes anymore. 👍

@camaran

This comment has been minimized.

Copy link

camaran commented Apr 18, 2017

Hi, i have this error again.
i think the problem is opcache

@jeherve jeherve reopened this Apr 19, 2017

@jeherve

This comment has been minimized.

Copy link
Member

jeherve commented Apr 19, 2017

@camaran Could you try to update PHP to 7.1.4? It includes the following fix, that may help you: https://bugs.php.net/bug.php?id=74213

Let me know how it goes!

@camaran

This comment has been minimized.

Copy link

camaran commented Apr 20, 2017

Hi, i installed php 7.1.4 but i still with the problem.
I can open a ticket on jetpack support if it's neccessary :-)

@rraptorr

This comment has been minimized.

Copy link

rraptorr commented Apr 20, 2017

I am also experiencing the same crash with PHP 7.1.4 (from deb.sury.org), Wordpress 4.7.3 and Jetpack 4.8.2 running on Debian Jessie.

@jeherve

This comment has been minimized.

Copy link
Member

jeherve commented Apr 21, 2017

@camaran @rraptorr Do you get errors similar to the ones mentioned in the original report above?

@rraptorr

This comment has been minimized.

Copy link

rraptorr commented Apr 22, 2017

I am getting segfaults on xmlrpc.php file. The problem is gone when switching back to php 7.0 or removing jetpack. This seems pretty similar to the original report.

I was testing every PHP 7.1 version, since PHP 7.1.0. I was always getting this segfault and I am getting it still with PHP 7.1.4.

@ChrisWiegman

This comment has been minimized.

Copy link
Contributor

ChrisWiegman commented Apr 22, 2017

After some testing I'm hitting the same issue:

Ubuntu 16.04
PHP v7.1.4-1+deb.sury.org~xenial+1
WordPress 4.7.4
Jetpack 4.8.2

The following in wp-config lets me perform a sync with Jetpack for the time being but I'm hoping this doesn't have to be permanent:

if ( defined( 'XMLRPC_REQUEST' ) && true === XMLRPC_REQUEST ) { ini_set( 'opcache.enable', 0 ); }

@xyzulu

This comment has been minimized.

Copy link

xyzulu commented Apr 26, 2017

php 7.1.4 does not help with this issue
Before it get's closed again, perhaps let a few of us test ;)

@michaelnordmeyer

This comment has been minimized.

Copy link

michaelnordmeyer commented Apr 30, 2017

Looks like I have the problem again after a while of success with either the WordPress.com and iOS app. 😩

The only configuration change in the last couple of days was removing php-xdebug. The rest were ordinary OS updates and reboots because of an updated kernel.

Selectively turning off the opcache does work for me as well:

if ( defined( 'XMLRPC_REQUEST' ) && true === XMLRPC_REQUEST ) { ini_set( 'opcache.enable', 0 ); }

@tbutcaru

This comment has been minimized.

Copy link

tbutcaru commented May 10, 2017

I have raised this issue last week.
If I switch back to php56 Wordpress Android App works, but with php7.1.4 doesn't work.

I have added opcache.enable = 0 to my domain's php.ini, but I cannot see any change.

@tbutcaru

This comment has been minimized.

Copy link

tbutcaru commented May 10, 2017

It is indeed a problem with opcache.
I have disabled opcache by creating a .user.ini file inside Wordpress's root dir containing the following line:
opcache.enable=0
and no more errors, Wordpress Android App works 💃

@AubreyHewes

This comment has been minimized.

Copy link

AubreyHewes commented May 15, 2017

Disabling opcache in wp-config.php for xmlrpc-only as @ChrisWiegman suggests fixes the problem;-) Cheers @ChrisWiegman !

The amount of headers sent/header validity still seems to me to be the possible root cause?

#3546
dokku/dokku#2607

My setup is a wp/jetpack via dokku (herokuish) ~ nginx-fpm

If you don't want the extra config in the wordpress config you can also blacklist xmlrpc in the opcache config...

http://php.net/manual/en/opcache.configuration.php#ini.opcache.blacklist-filename

#6106

@camaran

This comment has been minimized.

Copy link

camaran commented May 16, 2017

i think now is fixed with php 7.1.5

@rraptorr

This comment has been minimized.

Copy link

rraptorr commented May 16, 2017

Well, for me it segfaults just the same with 7.1.5

@xyzulu

This comment has been minimized.

Copy link

xyzulu commented Jun 9, 2017

@Ipstenu that isn't really a fix.. it's just a workaround IMHO

@Ipstenu

This comment has been minimized.

Copy link

Ipstenu commented Jun 9, 2017

I didn't call it a fix. I just said that it appears to be working as I'm one of the people facing this problem. If I don't report back with what has, and has not, worked, how can Automattic or anyone else know if their actual fix is headed in the right direction? :)

@xyzulu

This comment has been minimized.

Copy link

xyzulu commented Jun 9, 2017

@Ipstenu Sorry Mika, hope I didn't sound harsh.. this workaround has been mentioned before and will always work due to it disabling the feature in php that is not playing well with Jetpack in this instance. Going back months Automattic are well aware of this. (check the many threads where it is mentioned here on Github if you like)

Again.. no attack intended.. sorry for the wrong use of words

@Ipstenu

This comment has been minimized.

Copy link

Ipstenu commented Jun 9, 2017

Appears to always work.

More data, confirming the fix works in more environments, and replying to ones own bug reports, is considered good behavior in the open source community. I don't want to be DenverCoder9.

And it's spelled with an A. Not an E.

@xyzulu

This comment has been minimized.

Copy link

xyzulu commented Jun 9, 2017

It's an opcache error.. disabling opcache in php 7.1 will always work around this error

@Ipstenu

This comment has been minimized.

Copy link

Ipstenu commented Jun 9, 2017

If XMLRPC v OpCache is the only part that causes the core dump, sure.

@zinigor

This comment has been minimized.

Copy link
Member

zinigor commented Jun 9, 2017

Thank you all for the input! Just for the record, here's a bug report that I have submitted to PHP about this exact problem. It's not XMLRPC that causes the segfault, but apparently it's some code that gets called as part of the JSON API request that gets sent via XMLRPC.

If you happen to know why that code can cause a segfault with opcache, or maybe how I can narrow down the exact cause, I'll be very grateful:
https://bugs.php.net/bug.php?id=74543

@kevinlisota

This comment has been minimized.

Copy link

kevinlisota commented Jun 15, 2017

Has anyone tried this with PHP 7.1.6 yet? There is one opcache bug fix in that update.

Actually, I just tried 7.1.6 and the segfault bug still exists.

@lschuyler

This comment has been minimized.

Copy link

lschuyler commented Jun 16, 2017

Also reported here 3261025-t

@zinigor

This comment has been minimized.

Copy link
Member

zinigor commented Jun 19, 2017

@kevinlisota just tested on 7.1.6 too, processes still segfault for me when responding to WordPress.com JSON API. What were the issues that were fixed with opcache? Can you post some links please?

@kevinlisota

This comment has been minimized.

Copy link

kevinlisota commented Jun 19, 2017

Here is the changelog for PHP 7.1.6: http://php.net/ChangeLog-7.php#7.1.6

There is one bugfix to opcache here: https://bugs.php.net/bug.php?id=74596

@HansVanEijsden

This comment has been minimized.

Copy link

HansVanEijsden commented Jun 20, 2017

Changelog PHP 7.1.7RC1: https://github.com/php/php-src/blob/PHP-7.1.7/NEWS

I don't know if it fixes it?

. Fixed bug #74663 (Segfault with opcache.memory_protect and
validate_timestamp). (Laruence)
. Revert opcache.enable_cli to default disabled. (Nikita)

@lschuyler

This comment has been minimized.

Copy link

lschuyler commented Jun 22, 2017

Also reported in 3266547-t

@michakpl

This comment has been minimized.

Copy link

michakpl commented Jul 11, 2017

It's still not working in PHP 7.1.7

@kevinlisota

This comment has been minimized.

Copy link

kevinlisota commented Jul 15, 2017

@zinigor Any way to get PHP development team to look at the bug you filed? I see no activity on it

https://bugs.php.net/bug.php?id=74543

@kristarella

This comment has been minimized.

Copy link

kristarella commented Jul 20, 2017

Possible report 3332067-t

@zinigor

This comment has been minimized.

Copy link
Member

zinigor commented Jul 20, 2017

@kevinlisota A good way would be to create a simple way to reproduce the segfault, which I haven't been able to do. You can see how far I have been able to progress in the bug report, but I haven't had more time to dig in. If you are able to produce a contained reproduction case, that would help tremendously!

@lamdayap

This comment has been minimized.

Copy link

lamdayap commented Aug 14, 2017

https://wordpress.org/support/topic/jetpack-causing-502-bad-gateway/

User is experiencing similar issues with PHP 7.0.15-0ubuntu0.16.04.4

@aheckler

This comment has been minimized.

Copy link
Member

aheckler commented Aug 14, 2017

614936-zen

Sent them this workaround: #6629 (comment)

@mzakariya

This comment has been minimized.

Copy link

mzakariya commented Aug 16, 2017

630882-zen

Sent same work around as above. User would prefer to not troubleshoot.

@mzakariya

This comment has been minimized.

Copy link

mzakariya commented Aug 16, 2017

635556-zen

Sent same work around as above.

@danjjohnson

This comment has been minimized.

Copy link

danjjohnson commented Aug 17, 2017

620933-zen sent suggested workarounds.

@mzakariya

This comment has been minimized.

Copy link

mzakariya commented Aug 23, 2017

645154-zen

Sent same work around as above.

@railto

This comment has been minimized.

Copy link

railto commented Aug 23, 2017

Currently running on PHP 7.1.8 and having same issue. Disabling opcache works as a workaround

@psycholyzern

This comment has been minimized.

Copy link

psycholyzern commented Sep 4, 2017

PHP 7.1.9 and having the same problem.
Also disabling opcache fixed it too.

@liuyigh

This comment has been minimized.

Copy link

liuyigh commented Sep 5, 2017

PHP 7.1.8 on serverpilot and having the same problem. Trouble shooting for hours until reading this thread. Switch to 7.0 on serverpilot, then it works right away. Thanks, guys!

@liuyigh

This comment has been minimized.

Copy link

liuyigh commented Sep 5, 2017

The Jetpack people noticed me about the freshly baked 5.3 update today:

https://jetpack.com/2017/09/05/jetpack-5-3-php-7-1-compatibility/

Now works well with php 7.1. Hooray!

@jeherve jeherve closed this Sep 6, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment