Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

3rd Party Plugin Compatibility: MainWP WordPress Management Plugin #629

Closed
raamdev opened this issue Dec 4, 2015 · 31 comments
Closed

3rd Party Plugin Compatibility: MainWP WordPress Management Plugin #629

raamdev opened this issue Dec 4, 2015 · 31 comments

Comments

@raamdev
Copy link
Contributor

raamdev commented Dec 4, 2015

This GitHub issue exists to research compatibility with MainWP, a WordPress Management Plugin that provides a Dashboard Server/Client allowing you to manage several WordPress sites from a central site.

The following issue with ZenCache Pro was reported (ZenCache Lite, as of v151107, seems unaffected):

I have a dedicated server. Centos 7, Apache 2.4.16, PHP 5.6.15, MariaDb 10.0. It has a number of domains on, managed through cPanel/WHM. Zend OpCache is active. No other caching.
[...]
If I did any of the following from the MainWP Master panel, the domain with ZenCache Pro would crash and require http to be restarted for the server. The Lite domains did not crash:

  • Update or install any plugin.
  • Search for plugins (from the Manage Plugins menu).

Referencing (internal) ticket: https://websharks.zendesk.com/agent/tickets/9798

@CotswoldPhoto
Copy link

I will further try to tie this down. I have deliberately taken one site on this server off ZenCache Pro and subsequent updates to that site have not taken it down. On the other sites, the updates do take place, and MainWP reports all updates are successful. But the sites have gone down. It has to be something to do with the handshake process that MainWP uses, connecting the child to the master. One might suspect that ZenCache Pro is detecting the update (as a result of something that the child is doing?) and flushing the cache, and must be doing so at a time when the MainWP child is using a critical resource.

@CotswoldPhoto
Copy link

OK. Spent an hour or so this afternoon. Also upgraded my server to PHP5.6.16 to see if that made any difference. The surest way to crash random sites (assuming they all have ZC Pro installed, activated and cache enabled), in MainWP plugins panel do a search on all domains for any active plugins containing the string 'zen'. Random sites will crash (never all, rarely none), requiring a http restart. If the ZC Pro plugin is active, but the cache off, the same search will cause no problems. All cache options (when the cache is active) are at default (except the include test string).

@raamdev
Copy link
Contributor Author

raamdev commented Dec 5, 2015

@CotswoldPhoto I noticed that you're running PHP 5.6; these issues are possibly related to #630 and #624.

@CotswoldPhoto
Copy link

OK, following suggestions on those and linked threads, I tried the following:

  1. Increased WP_MEMORY_LIMIT to 256M => No effect
  2. Set opcache.enable_cli=0 => No effect (was =1)
  3. Increased PHP memory_limit to 1024M => No effect
  4. Set opcache.memory_consumption=512 (was =256) => Fixed

My conclusion is that Zend OpCache is running out of memory. Now, this is a Server related issue. Luckily, I have a dedicated server and I manage it and footling around in Linux is not an issue for me, but, for others ....

My guess is that OpCache has a bug, which by me throwing resources at it gets around it. Bear in mind that I am on the latest release of PHP5.6 (5.6.16), that suggests that this bug still persists, maybe even into PHP7.0.

Trevor

@raamdev
Copy link
Contributor Author

raamdev commented Dec 6, 2015

Set opcache.memory_consumption=512 (was =256) => Fixed

@CotswoldPhoto Woohoo! Thank you SO much for testing that and reporting what you've found. cc @jaswsinc

@raamdev
Copy link
Contributor Author

raamdev commented Dec 6, 2015

@CotswoldPhoto Just to confirm: When you say "Fixed" are you referring to the Error 500 going away, or the issues with MainWP going away, or both?

@CotswoldPhoto
Copy link

Both.

@CotswoldPhoto
Copy link

Looking at the apache error log, it was reporting the same corruption error as other reports (not related to MainWP or ZenCache), ie zend_mm_heap corrupted. When you look at the bug reports for PHP5.5 through to PHP7, only PHP5.6 has this problem with Zend OpCache, with zero reports in PHP5.5 or PHP7 (clearly the PHP7 reports will mainly using the betas and Release Candidates, as the final release was only on December 3).

@CotswoldPhoto
Copy link

This morning it is back. Not sure if that is because the resources I gave it were used up, so I restarted the server and tried again, and 3 of my sites immediately gave 500's. I have now set my php.ini as follows:

opcache.memory_consumption=1024 # was 256, then up to 512, now up to 1024
opcache.interned_strings_buffer=32 # was 16
opcache.max_accelerated_files=8000 # was 4000
opcache.fast_shutdown=1
opcache.enable_cli=0 # was 1
opcache.validate_timestamps=1

So far no crashes. Early days. The same error in apache log (zend_mm_heap corrupted).

@jaswrks
Copy link

jaswrks commented Dec 6, 2015

@CotswoldPhoto Thanks for the follow-ups. Interesting!

I think you're right about there being a bug in Opcache. 1024 seems very high; i.e., like you have a memory leak somewhere. I can't imagine a site powered by WordPress requiring any more than about 128MB of RAM. 256 should be more than enough. That's memory for PHP scripts (i.e., the scripts themselves), nothing more. So unless you are using Opcache for something more (not likely), then I'd say 1024 is unnecessary; i.e., for you to need that much RAM there must be a leak somewhere.


opcache.validate_timestamps=1

Be sure to also set opcache.revalidate_freq

opcache.revalidate_freq = 900

Just in case you have a rogue script somewhere in the site, you might also want to have a look at this setting, which would help you guard against a huge PHP file trying to eat up all of your RAM. See: http://php.net/manual/en/opcache.configuration.php#ini.opcache.max-file-size


My guess is that OpCache has a bug, which by me throwing resources at it gets around it. Bear in mind that I am on the latest release of PHP5.6 (5.6.16), that suggests that this bug still persists, maybe even into PHP7.0.

Agree. I haven't seen this behavior in PHP 7 myself, but you could be right. In either case, if I were you I'd strongly consider either an upgrade to PHP 7 or a downgrade back to PHP 5.5. Just to see if that will correct this problem for you. At the moment, it seems like your best chance of success.

@jaswrks
Copy link

jaswrks commented Dec 6, 2015

@CotswoldPhoto A tool like this is quick and easy, and it might assist you in diagnosing the problem. See: https://github.com/rlerdorf/opcache-status

Or, you can just put together a PHP script and output the array and review.

<?php print_r(opcache_get_status()); ?>

@CotswoldPhoto
Copy link

OK. I had tried that tool already and it showed me nothing. I have re-set the opcache settings as follows:

opcache.memory_consumption=256
opcache.interned_strings_buffer=16
opcache.max_accelerated_files=4000
opcache.revalidate_freq=900
opcache.fast_shutdown=1
opcache.enable_cli=0
opcache.validate_timestamps=1
opcache.max_file_size=102400

So far all is OK. I will probably migrate to PHP7 when cPanel release 54 is out and gets to the 'current' tier, in a month or two.

@CotswoldPhoto
Copy link

What I CAN see is that by making these changes:

OpCache has used up almost all of the 256M straight away.
OpCache is getting only 55-65% hit rate (before it was getting 90-95%).
The sites are a LOT faster loading!

I am thinking that by limiting the filesize to 100K, I have stopped it loading any large files, hence the drop in hit rate, but freeing up the cache from these large files has let the small ones in and they must be the critical ones to loading WordPress fast.

We shall see if this works. It is shame that Zend have not (better) explained all the OpCache settings, but as time goes by bloggers are doing this for them.

@CotswoldPhoto
Copy link

I can confirm that the problem is back. I am going to push the memory back up for now.

@raamdev raamdev added this to the Next Release (Pro) milestone Dec 7, 2015
@CotswoldPhoto
Copy link

I have to confirm that only when I have increased the settings like this:

opcache.memory_consumption=1024
opcache.interned_strings_buffer=32
opcache.max_accelerated_files=8000
opcache.revalidate_freq=900
opcache.fast_shutdown=1
opcache.enable_cli=0
opcache.validate_timestamps=1
opcache.max_file_size=102400

It NOW seems 'stable'. I was 'lucky' in that WooCommerce just released an update. I ran it on one site at a time from MainWP, and each time the update crashed at least one site (not the one that was updated!!). It was only when the settings were as above did it all go OK. BTW, given 1024MB to play with, OpCache seems only to want under 400MB, but the hit rate is up to 80-90% now. Probably means I could increase the max file size safely.

@raamdev raamdev modified the milestones: Future Release (Pro), Next Release (Pro) Dec 16, 2015
@raamdev
Copy link
Contributor Author

raamdev commented Dec 16, 2015

@CotswoldPhoto A new version of ZenCache was just released that includes many bug fixes and enhancements. You might want to give this release candidate a try to see if it addresses any of the issues you were having here. See http://zencache.com/zencache-pro-v151216-rc-release-candidate/

@CotswoldPhoto
Copy link

I can confirm that this RC fixes the problem, although I AM having some issues, and these I have reported on another ticket.

@raamdev
Copy link
Contributor Author

raamdev commented Dec 16, 2015

@CotswoldPhoto Awesome! Thank you for confirming.

@CotswoldPhoto
Copy link

Sad to say, but it is doing it again :-(

@raamdev
Copy link
Contributor Author

raamdev commented Dec 18, 2015

@CotswoldPhoto Sorry to hear that. :( It sounds like this may be more of a compatibility issue between ZenCache and MainWP. Am I correct to assume that you've only been seeing these issues while using MainWP?

@CotswoldPhoto
Copy link

What I HAVE done that DOES seem to avoid the issue, is to install a WordPress OPCache plugin:

https://wordpress.org/plugins/opcache/

And running its Reset function before the update has NO effect on the cache, as though the OPCache has somehow become locked.

Running it immediately after the update has the desired effect of flushing the cache and then Apache does not crash. It is as though ZenCache has somehow grabbed the communication channel to the OPCache and locked it, rather like locking a record on a database.

@CotswoldPhoto
Copy link

Well, I thought that had worked, but it does not. Roll on PHP7 support for my server stack.

@raamdev
Copy link
Contributor Author

raamdev commented Dec 18, 2015

@CotswoldPhoto This definitely sounds like a bug in OPcache and your best bet for now (until you upgrade to PHP7) is probably to disable OPcache altogether. Since you're running WordPress, ZenCache is going to give you more bang-for-your-buck than OPcache when it comes to speeding up your site(s).

We haven't any any reports like this with PHP 5.5 + OPcache, however there have been a couple of similar reports with PHP 5.6 + OPcache, so it sounds to me like there's something buggy with OPcache in PHP 5.6. (We went through a similar issue as this with PHP 5.3 and PHP 5.4 when running the APC extension, which we discovered is a lot more buggy than OPcache.)

If you're running PHP as a module in Apache, you should be able to add this to your .htaccess file to disable OPcache:

php_flag opcache.enable Off

Or if it's running as a CGI/FastCGI, you can add this to a user.ini or system-wide php.ini:

opcache.enable=0

@CotswoldPhoto
Copy link

I can disable it by editing php.ini and stop the module loading. Easier to do that than to edit the .htaccess in every domain.

Thanks for all your help.

@raamdev raamdev removed this from the Next Release (Pro) milestone Dec 28, 2015
@raamdev raamdev modified the milestones: Future Release (Pro), Next Release (Pro) Dec 28, 2015
@raamdev raamdev modified the milestones: Next Release (Pro); 1st CC Notice, Future Release (Pro) Feb 13, 2016
@raamdev raamdev modified the milestones: Next Release, Future Release Feb 26, 2016
@raamdev raamdev modified the milestone: Next Release Apr 4, 2016
@raamdev
Copy link
Contributor Author

raamdev commented Nov 15, 2016

@renzms Following the improvements to the Comet Cache Pro updater in the last release, which made Comet Cache Pro compatible with ManageWP and InfiniteWP, we should run some tests against MainWP to see if we can close this GitHub issue.

@renzms
Copy link

renzms commented Nov 15, 2016

@raamdev

Cannot reproduce bug

MainWP Dashboard Site


WordPress Version: 4.6.1
Current WordPress Theme: Twenty Sixteen version 1.3
Theme Author: the WordPress team - https://wordpress.org/
Theme URI: https://wordpress.org/themes/twentysixteen/
Active Plugins: | mainwp-updraftplus-extension/mainwp-updraftplus-extension.php | mainwp/mainwp.php |
PHP Version: 7.0.10-2+deb.sury.org~xenial+1
MySQL Version: 10.0.27-MariaDB-0ubuntu0.16.04.1

Child MainWP Site


Child Site PHP 5.6
WordPress Version: 4.6.1
Current WordPress Theme: Twenty Sixteen version 1.3
Theme Author: the WordPress team - https://wordpress.org/
Theme URI: https://wordpress.org/themes/twentysixteen/
Active Plugins: comet-cache-pro/comet-cache-pro.php | mainwp-child/mainwp-child.php |
PHP Version: 5.6.22
MySQL Version: 10.0.27-MariaDB-0ubuntu0.16.04.1
Apache Version: Apache/2.4.10 (Debian)

Child Site PHP 7.0
WordPress Version: 4.6.1
Current WordPress Theme: Twenty Sixteen version 1.3
Theme Author: the WordPress team - https://wordpress.org/
Theme URI: https://wordpress.org/themes/twentysixteen/
Active Plugins: comet-cache-pro/comet-cache-pro.php | mainwp-child/mainwp-child.php |
PHP Version: 7.0.10
MySQL Version: 10.0.27-MariaDB-0ubuntu0.16.04.1
Apache Version: Apache/2.4.10 (Debian)

Steps taken

  • Configured MainWP for Main Dashboard site and Child site
  • Installed Comet Cache Pro v160917 and enabled on Child site
  • Used plugin search and installed/updated plugins via the Main Dashboard Site for the Child Site
  • Everything working as normal

@raamdev
Copy link
Contributor Author

raamdev commented Nov 16, 2016

@renzms Did you test updating Comet Cache Pro via MainWP?

@CotswoldPhoto
Copy link

For me, the last update (and none before that either) did not appear in the updates panel dashboard in MainWP. They only showed up when you visited the individual sites own admin.

@CotswoldPhoto
Copy link

So, minutes after I post that, up pops a notification in MainWP that a RC version is available on my dev site. Cool. Guess that's sorted then.

@renzms
Copy link

renzms commented Nov 18, 2016

@raamdev

Did you test updating Comet Cache Pro via MainWP?

Yup! 😄 Was able to update Comet Cache Pro 160917 to the latest Comet Cache Pro 161116-RC via MainWP.

Tested two ways:

  • first with adding Update Credentials on Child site + Notify for RC release
  • second, manual update via upload on MainWP Dashboard and remote install to Child site

@raamdev
Copy link
Contributor Author

raamdev commented Nov 18, 2016

@CotswoldPhoto writes...

Guess that's sorted then.

@renzms writes...

Was able to update Comet Cache Pro 160917 to the latest Comet Cache Pro 161116-RC via MainWP.

Great! It sounds to me like this issue has been resolved then. I'm going to mark this issue as closed for now, but if anyone discovers a problem here again I can reopen it.

@raamdev raamdev closed this as completed Nov 18, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants