Provisioning apache2 prefork replacement killed my server(s) performance

[mcheck]

One thing I have been chasing after for several weeks now, is the performance of apache and varnish. Turns out where I was looking at varnish being so finicky, it was apache2 when the changes from @EvanK were merged into master at (I think) 0.2.29.

When the prefork.conf file was broken out to determine values from calculations, my 1G server provisioning performance was horrendous. 30s to load a jquery file. Once I isolated and found that it was the prefork settings, I changed them to those set with a version BEFORE 0.2.29 and performance was restored.

I'm no ansible or apache2 expert, so I can't understand why less requests per child will give better results. But overall, I can tell you the results is that prefork.conf 1 is TERRIBLE compared to prefork 2, below.

So, 1G server (as well as other current settings) with 0.2.29 and later:

prefork.conf (TERRIBLE performance) on 1G server

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_prefork_module>
     StartServers          2
     MinSpareServers       2
     MaxSpareServers       3
     MaxClients            1
     ServerLimit           1
    MaxRequestsPerChild   1000
</IfModule>

So, 1G server (as well as other current settings) with 0.2.28 and earlier:

prefork.conf (great/expected performance) on 1G server

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>

[ericclemmons]

Wow, you're right. MaxClients should have never been that low! That's like running a single-core node server!

With the current release, you can set those settings in your provisioning/group_vars/webservers file, as seen here:

https://github.com/genesis/wordpress/blob/master/generator/app/templates/provisioning/group_vars/webservers#L13-L18

You can see how this is all calculated here:

https://github.com/genesis/wordpress/blob/master/provisioning/roles/common/templates/prefork.conf

It looks like our low-end is severely off.

The one issue we do know exists is MaxRequestsPerChild being 0, which is unlimited. This can cause serious performance & memory leak issues. Your problems are almost certainly because of MaxClients.

Do you happen to know of a good resource for performance-tuning Apache based on box memory size?

[mcheck]

Sorry, I don't know of any offhand.

[EvanK]

I know that we are currently calculating MaxClients from the larger of (A) available memory divided by half php's memory limit or (B) the number of virtual cpus on the server.

We can instead have a final check to give it a minimum value of...let's say between 2 and 5?

[mcheck]

I'm having trouble with this issue again, I think.

I have a digital ocean vps that is 1Gb/1 core/30Gb ubuntu 12.04 and the genesis provisioning sets the prefork at:

<IfModule mpm_prefork_module>
    StartServers          2
    MinSpareServers       2
    MaxSpareServers       3
    MaxClients            7
    ServerLimit           7
    MaxRequestsPerChild   1000
</IfModule>

MaxClients and ServerLimit (well, really all but the MaxRequestsPerChild) seem too low. Would you say so?

Is there a good reference for fine tuning the prefork that gives reasonable estimates based on box size?

As it is, my server is very pokey and wasn't like this before we started calculating the prefork settings. As the original shows up top, those value seem more in line, but I don't want to change well known values to configs that may result in memory issues.

Thanks!

[mcheck]

@EvanK @ericclemmons
Along those lines, I notice that the default apache prefork setting does not specify ServerLimit.
In the docs it describes ServerLimit as should be set no higher than MaxClients. So for the calculated prefork settings above, the setting is correct (i.e. MaxClients==ServerLimit), but these numbers are WAY lower than the apache default MaxClients 150 (from the apache2 extras for prefork) and the MaxClients 256 from the MaxClients docs

Suggestions on optimal settings? Thanks!

[AntJanus]

@mcheck I'm currently digging into the best configuration for a server. The MaxClients at 7 is way too low, obviously, but Apache's default is not a good way to go either. A MaxClients of 256 is much worse as well.

The problem with pretuning it out of the box (and I'll do my best to create some type of equation or logic) is that each application works differently and takes up a different amount of RAM. Here's a good equation to follow:

MaxClients = (Total RAM - Operating System memory usage - MySQL memory usage) / size per apache process

The last bit (size per apache process) is the crucial one to get the best results. Generally this size can range between 12mb - 30mb per process. Let's try to be average and say that your site takes up 20mb per process and MySQL takes up 70mb (based on my own site results). For a 1gb machine that'd be:

MaxClients = 1024mb - 70mb (mysql) - 512mb (varnish overhead) / 20mb

That leaves you with a prefork MaxClients at around 20-30 since varnish is eating up a good chunk (but then again, it should serve all of your dynamic requests).

For the rest of the values, it all depends on how you use your server. Is your server ONLY for serving web (ie. you're not running any other services on it)? I've seen a lot of advice on the topic but it's hard to tell what the number should be. From what I've read, anywhere from leaving it to defaults to setting all the numbers close to the MaxClients. Here's what i'm thinking is a good rule of thumb:

• MaxSpareServers cannot exceed MaxClients but if you want to keep your machine heavily loaded (and eager to respond), I'd keep it closer to the MaxClients, especially for 1gb-2gb machines. In this example, i'd keep it at 15 to 20 (if you chose to go the 30 MaxClients route)
• MinSpareServers should be somewhere lower but still pretty high up so that there are always processes available to handle requests. Let's say 10 in this case
• StartServers should be somewhere in the middle, again, closer to MaxSpareServers if you want the machine to be more eager. We can do a 15
• MaxRequestsPerChild is good at 1000 in most cases.

Hope that helps! I'll be researching more into this and #92 should change the size of apache processes so that will impact your tuning in the future.

[AntJanus]

@mcheck I should probably also let you know that you should test and monitor your server when you change these values. If things get out of hand, lower the *SpareServers values to a 2,3,3 spread (Min: 2, Start: 3, Max: 3) as that seems to be the safest (non-bleeding-edge) valueset that I've seen across configs.

[AntJanus]

I just realized that it's not really possible to provision the server with the correct amount of MaxClients because there's no way to know the size of each apache process before the server actually runs.

Once the server (and final application) is running, THEN we can estimate the MaxClients since they're based off the amount of RAM each process will consume.

The reason the original prefork config ran faster is because it kept spinning up new processes for users. The end results (given enough traffic) would have been what's called thrashing which is where the RAM overflows into swap at which point the server would no longer respond correctly (memory would keep switching between RAM and swap without doing any productive work whatsoever).

[ericclemmons]

So, what's the best course of action? Only set sane defaults and encourage the user to change the settings after monitoring application performance?

[mcheck]

@AntJanus Thanks for all the work and info.
Here are my current prefork settings:

<IfModule mpm_prefork_module>
    StartServers          3
    MinSpareServers       3
    MaxSpareServers       3
    MaxClients            7
    ServerLimit           7
    MaxRequestsPerChild   5000
</IfModule>

But the server is far too slow. It is a 1G server without varnish and my apache is at 62m and mysql 159m ! Basically double yours!

So my numbers according to your calcs would be:

MaxClients =
(1024m - 159m (mysql) - 32m (reserve for no good reason) ) / 62m (apache/php) ) = 13.4m

I am not running anything else on the server, really. I have it all locked down and is specific to only apache, php and mysql. I do not use varnish because the I could never get it to reliably run with ssl. We're running an ecomm shop and the ssl, obviously, is required.

But as you see above, my mem numbers are double yours. Can you help me determine a reason for that? I was starting to look down the path of PHP-FPM and nginx because of the mem sizes and the voodoo that seems to be the MacClients due to the size of these processes. A long rabbit hole....

What do you use for monitoring the system? I would like something that would allow me to react to apache and mysql performance.

@ericclemmons I think that sane defaults makes sense then encourage changes based on application monitoring. My $0.02.

[AntJanus]

@mcheck I think you're looking at the parent apache process, not the child processes which are under: httpd (there will be several of those). But if not, then, set your MaxClients to 15 (along with ServerLimit) and that should work well! As far as MySQL, it varies so I would not worry about it.

The best thing for monitoring would be New Relic because it will give you even the number of processes you use (it'll show httpd, 15 processes, at 765mb for example) . You can also use htop (the server should have provisioned with it!) which will give you the processes, look for apache processes (this time actually with the actual name apache) the virtual memory will say some ludicrously huge number (like 250mb) but there should be a longer number (in bytes) around there which should give you the real memory footprint.

@ericclemmons sane defaults would work best. Anything else will need proper tuning.

[mcheck]

@AntJanus Just wanted to say 'thanks' for the guidance and advice on the MaxClients. Using your revised formula, and my RES mem from htop, I plugged in:

    StartServers          5
    MinSpareServers       5
    MaxSpareServers       10
    MaxClients            14
    ServerLimit           14
    MaxRequestsPerChild   1000

And the server is noticeably snappier. Fantastic.

htop memory bar in the upper left seems to indicate that I'm right under full memory usage, which is great. When I install New Relic, I hope that will give me a better read over time and with load.

Thanks again.