UsePerVhostStatistics memory footprint

[pavel-paulau]

It looks like this option can increase memory footprint dramatically when number of hosts is relatively large.

pagespeed UsePerVhostStatistics off on (right after nginx restart, idle instance):

$ ps -C nginx -o rss,cmd
  RSS CMD
506312 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
663140 nginx: worker process                   
664364 nginx: worker process                   
657080 nginx: worker process                   
657080 nginx: worker process                   
657080 nginx: worker process

pagespeed UsePerVhostStatistics off (right after nginx restart, idle instance):

$ ps -C nginx -o rss,cmd
  RSS CMD
154272 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
156372 nginx: worker process                   
156364 nginx: worker process                   
156364 nginx: worker process                   
156364 nginx: worker process                   
156364 nginx: worker process

So the questions I have:

1. Is there any way to [systematically] estimate memory allocation (over time as well).
2. Is there any way to tune memory usage?
3. Is it possible to enable stats only for selected hosts?

[jeffkaufman]

You wrote "pagespeed UsePerVhostStatistics off" twice; which one is off and which one is actually on? I'd guess the first one is on, since that has higher memory usage, but I wanted to confirm.

Is there any way to [systematically] estimate memory allocation (over time as well).

Can you script a test workload?

Is there any way to tune memory usage?

For the statistics that's not something we have. You can adjust MessageBufferSize, but that defaults to 0. On the other hand, our statistics usage shouldn't rise over time. It's basically a large number of counters that all get initialized at 0.

Is it possible to enable stats only for selected hosts?

Sorry, there's not. Adding this would be practical, changing the code from asking "are there per-vhost statistics" to "are there per-vhost statistics for this vhost" but it's not something we've looked into.

[morlovich]

It seems to be ~240KiB per VHost for statistic object itself.

[morlovich]

Most of that isn't actually the counters (~12KiB), but the histograms.

[jeffkaufman]

Maybe we should add a way to turn off the histograms, and maybe also the counters. Turning off statistics entirely disables things like cache flushing, but that needs much less memory than full stats.

[pavel-paulau]

Sorry, there was a typo in the first statement.

240KiB is absolutely reasonable usage. I don't think that the majority of people maintains thousands of hosts on the same server, so it should be OK.

However, per-host switch is indeed very practical. I would recommend to close this ticket and file a separate enhancement request for per-host stats.

[jeffkaufman]

Renamed the ticket; now this is an enhancement request for functionality like:

pagespeed UsePerVhostStatistics off
server {
    # won't have its own stats
}
server {
    pagespeed UsePerVhostStatistics on;
    # will have its own stats
}