Apache cannot restart if too many mutices from stats remain from unclean exits

[GoogleCodeExporter]

Originally eported by bogdan2ky, Nov 03, 2010, on the Page Speed issue tracker 
here: http://code.google.com/p/page-speed/issues/detail?id=389

------------------------

Mod was enabled and worked for a while, it was correctly speeding'up all of my 
sites:
cs.dude.ro, sportarges.ro, anvelopepitesti.ro etc.
then i played with the rewrite options in pagespeed.conf of 
/mod_pagespeed_beacon and /mod_pagespeed_statistics location, afer that apache 
won't work until i comment out the "LoadModule pagespeed_module"
Apache/2.2.11 (Ubuntu)
Ubuntu 9.04
PHP 5.2.6-3ubuntu4.6

apache error logs attached..

Original issue reported on code.google.com by mdste...@google.com on 11 Nov 2010 at 2:52

Attachments:

• apache.error.log

[GoogleCodeExporter]

From error log:

...
[Thu Nov 04 01:00:25 2010] [error] 
[1104/010025:ERROR:net/instaweb/apache/apr_statistics.cc(97)] Variable 
serf_fetch_cancel_count cannot create mutex: No space left on device 
/etc/apache2/mod_pagespeed/stats_mutex.serf_fetch_cancel_count
[Thu Nov 04 01:00:25 2010] [error] 
[1104/010025:ERROR:net/instaweb/apache/apr_statistics.cc(198)] Variable 
serf_fetch_cancel_count will not increment in PID 25060
[Thu Nov 04 01:00:25 2010] [crit] (28)No space left on device: mod_rewrite: 
could not create rewrite_log_lock
Configuration Failed

Original comment by sligocki@google.com on 11 Nov 2010 at 3:12

[GoogleCodeExporter]

Original comment by jmara...@google.com on 13 Nov 2010 at 12:54

[GoogleCodeExporter]

Sorry this should be part of the FAQ.  To monitor statistics across multiple 
Apace subprocesses, mod_pagespeed uses shared memory.  When Apache exits 
uncleanly, the shared memory is not always cleaned up.  We use a script to help 
clean up the old shared memory segments, which I've pasted below.

#!/bin/sh

# See 
http://carlosrivero.com/fix-apache---no-space-left-on-device-couldnt-create-acce
pt-lock
#
# This script might help cleanup any leftover resources that prevent
# Apache from restarting.  The error message might look a little something
# like this:
#
# [Sat Oct 16 21:22:46 2010] [warn] pid file /usr/local/apache2/logs/httpd.pid
# overwritten -- Unclean shutdown of previous Apache run?
#
# [Sat Oct 16 21:22:46 2010] [emerg] (28)No space left on device: Couldn't
# create accept lock (/usr/local/apache2/logs/accept.lock.16025) (5)
#
#   ./apache_cleanup $USER
#
# You may want to see the owners of the IPC blocks by running ipcs -s
# manually.  For example, you might need to run:
#
#   sudo net/instaweb/scripts/apache_cleanup www-data
# or
#   sudo net/instaweb/scripts/apache_cleanup root

apache_user=$1

for ipsemId in $(ipcs -s | grep $apache_user | cut -f 2 -d ' '); do
  echo ipcrm -s $ipsemId
  ipcrm -s $ipsemId
done
for ipsemId in $(ipcs -m | grep $apache_user | cut -f 2 -d ' '); do
  echo ipcrm -m $ipsemId
  ipcrm -m $ipsemId
done


However ultimately I think mod_pagespeed should try to do this cleanup on 
startup if required, so I'll leave this bug open.

Original comment by jmara...@google.com on 17 Nov 2010 at 11:29

[GoogleCodeExporter]

Original comment by jmara...@google.com on 18 Nov 2010 at 2:17

• Changed title: Apache cannot restart if shared memory segments from stats remain from unclean exit

[GoogleCodeExporter]

Note that hanging shared memory segments and hanging cross-process mutices are 
slightly different problems.  The former only breaks the statistics (everything 
else still works) while the latter can keep apache from starting at all.

Original comment by abl...@google.com on 18 Nov 2010 at 2:50

• Changed title: Apache cannot restart if too many mutices from stats remain from unclean exits
• Changed state: Accepted

[GoogleCodeExporter]

Hi I need some help, running this script because I get the error...


[root@rdns ~]# touch cleanup.py
[root@rdns ~]# nano cleanup.py (this is where I copied in the entire 
script/including comments into the file)
[root@rdns ~]# python cleanup.py
  File "cleanup.py", line 24
    apache_user=$1
                ^
SyntaxError: invalid syntax


Any ideas why this would error?

The reason I'm trying to run this script is because I started to get the 
following error logs...

[Wed Feb 16 12:16:33 2011] [info] Server: Apache/2.2.3, Interface: 
mod_ssl/2.2.3, Library: OpenSSL/0.9.8e-fips-rhel5
[Wed Feb 16 12:16:34 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0216/121634:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
serf_fetch_outstanding_count cannot create mutex: No space left on device 
/var/www/mod_pagespeed/files/statistics/stats_mutex.serf_fetch_outstanding_count
[Wed Feb 16 12:16:34 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0216/121634:ERROR:net/instaweb/apache/apr_statistics.cc(255)] Statistics 
initialization failed in pid 1592

Kind regards,
Mark

Original comment by storm.m...@gmail.com on 16 Feb 2011 at 12:59

[GoogleCodeExporter]

That's a bash script, not Python.  Please type this:

  mv cleanup.py cleanup.sh
  chmod +x cleanup.sh
  ./cleanup.sh $USER

Please let us know if this helps.

Original comment by jmara...@google.com on 16 Feb 2011 at 1:08

[GoogleCodeExporter]

I'm able to run this but now I'm unable to have the module enabled as it 
crashes ALL websites on our server.

I'm just about to run some tests and see what the error logs show up.

Original comment by storm.m...@gmail.com on 16 Feb 2011 at 2:02

[GoogleCodeExporter]

[Wed Feb 16 14:34:48 2011] [info] Server: Apache/2.2.3, Interface: 
mod_ssl/2.2.3, Library: OpenSSL/0.9.8e-fips-rhel5
[Wed Feb 16 14:34:48 2011] [notice] mod_bw : Memory Allocated 0 bytes (each 
conf takes 32 bytes)
[Wed Feb 16 14:34:48 2011] [notice] mod_bw : Version 0.8 - Initialized [0 Confs]
[Wed Feb 16 14:34:48 2011] [info] mod_fcgid: Process manager 7712 started
[Wed Feb 16 14:34:48 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0216/143448:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
serf_fetch_bytes_count cannot create mutex: No space left on device 
/var/www/mod_pagespeed/files/statistics/stats_mutex.serf_fetch_bytes_count
[Wed Feb 16 14:34:48 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0216/143448:ERROR:net/instaweb/apache/apr_statistics.cc(255)] Statistics 
initialization failed in pid 7710
[Wed Feb 16 14:34:48 2011] [notice] mod_python: Creating 4 session mutexes 
based on 256 max processes and 0 max threads.
[Wed Feb 16 14:34:48 2011] [error] (28)No space left on device: mod_python: 
Failed to create global mutex 0 of 4 (/tmp/mpmtx77100).
Configuration Failed
[Wed Feb 16 14:34:48 2011] [info] mod_fcgid: Process manager 7712 stopped
[root@rdns httpd]# nano /etc/httpd/conf.d/pagespeed.conf
[root@rdns httpd]# service httpd restart

Original comment by storm.m...@gmail.com on 16 Feb 2011 at 2:38

[GoogleCodeExporter]

2 things:

1. The instructions I gave above are not exactly right: rather than $USER you 
should use the owner of the httpd process, which is dependent on your 
distribution.  E.g. on Ubuntu it's usually www-user.  Please try that script 
again using that.

2. If Apache still fails to start then an OS reboot will be required.

I see from your log that mod_python also consumes global mutexes, which means 
that disabling mod_pagespeed will not be sufficient.  You'll need to reboot.

Original comment by jmara...@google.com on 16 Feb 2011 at 2:42

[GoogleCodeExporter]

Hi, this has worked. I had to run the script like so...

./cleanup.sh apache

Original comment by storm.m...@gmail.com on 16 Feb 2011 at 2:50

[GoogleCodeExporter]

"However ultimately I think mod_pagespeed should try to do this cleanup on 
startup if required, so I'll leave this bug open."

I disagree -- quite strongly, in fact! It's entirely possible for a single 
machine to have multiple Apache instances running on it, under the same user.

Implementing this would negatively impact every other Apache instance on the 
machine, possibly even breaking every other one.

In our workspace (well beyond 200 boxes), we have multiple Apaches running on a 
single machine, with each Apache handling a fraction of the work-load. This 
way, if any single Apache starts having issues, we can narrow it down to a 
specific sample of sites as being problematic, and work from there.

Original comment by Wesley.S...@gmail.com on 28 Feb 2011 at 10:02

[GoogleCodeExporter]

Well, I think the only real solution is to have it as a configuration setting. 
For users like myself having this automatic cleanup is essential, but for other 
users with multiple Apache instances NOT having it is also essential. So this 
leaves the only solution being a configurable option.

Original comment by storm.m...@gmail.com on 28 Feb 2011 at 10:20

[GoogleCodeExporter]

We are on our way to a good solution to this problem.

Original comment by jmara...@google.com on 7 Apr 2011 at 4:25

[GoogleCodeExporter]

Original comment by jmara...@google.com on 6 May 2011 at 2:36

• Changed state: Fixed
• Added labels: release-note

[didiandalucia]

The Problem is not resolved! We get in Version mod_bw Version 0.92 the same problem and every day the vserver hang with mod_bw Memory Allocated 0 bytes, but the server has 7 GB RAM free!

Under https://wiki.1awww.com/index.php?title=Plesk-Panel_Auto-Update_Probleme we have written a workarround!

[jmarantz]

didiandalucia: is your issue related to mod_pagespeed? Note that this issue has been fixed since early 2011 and we haven't heard any other complaints of this since then (in our migration from Google Code to Github, the dates on the comments got a little confusing).

It sounds like what you observed is only an issue for mod_bw.

[didiandalucia]

Yes this is about mod_bw. I have found a other project in sourceforge with bw_mod. Thanks for answer