Skip to content
This repository has been archived by the owner on Apr 21, 2023. It is now read-only.

Statistics failing with "cannot lock mutex: Invalid argument " #215

Closed
GoogleCodeExporter opened this issue Apr 6, 2015 · 19 comments
Closed

Statistics failing with "cannot lock mutex: Invalid argument " #215

GoogleCodeExporter opened this issue Apr 6, 2015 · 19 comments
Labels
auto-migrated Priority-Medium Type-Defect

Comments

@GoogleCodeExporter
Copy link 

Several users have complained of getting messages like this:

[Tue Feb 15 10:58:05 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0215/105805:ERROR:net/instaweb/apache/apr_statistics.cc(141)] 
Variable cache_hits cannot lock mutex: Invalid argument 

This disables the statistics (everything shows up as -1) but should not 
otherwise impact mod_pagespeed functionality.

Original issue reported on code.google.com by abl...@google.com on 17 Feb 2011 at 10:50
@GoogleCodeExporter
Copy link
Author 

Version: 0.9.15.3-404
OS: Ubuntu server 10.10 (2.6.35-25-server)
Rebooting: tried, problem persists
Statistics: I have never checked statistics, I found this problem only because 
error messages makes error log huge.

ls -l /var/mod_pagespeed/files/statistics/ | head
total 0

Original comment by justinas...@gtempaccount.com on 18 Feb 2011 at 12:22

@GoogleCodeExporter
Copy link
Author 

Thanks for the info.  If you're not interested in statistics and just want a 
clean logfile, you can add "ModPagespeedStatistics off" to your configuration 
file.

However, it'd be helpful for me if you could first restart the http server and 
check the logs right at startup.  There should be some other logs from 
apr_statistics.cc that indicate why the mutex could not be created in the first 
place.

Original comment by abl...@google.com on 18 Feb 2011 at 3:06

@GoogleCodeExporter
Copy link
Author 

Version: 0.9.15.3-404
OS: Ubuntu server 10.10 (2.6.35-25-server)
Have tried rebooting with noi success
statistics worked fine until I upgraded to 10.10
all statistics fail
total 0 is the result of the ls ... | head

First few lines of log after reboot and apache restart:


[Fri Feb 18 15:03:04 2011] [notice] ModSecurity for Apache/2.5.12 
(http://www.modsecurity.org/) configured.
[Fri Feb 18 15:03:04 2011] [notice] Original server signature: Apache/2.2.16 
(Ubuntu)
[Fri Feb 18 15:03:05 2011] [notice] Apache/2.2.16 (Ubuntu) PHP/5.3.3-1ubuntu9.3 
with Suhosin-Patch mod_ssl/2.2.16 OpenSSL/0.9.8o Apache/2.2.0 (Fedora) 
configured -- resuming normal operations
[Fri Feb 18 15:03:05 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0218/150305:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
html_rewrite_time_us cannot lock mutex: Invalid argument 
[Fri Feb 18 15:03:05 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0218/150305:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_time_us cannot lock mutex: Invalid argument 
[Fri Feb 18 15:03:05 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0218/150305:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_hits cannot lock mutex: Invalid argument 
[Fri Feb 18 15:03:05 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0218/150305:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_time_us cannot lock mutex: Invalid argument 
[Fri Feb 18 15:03:05 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0218/150305:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_hits cannot lock mutex: Invalid argument 
[Fri Feb 18 15:03:05 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0218/150305:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_time_us cannot lock mutex: Invalid argument 
[Fri Feb 18 15:03:05 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0218/150305:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_hits cannot lock mutex: Invalid argument 
[Fri Feb 18 15:03:05 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0218/150305:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_time_us cannot lock mutex: Invalid argument

Original comment by charlie....@gmail.com on 18 Feb 2011 at 8:30

@GoogleCodeExporter
Copy link
Author 

> If you are experiencing the problem, could you please post on that bug with
> the following details:
> - What version of mod_pagespeed you're running and what MPM

mod_pagespeed 0.9.15.3-404
Apache/2.2.16 (Ubuntu) mod_perl/2.0.4 Perl/v5.10.1
apache2-mpm-worker

> What Operating System you're running

Ubuntu 10.10

> Have you tried rebooting the machine? If so, does the issue persist?

Yes and yes.

> Did statistics ever work for you? If so, what was the last version of
> mod_pagespeed where they worked?

Only tried this version.

> Do all statistics fail, or just some of them?

All. Looks like this:

resource_fetches_cached: -1
resource_fetch_construct_successes: -1
resource_fetch_construct_failures: -1
total_page_load_ms: -1
page_load_count: -1
cache_extensions: -1
not_cacheable: -1
css_file_count_reduction: -1
css_elements: -1
image_inline: -1
image_rewrite_saved_bytes: -1
image_rewrites: -1
image_ongoing_rewrites: -1
javascript_blocks_minified: -1
javascript_bytes_saved: -1
javascript_minification_failures: -1
javascript_total_blocks: -1
resource_url_domain_rejections: -1
url_trims: -1
url_trim_saved_bytes: -1
css_filter_files_minified: -1
css_filter_minified_bytes_saved: -1
css_filter_parse_failures: -1
resource_404_count: -1
slurp_404_count: -1
cache_time_us: -1
cache_hits: -1
cache_misses: -1
cache_expirations: -1
cache_inserts: -1
serf_fetch_request_count: -1
serf_fetch_bytes_count: -1
serf_fetch_time_duration_ms: -1
serf_fetch_cancel_count: -1
serf_fetch_outstanding_count: -1
serf_fetch_timeout_count: -1
merge_time_us: -1
parse_time_us: -1
html_rewrite_time_us: -1

> What's the output of `ls -l /var/www/mod_pagespeed/files/statistics/ |
> head` (change the first to your ModPagespeedGeneratedFilePrefix)

$ ls -l /var/mod_pagespeed/files/statistics/ | head
total 0

That directory is empty. 

And my error log looks like this:

[Sat Feb 19 11:05:28 2011] [notice] Apache/2.2.16 (Ubuntu) mod_perl/2.0.4 
Perl/v5.10.1 configured -- resuming normal operations
[Sat Feb 19 11:05:36 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0219/110536:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
html_rewrite_time_us cannot lock mutex: Invalid argument 
[Sat Feb 19 11:05:36 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0219/110536:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_time_us cannot lock mutex: Invalid argument 
[Sat Feb 19 11:05:36 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0219/110536:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_hits cannot lock mutex: Invalid argument 
[Sat Feb 19 11:05:36 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0219/110536:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_time_us cannot lock mutex: Invalid argument 
[Sat Feb 19 11:05:36 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0219/110536:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_hits cannot lock mutex: Invalid argument 
[Sat Feb 19 11:05:36 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0219/110536:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_expirations cannot lock mutex: Invalid argument 
[Sat Feb 19 11:05:36 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0219/110536:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_time_us cannot lock mutex: Invalid argument 
[Sat Feb 19 11:05:36 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0219/110536:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_misses cannot lock mutex: Invalid argument 
[Sat Feb 19 11:05:36 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0219/110536:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_time_us cannot lock mutex: Invalid argument 
[Sat Feb 19 11:05:36 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0219/110536:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_inserts cannot lock mutex: Invalid argument

Original comment by maxba...@gmail.com on 19 Feb 2011 at 12:14

@GoogleCodeExporter
Copy link
Author 

Same error:
[Sat Feb 19 00:29:11 2011] [error] [mod_pagespeed 0.9.15.4-467] 
[0219/002911:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
html_rewrite_time_us cannot lock mutex: Permission denied 


[root@centos04 conf]# ls -l /opt/ramdisk/mod_pagespeed/files/statistics/ | head
total 39
-rw-r--r-- 1 root root 4 Feb 19 00:23 stats_value.cache_expirations
-rw-r--r-- 1 root root 4 Feb 19 00:23 stats_value.cache_extensions
-rw-r--r-- 1 root root 4 Feb 19 00:23 stats_value.cache_hits
-rw-r--r-- 1 root root 4 Feb 19 00:23 stats_value.cache_inserts
-rw-r--r-- 1 root root 4 Feb 19 00:23 stats_value.cache_misses
-rw-r--r-- 1 root root 4 Feb 19 00:23 stats_value.cache_time_us
-rw-r--r-- 1 root root 4 Feb 19 00:23 stats_value.css_elements
-rw-r--r-- 1 root root 4 Feb 19 00:23 stats_value.css_file_count_reduction
-rw-r--r-- 1 root root 4 Feb 19 00:23 stats_value.css_filter_files_minified

centos 5.5

[root@centos04 conf]# /opt/servers/apache/sbin/httpd-sc -V
Server version: Apache/2.2.17 (Unix)
Server built:   Feb 18 2011 02:21:17
Server's Module Magic Number: 20051115:25
Server loaded:  APR 1.4.2, APR-Util 1.3.10
Compiled using: APR 1.4.2, APR-Util 1.3.10
Architecture:   32-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/opt/servers/apache/"
 -D SUEXEC_BIN="/opt/servers/apache//bin/suexec"
 -D DEFAULT_PIDLOG="logs/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="logs/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd-sc.conf"


stats show:

resource_fetches_cached: 0
resource_fetch_construct_successes: 0
resource_fetch_construct_failures: 0
total_page_load_ms: 0
page_load_count: 0
cache_extensions: 0
not_cacheable: 0
css_file_count_reduction: 0
css_elements: 0
image_inline: 0
image_rewrite_saved_bytes: 0
image_rewrites: 0
image_ongoing_rewrites: 0
javascript_blocks_minified: 0
javascript_bytes_saved: 0
javascript_minification_failures: 0
javascript_total_blocks: 0
resource_url_domain_rejections: 0
url_trims: 0
url_trim_saved_bytes: 0
css_filter_files_minified: 0
css_filter_minified_bytes_saved: 0
css_filter_parse_failures: 0
resource_404_count: 0
slurp_404_count: 0
cache_time_us: 0
cache_hits: 0
cache_misses: 0
cache_expirations: 0
cache_inserts: 0
serf_fetch_request_count: 0
serf_fetch_bytes_count: 0
serf_fetch_time_duration_ms: 0
serf_fetch_cancel_count: 0
serf_fetch_outstanding_count: 0
serf_fetch_timeout_count: 0
merge_time_us: 0
parse_time_us: 69
html_rewrite_time_us: 0

Original comment by srpom...@gmail.com on 19 Feb 2011 at 12:32

@GoogleCodeExporter
Copy link
Author 

Adam...

i run mod_ruid in the server, removing mod_ruid i dont see the error!

but... i need mod_ruid :-(

Original comment by srpom...@gmail.com on 19 Feb 2011 at 2:53

@GoogleCodeExporter
Copy link
Author 

That's very good news: we've found the incompatibility.  In the short term, 
please try working around the issue by issuing the directive

   ModPagespeedStatistics off

We should now be able to reproduce the symptom, and hopefully find some way to 
resolve it while keeping statistics and mod_ruid working.  We are just starting 
a long weekend but I'm hopeful we'll be able to get to this before too long.

Original comment by jmara...@google.com on 19 Feb 2011 at 2:59

@GoogleCodeExporter
Copy link
Author

Original comment by jmara...@google.com on 19 Feb 2011 at 3:00

  • Changed title: Statistics failing with "cannot lock mutex: Invalid argument " when running wiht mod_ruid

@GoogleCodeExporter
Copy link
Author 

It would be useful to know if any of the people who've experienced the symptom 
are *not* running mod_ruid.

Original comment by jmara...@google.com on 19 Feb 2011 at 3:00

  • Changed title: Statistics failing with "cannot lock mutex: Invalid argument " when running with mod_ruid

@GoogleCodeExporter
Copy link
Author 

As asked, posting here log after apache restart. Unfortunately nothing new and 
interesting:

[Sat Feb 19 10:34:07 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0219/103407:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_time_us cannot lock mutex: Invalid argument 
[Sat Feb 19 10:34:07 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0219/103407:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_hits cannot lock mutex: Invalid argument 
[Sat Feb 19 10:34:07 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0219/103407:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_time_us cannot lock mutex: Invalid argument 
[Sat Feb 19 10:34:07 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0219/103407:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_hits cannot lock mutex: Invalid argument 
[Sat Feb 19 10:34:07 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0219/103407:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_time_us cannot lock mutex: Invalid argument 
[Sat Feb 19 10:34:07 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0219/103407:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_hits cannot lock mutex: Invalid argument 
[Sat Feb 19 10:34:07 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0219/103407:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_time_us cannot lock mutex: Invalid argument 
[Sat Feb 19 10:34:07 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0219/103407:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_hits cannot lock mutex: Invalid argument 
[Sat Feb 19 10:34:07 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0219/103407:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_time_us cannot lock mutex: Invalid argument 
[Sat Feb 19 10:34:07 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0219/103407:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_hits cannot lock mutex: Invalid argument 
[Sat Feb 19 10:34:07 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0219/103407:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_time_us cannot lock mutex: Invalid argument 
[Sat Feb 19 10:34:07 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0219/103407:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_hits cannot lock mutex: Invalid argument 
[Sat Feb 19 10:34:07 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0219/103407:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
cache_time_us cannot lock mutex: Invalid argument 


And mods I am using with apache:


/etc/apache2/mods-enabled# ls -la
total 8
drwxr-xr-x 2 root root 4096 2010-11-28 01:22 .
drwxr-xr-x 8 root root 4096 2011-02-08 11:31 ..
lrwxrwxrwx 1 root root   28 2010-11-24 17:14 alias.conf -> 
../mods-available/alias.conf
lrwxrwxrwx 1 root root   28 2010-11-24 17:14 alias.load -> 
../mods-available/alias.load
lrwxrwxrwx 1 root root   33 2010-11-24 17:14 auth_basic.load -> 
../mods-available/auth_basic.load
lrwxrwxrwx 1 root root   33 2010-11-24 17:14 authn_file.load -> 
../mods-available/authn_file.load
lrwxrwxrwx 1 root root   36 2010-11-24 17:14 authz_default.load -> 
../mods-available/authz_default.load
lrwxrwxrwx 1 root root   38 2010-11-24 17:14 authz_groupfile.load -> 
../mods-available/authz_groupfile.load
lrwxrwxrwx 1 root root   33 2010-11-24 17:14 authz_host.load -> 
../mods-available/authz_host.load
lrwxrwxrwx 1 root root   33 2010-11-24 17:14 authz_user.load -> 
../mods-available/authz_user.load
lrwxrwxrwx 1 root root   32 2010-11-24 17:14 autoindex.conf -> 
../mods-available/autoindex.conf
lrwxrwxrwx 1 root root   32 2010-11-24 17:14 autoindex.load -> 
../mods-available/autoindex.load
lrwxrwxrwx 1 root root   26 2010-11-24 17:15 cgi.load -> 
../mods-available/cgi.load
lrwxrwxrwx 1 root root   30 2010-11-24 17:14 deflate.conf -> 
../mods-available/deflate.conf
lrwxrwxrwx 1 root root   30 2010-11-24 17:14 deflate.load -> 
../mods-available/deflate.load
lrwxrwxrwx 1 root root   26 2010-11-24 17:14 dir.conf -> 
../mods-available/dir.conf
lrwxrwxrwx 1 root root   26 2010-11-24 17:14 dir.load -> 
../mods-available/dir.load
lrwxrwxrwx 1 root root   26 2010-11-24 17:14 env.load -> 
../mods-available/env.load
lrwxrwxrwx 1 root root   30 2010-11-28 01:22 headers.load -> 
../mods-available/headers.load
lrwxrwxrwx 1 root root   27 2010-11-24 17:14 mime.conf -> 
../mods-available/mime.conf
lrwxrwxrwx 1 root root   27 2010-11-24 17:14 mime.load -> 
../mods-available/mime.load
lrwxrwxrwx 1 root root   34 2010-11-24 17:14 negotiation.conf -> 
../mods-available/negotiation.conf
lrwxrwxrwx 1 root root   34 2010-11-24 17:14 negotiation.load -> 
../mods-available/negotiation.load
lrwxrwxrwx 1 root root   32 2010-11-27 23:46 pagespeed.conf -> 
../mods-available/pagespeed.conf
lrwxrwxrwx 1 root root   32 2010-11-27 23:46 pagespeed.load -> 
../mods-available/pagespeed.load
lrwxrwxrwx 1 root root   27 2010-11-24 17:15 php5.conf -> 
../mods-available/php5.conf
lrwxrwxrwx 1 root root   27 2010-11-24 17:15 php5.load -> 
../mods-available/php5.load
lrwxrwxrwx 1 root root   33 2010-11-24 17:14 reqtimeout.conf -> 
../mods-available/reqtimeout.conf
lrwxrwxrwx 1 root root   33 2010-11-24 17:14 reqtimeout.load -> 
../mods-available/reqtimeout.load
lrwxrwxrwx 1 root root   30 2010-11-24 17:19 rewrite.load -> 
../mods-available/rewrite.load
lrwxrwxrwx 1 root root   31 2010-11-24 17:14 setenvif.conf -> 
../mods-available/setenvif.conf
lrwxrwxrwx 1 root root   31 2010-11-24 17:14 setenvif.load -> 
../mods-available/setenvif.load
lrwxrwxrwx 1 root root   29 2010-11-24 17:14 status.conf -> 
../mods-available/status.conf
lrwxrwxrwx 1 root root   29 2010-11-24 17:14 status.load -> 
../mods-available/status.load

There's no ruid.

Original comment by justinas...@gtempaccount.com on 19 Feb 2011 at 8:42

@GoogleCodeExporter
Copy link
Author 

justi...@biworld.lt : I wonder if there's something else on your system related 
to file ownership or permissions.

The owner of the 'apache2' or 'httpd' process, sometimes 'www-user', should be 
the same as the owner of the pagespeed files/ directory.  Is that the case?

This might be another way, besides ruid, to wind up with permissions problems 
on the files needed to implement statistics.

Original comment by jmara...@google.com on 19 Feb 2011 at 2:38

@GoogleCodeExporter
Copy link
Author 

I noticed the other day that the owner of 'files' was root, unlike 'cache' 
which was www-user. I changed the owner and group of 'files' to www-user, 
rebooted, and no change in errors.

Original comment by charlie....@gmail.com on 19 Feb 2011 at 2:55

@GoogleCodeExporter
Copy link
Author 

- What version of mod_pagespeed you're running and what MPM
mod_pagespeed: 0.9.15.3-404

Server version: Apache/2.2.16 (Ubuntu)
Server built:   Nov 18 2010 21:17:43
Server's Module Magic Number: 20051115:24
Server loaded:  APR 1.4.2, APR-Util 1.3.9
Compiled using: APR 1.4.2, APR-Util 1.3.9
Architecture:   64-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/etc/apache2"
 -D SUEXEC_BIN="/usr/lib/apache2/suexec"
 -D DEFAULT_PIDLOG="/var/run/apache2.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="/var/run/apache2/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="mime.types"
 -D SERVER_CONFIG_FILE="apache2.conf"

- What Operating System you're running 
Ubunutu 10.10

- Have you tried rebooting the machine?  If so, does the issue persist? 
Yes

- Did statistics ever work for you?  If so, what was the last version of 
mod_pagespeed where they worked? 
No, this is the only version I've used.

- Do all statistics fail, or just some of them? 
All.. They all show -1

- What's the output of `ls -l /var/www/mod_pagespeed/files/statistics/ | 
head`  (change the first to your ModPagespeedGeneratedFilePrefix)

We have it in /var/mod_pagespeed/files/statistics/, but it's an empty folder 
and owned by www-data.

# ls -l /var/mod_pagespeed/files/statistics/ | head
total 0

- It would be useful to know if any of the people who've experienced the 
symptom are *not* running mod_ruid.

/etc/apache2/mods-enabled# ls -lh
total 0
lrwxrwxrwx 1 root root 28 2011-02-14 21:33 alias.conf -> 
../mods-available/alias.conf
lrwxrwxrwx 1 root root 28 2011-02-14 21:33 alias.load -> 
../mods-available/alias.load
lrwxrwxrwx 1 root root 33 2011-02-14 21:33 auth_basic.load -> 
../mods-available/auth_basic.load
lrwxrwxrwx 1 root root 33 2011-02-14 21:33 authn_file.load -> 
../mods-available/authn_file.load
lrwxrwxrwx 1 root root 36 2011-02-14 21:33 authz_default.load -> 
../mods-available/authz_default.load
lrwxrwxrwx 1 root root 38 2011-02-14 21:33 authz_groupfile.load -> 
../mods-available/authz_groupfile.load
lrwxrwxrwx 1 root root 33 2011-02-14 21:33 authz_host.load -> 
../mods-available/authz_host.load
lrwxrwxrwx 1 root root 33 2011-02-14 21:33 authz_user.load -> 
../mods-available/authz_user.load
lrwxrwxrwx 1 root root 32 2011-02-14 21:33 autoindex.conf -> 
../mods-available/autoindex.conf
lrwxrwxrwx 1 root root 32 2011-02-14 21:33 autoindex.load -> 
../mods-available/autoindex.load
lrwxrwxrwx 1 root root 36 2011-02-14 21:34 cgi.load -> 
/etc/apache2/mods-available/cgi.load
lrwxrwxrwx 1 root root 30 2011-02-14 21:33 deflate.conf -> 
../mods-available/deflate.conf
lrwxrwxrwx 1 root root 30 2011-02-14 21:33 deflate.load -> 
../mods-available/deflate.load
lrwxrwxrwx 1 root root 26 2011-02-14 21:33 dir.conf -> 
../mods-available/dir.conf
lrwxrwxrwx 1 root root 26 2011-02-14 21:33 dir.load -> 
../mods-available/dir.load
lrwxrwxrwx 1 root root 26 2011-02-14 21:33 env.load -> 
../mods-available/env.load
lrwxrwxrwx 1 root root 40 2011-02-14 22:20 expires.load -> 
/etc/apache2/mods-available/expires.load
lrwxrwxrwx 1 root root 40 2011-02-14 22:20 headers.load -> 
/etc/apache2/mods-available/headers.load
lrwxrwxrwx 1 root root 27 2011-02-14 21:33 mime.conf -> 
../mods-available/mime.conf
lrwxrwxrwx 1 root root 27 2011-02-14 21:33 mime.load -> 
../mods-available/mime.load
lrwxrwxrwx 1 root root 34 2011-02-14 21:33 negotiation.conf -> 
../mods-available/negotiation.conf
lrwxrwxrwx 1 root root 34 2011-02-14 21:33 negotiation.load -> 
../mods-available/negotiation.load
lrwxrwxrwx 1 root root 42 2011-02-14 21:34 pagespeed.conf -> 
/etc/apache2/mods-available/pagespeed.conf
lrwxrwxrwx 1 root root 42 2011-02-14 21:34 pagespeed.load -> 
/etc/apache2/mods-available/pagespeed.load
lrwxrwxrwx 1 root root 37 2011-02-14 21:34 php5.conf -> 
/etc/apache2/mods-available/php5.conf
lrwxrwxrwx 1 root root 37 2011-02-14 21:34 php5.load -> 
/etc/apache2/mods-available/php5.load
lrwxrwxrwx 1 root root 33 2011-02-14 21:33 reqtimeout.conf -> 
../mods-available/reqtimeout.conf
lrwxrwxrwx 1 root root 33 2011-02-14 21:33 reqtimeout.load -> 
../mods-available/reqtimeout.load
lrwxrwxrwx 1 root root 40 2011-02-14 21:38 rewrite.load -> 
/etc/apache2/mods-available/rewrite.load
lrwxrwxrwx 1 root root 31 2011-02-14 21:33 setenvif.conf -> 
../mods-available/setenvif.conf
lrwxrwxrwx 1 root root 31 2011-02-14 21:33 setenvif.load -> 
../mods-available/setenvif.load
lrwxrwxrwx 1 root root 29 2011-02-14 21:33 status.conf -> 
../mods-available/status.conf
lrwxrwxrwx 1 root root 29 2011-02-14 21:33 status.load -> 
../mods-available/status.load

Original comment by michael....@zumba.com on 21 Feb 2011 at 2:37

@GoogleCodeExporter
Copy link
Author 

Further to my info in Comment #4:
- I am not running mod_ruid
- Like Comment #12, I noticed that /var/mod_pagespeed/files/ was owned by root, 
not www-data, but changing ownership made no difference


$ ls -la /etc/apache2/mods-enabled
total 8
drwxr-xr-x 2 root root 4096 2011-02-21 08:32 .
drwxr-xr-x 9 root root 4096 2011-02-16 15:26 ..
lrwxrwxrwx 1 root root   28 2008-03-05 16:06 alias.conf -> 
../mods-available/alias.conf
lrwxrwxrwx 1 root root   28 2008-03-05 16:06 alias.load -> 
../mods-available/alias.load
lrwxrwxrwx 1 root root   33 2008-03-05 16:06 auth_basic.load -> 
../mods-available/auth_basic.load
lrwxrwxrwx 1 root root   33 2008-03-05 16:06 authn_file.load -> 
../mods-available/authn_file.load
lrwxrwxrwx 1 root root   36 2008-03-05 16:06 authz_default.load -> 
../mods-available/authz_default.load
lrwxrwxrwx 1 root root   38 2008-03-05 16:06 authz_groupfile.load -> 
../mods-available/authz_groupfile.load
lrwxrwxrwx 1 root root   33 2008-03-05 16:06 authz_host.load -> 
../mods-available/authz_host.load
lrwxrwxrwx 1 root root   33 2008-03-05 16:06 authz_user.load -> 
../mods-available/authz_user.load
lrwxrwxrwx 1 root root   32 2008-03-05 16:06 autoindex.conf -> 
../mods-available/autoindex.conf
lrwxrwxrwx 1 root root   32 2008-03-05 16:06 autoindex.load -> 
../mods-available/autoindex.load
lrwxrwxrwx 1 root root   27 2011-01-11 16:09 cgid.conf -> 
../mods-available/cgid.conf
lrwxrwxrwx 1 root root   27 2011-01-11 16:09 cgid.load -> 
../mods-available/cgid.load
lrwxrwxrwx 1 root root   26 2009-03-22 12:27 cgi.load -> 
../mods-available/cgi.load
lrwxrwxrwx 1 root root   30 2010-05-07 09:01 deflate.conf -> 
../mods-available/deflate.conf
lrwxrwxrwx 1 root root   30 2010-05-07 09:01 deflate.load -> 
../mods-available/deflate.load
lrwxrwxrwx 1 root root   26 2008-03-05 16:06 dir.conf -> 
../mods-available/dir.conf
lrwxrwxrwx 1 root root   26 2008-03-05 16:06 dir.load -> 
../mods-available/dir.load
lrwxrwxrwx 1 root root   26 2008-03-05 16:06 env.load -> 
../mods-available/env.load
lrwxrwxrwx 1 root root   30 2009-06-25 12:10 expires.load -> 
../mods-available/expires.load
lrwxrwxrwx 1 root root   27 2008-03-05 16:06 mime.conf -> 
../mods-available/mime.conf
lrwxrwxrwx 1 root root   27 2008-03-05 16:06 mime.load -> 
../mods-available/mime.load
lrwxrwxrwx 1 root root   34 2008-03-05 16:06 negotiation.conf -> 
../mods-available/negotiation.conf
lrwxrwxrwx 1 root root   34 2008-03-05 16:06 negotiation.load -> 
../mods-available/negotiation.load
lrwxrwxrwx 1 root root   27 2009-02-23 10:58 perl.load -> 
../mods-available/perl.load
lrwxrwxrwx 1 root root   28 2009-02-12 13:49 proxy.conf -> 
../mods-available/proxy.conf
lrwxrwxrwx 1 root root   33 2009-02-12 13:53 proxy_http.load -> 
../mods-available/proxy_http.load
lrwxrwxrwx 1 root root   28 2009-02-12 13:49 proxy.load -> 
../mods-available/proxy.load
lrwxrwxrwx 1 root root   33 2010-05-02 12:38 reqtimeout.conf -> 
../mods-available/reqtimeout.conf
lrwxrwxrwx 1 root root   33 2010-05-02 12:38 reqtimeout.load -> 
../mods-available/reqtimeout.load
lrwxrwxrwx 1 root root   30 2008-03-05 16:25 rewrite.load -> 
../mods-available/rewrite.load
lrwxrwxrwx 1 root root   27 2010-10-24 10:35 rpaf.conf -> 
../mods-available/rpaf.conf
lrwxrwxrwx 1 root root   27 2010-10-24 10:35 rpaf.load -> 
../mods-available/rpaf.load
lrwxrwxrwx 1 root root   31 2008-03-05 16:06 setenvif.conf -> 
../mods-available/setenvif.conf
lrwxrwxrwx 1 root root   31 2008-03-05 16:06 setenvif.load -> 
../mods-available/setenvif.load
lrwxrwxrwx 1 root root   29 2008-03-05 16:06 status.conf -> 
../mods-available/status.conf
lrwxrwxrwx 1 root root   29 2008-03-05 16:06 status.load -> 
../mods-available/status.load

Original comment by maxba...@gmail.com on 21 Feb 2011 at 9:32

@GoogleCodeExporter
Copy link
Author 

I just tried disabling every mod possible, in search of a conflict. After 
getting down to this state:

$ ls -la /etc/apache2/mods-enabled/
total 8
drwxr-xr-x 2 root root 4096 2011-02-22 08:49 .
drwxr-xr-x 9 root root 4096 2011-02-22 08:37 ..
lrwxrwxrwx 1 root root   33 2011-02-22 08:48 authz_host.load -> 
../mods-available/authz_host.load
lrwxrwxrwx 1 root root   27 2011-02-22 08:44 mime.conf -> 
../mods-available/mime.conf
lrwxrwxrwx 1 root root   27 2011-02-22 08:44 mime.load -> 
../mods-available/mime.load
lrwxrwxrwx 1 root root   32 2011-02-22 08:33 pagespeed.conf -> 
../mods-available/pagespeed.conf
lrwxrwxrwx 1 root root   32 2011-02-22 08:33 pagespeed.load -> 
../mods-available/pagespeed.load

... I still got this:

[Tue Feb 22 08:49:03 2011] [notice] Apache/2.2.16 (Ubuntu) configured -- 
resuming normal operations
[Tue Feb 22 08:49:05 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0222/084905:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
html_rewrite_time_us cannot lock mutex: Invalid argument 
[Tue Feb 22 08:49:05 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0222/084905:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
total_page_load_ms cannot lock mutex: Invalid argument 
[Tue Feb 22 08:49:05 2011] [error] [mod_pagespeed 0.9.15.3-404] 
[0222/084905:ERROR:net/instaweb/apache/apr_statistics.cc(141)] Variable 
page_load_count cannot lock mutex: Invalid argument 

And all statistics were "-1", as before.

$ apache2 -V
Server version: Apache/2.2.16 (Ubuntu)
Server built:   Nov 18 2010 21:18:34
Server's Module Magic Number: 20051115:24
Server loaded:  APR 1.4.2, APR-Util 1.3.9
Compiled using: APR 1.4.2, APR-Util 1.3.9
Architecture:   64-bit
Server MPM:     Worker
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/worker"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/etc/apache2"
 -D SUEXEC_BIN="/usr/lib/apache2/suexec"
 -D DEFAULT_PIDLOG="/var/run/apache2.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="mime.types"
 -D SERVER_CONFIG_FILE="apache2.conf"

Original comment by maxba...@gmail.com on 21 Feb 2011 at 9:51

@GoogleCodeExporter
Copy link
Author 

Thanks for all the info, folks.  Note that comment #5 is a separate issue.  
That error, "permission denied", with stats_value.* files present but 
unreadable, is likely related to mod_ruid. Everyone else is getting "invalid 
argument" and the stats files are not created in the first place.

Original comment by abl...@google.com on 22 Feb 2011 at 3:04

  • Changed title: Statistics failing with "cannot lock mutex: Invalid argument "

@GoogleCodeExporter
Copy link
Author 

Issue 278 has been merged into this issue.

Original comment by abl...@google.com on 27 Apr 2011 at 3:37

@GoogleCodeExporter
Copy link
Author 

follow up to my merged Issue278 ...
not sure if same/related issue:

checking my SYSLOG, rather than just apache log,

ENable mod_pagespeed
service apache2 restart
@ syslog
    Apr 27 08:55:08 test kernel: [150213.719044] httpd2-prefork[18592]: segfault at b74e5ff0 ip b74e5ff0 sp bf977dcc error 14

DISable mod_pagespeed
service apache2 restart
@ syslog
    (no error)

fully reproducible here across 3 machines, apache restarts, and system restarts

Original comment by pgnet.dev on 27 Apr 2011 at 4:01

@GoogleCodeExporter
Copy link
Author 

This was fixed in a prior release (0.9.17.* definitely has the fix) by moving 
to shared-memory-based statistics.

Original comment by jmara...@google.com on 14 Jul 2011 at 6:21

  • Changed state: Fixed

@GoogleCodeExporter GoogleCodeExporter mentioned this issue Apr 6, 2015
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
auto-migrated Priority-Medium Type-Defect
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@GoogleCodeExporter