New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DCheck failure if pagespeed is off and no file cache path is configured #956

Closed
lbensaad opened this Issue Apr 20, 2015 · 18 comments

Comments

Projects
None yet
6 participants
@lbensaad
Copy link

lbensaad commented Apr 20, 2015

I have followed the steps here:
https://developers.google.com/speed/pagespeed/module/downstream-caching
To configure proxy caching with pagespeed, but I found that when adding "pagespeed off" in the proxy_cache server configuration file then the cpu usage will be 100% and the site does not load, and when i remove it, it works fine.
The configuration file is here: https://gist.github.com/bml13/505e5bf840a9d09143d3

What could be wrong?

@oschaaf

This comment has been minimized.

Copy link
Member

oschaaf commented Apr 20, 2015

@bml13 I tried to reproduce based on your configuration with static content instead of wordpress, no luck yet. I'll try some more, but meantime I have a few questions:

  • Could you post more details of your system, like the OS and pagespeed version?
  • Is there anything noteworthy logged to nginx's error.log when nginx is in a state where it is taking 100% cpu (or before entering that state?).
  • Does the problem show up immediately after starting nginx, or after you request one or more pages?
@lbensaad

This comment has been minimized.

Copy link

lbensaad commented Apr 20, 2015

Sorry i forget to post them.
I am using Ubuntu 14.04.2 LTS, nginx 1.6.3, ngx_pagespeed 1.9.32.3-4448
The problem is that when I added "pagespeed off" and started nginx, then the nginx error log file gets full quickly with these lines that are repeated hundreds of times right after the start of nginx and even before loading any page, and since the cpu is 100% busy then no page will load at all:

2015/04/20 11:02:09 [alert] 11704#0: [ngx_pagespeed 1.9.32.3-4448] [0420/110209:FATAL:system_cache_path.cc(209)] Check failed: path.starts_with("/").
nginx: master process /usr/sbin/nginx() [0x51de19]
nginx: master process /usr/sbin/nginx() [0x51ae32]
nginx: master process /usr/sbin/nginx() [0xa0ab39]
nginx: master process /usr/sbin/nginx() [0x661983]
nginx: master process /usr/sbin/nginx() [0x66b678]
nginx: master process /usr/sbin/nginx() [0x4da6fd]
nginx: master process /usr/sbin/nginx() [0x4452a6]
nginx: master process /usr/sbin/nginx() [0x446062]
nginx: master process /usr/sbin/nginx(ngx_spawn_process+0x170) [0x444310]
nginx: master process /usr/sbin/nginx() [0x445c7d]
nginx: master process /usr/sbin/nginx(ngx_master_process_cycle+0x1e6) [0x446846]
nginx: master process /usr/sbin/nginx(main+0x77f) [0x424e4f]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f7bedd3eec5]
nginx: master process /usr/sbin/nginx() [0x4253cf]

nginx: master process /usr/sbin/nginx() [0x51de19]
nginx: master process /usr/sbin/nginx() [0x4d51f0]
nginx: master process /usr/sbin/nginx() [0x51af14]
nginx: master process /usr/sbin/nginx() [0xa0ab39]
nginx: master process /usr/sbin/nginx() [0x661983]
nginx: master process /usr/sbin/nginx() [0x66b678]
nginx: master process /usr/sbin/nginx() [0x4da6fd]
nginx: master process /usr/sbin/nginx() [0x4452a6]
nginx: master process /usr/sbin/nginx() [0x446062]
nginx: master process /usr/sbin/nginx(ngx_spawn_process+0x170) [0x444310]
nginx: master process /usr/sbin/nginx() [0x445c7d]
nginx: master process /usr/sbin/nginx(ngx_master_process_cycle+0x1e6) [0x446846]
nginx: master process /usr/sbin/nginx(main+0x77f) [0x424e4f]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f7bedd3eec5]
nginx: master process /usr/sbin/nginx() [0x4253cf]

@oschaaf

This comment has been minimized.

Copy link
Member

oschaaf commented Apr 20, 2015

Great thanks. I think I know what happens.
You should be able to disable pagespeed without having a valid FileCachePath configured,
but currently a check fails.

I suspect a workaround to this is moving this line into the http{} block:

pagespeed FileCachePath /var/run/ngx_pagespeed_cache;
@lbensaad

This comment has been minimized.

Copy link

lbensaad commented Apr 20, 2015

Brilliant, It is solved.
Many thanks

@lbensaad lbensaad closed this Apr 20, 2015

@jmarantz

This comment has been minimized.

Copy link
Contributor

jmarantz commented Apr 20, 2015

Let's leave this open for now. The corresponding error in Apache I think would prevent Apache from starting up and give clean error message to stdout, rather than spinning the CPU and logging hex stack traces to the error log.

We should get the nginx failure-mode here to be a little easier to diagnose.

@jmarantz jmarantz reopened this Apr 20, 2015

@jeffkaufman

This comment has been minimized.

Copy link
Contributor

jeffkaufman commented May 4, 2015

@crowell is looking at this

@crowell crowell self-assigned this May 4, 2015

@crowell

This comment has been minimized.

Copy link
Contributor

crowell commented May 5, 2015

currently testing a fix that will go into mod_pagespeed, I'll reference this bug when it goes in.

crowell added a commit to apache/incubator-pagespeed-mod that referenced this issue May 6, 2015

@crowell

This comment has been minimized.

Copy link
Contributor

crowell commented May 6, 2015

fixed in mod_pagespeed commit apache/incubator-pagespeed-mod@b4a2ed8

@crowell crowell closed this May 6, 2015

crowell added a commit to apache/incubator-pagespeed-mod that referenced this issue May 11, 2015

@mital

This comment has been minimized.

Copy link

mital commented May 12, 2015

how can I fix this in my ngx_pagespeed ? I am using master branch and 1.9.32.3.tar.gz psol

@oschaaf

This comment has been minimized.

Copy link
Member

oschaaf commented May 12, 2015

@mital

This comment has been minimized.

Copy link

mital commented May 12, 2015

@oschaaf thanks, I have tried that .. the cpu usage is still high and the nginx actually stops working (does not serve any page)

I am on centos 7 and compiling nginx 1.8.0-1 with pagespeed. Which version / os you have tried this on and working ?

@oschaaf

This comment has been minimized.

Copy link
Member

oschaaf commented May 12, 2015

@mital Are there any messages in error.log when the cpu is high and/or nginx stops working?
My system runs ubuntu 14.04 + nginx 1.7.9

@mital

This comment has been minimized.

Copy link

mital commented May 12, 2015

Yes Following are the errors I am getting

[alert] 3037#0: [ngx_pagespeed 1.9.32.3-4448] [0512/044807:FATAL:ngx_rewrite_driver_factory.cc(208)] Check failed: ok. Unable to start scheduler thread
nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf() [0x4d541a]
nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf() [0x4a05ed]
nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf() [0x4d2332]
nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf() [0x4ae5bb]
nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf() [0x4a5d7b]
nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf() [0x42d276]
nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf() [0x42dd85]
nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf() [0x42c284]
nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf() [0x42ea42]
nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf() [0x40e77f]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f21f1a80af5]
nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf() [0x40ed81]
@oschaaf

This comment has been minimized.

Copy link
Member

oschaaf commented May 12, 2015

@mital could you be hitting this? #886 (comment)

@mital

This comment has been minimized.

Copy link

mital commented May 12, 2015

@oschaaf oh this is selinux issue.
Following is the selinux policy I have to modify

module nginx_pagespeed 1.0;

require {
        type httpd_t;
        class process execmem;
}

#============= httpd_t ==============

#!!!! This avc can be allowed using the boolean 'httpd_execmem'
allow httpd_t self:process execmem;

thanks for pointing this to me.. this solved my issue !

@jeffkaufman jeffkaufman changed the title pagespeed off causes 100% cpu usage DCheck failure if pagespeed is off and no file cache path is configured Jul 27, 2015

@crowell

This comment has been minimized.

Copy link
Contributor

crowell commented Jul 28, 2015

is there a reason why this bug was reopened? the fix is in apache/incubator-pagespeed-mod@b4a2ed8 which will be included in the next release.

@jeffkaufman

This comment has been minimized.

Copy link
Contributor

jeffkaufman commented Jul 28, 2015

The bug looks closed to me?

@crowell

This comment has been minimized.

Copy link
Contributor

crowell commented Jul 28, 2015

ah, just saw it was a "referenced issue" that was open still, my mistake.

@pono pono unassigned crowell Jan 8, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment