Skip to content

Loading…

Apache hits "MaxClients" limit, stops serving pages; seen on Arborwiki #355

Closed
vielmetti opened this Issue · 46 comments

4 participants

@vielmetti

On Arborwiki -

Apache is getting stuck from time to time; I've seen it twice. The presenting symptom is a "MaxClients exceeded" error message in the logs, and the server then refuses to respond to more requests.

@vielmetti

Noted by Ryan Eby:

Well from the looks of it you are hitting the MaxClients which may be the localwiki not responding quick enough and keeping processes open or just being crawled too quickly. Or it could be the script erroring out and the process not closing well which seems possible with the number of entries in the apache error log.

I noticed the robots.txt doesn't have a crawl delay. That might be something to look into if things seem scraping heavy.

But more troublesome is the number of errors in the log which may be a red herring or something with the localwiki setup. Lots of:

Premature end of script headers: localwiki.wsgi

The ones I glanced up by IP are from BingBot among others. There are also quite a few backtraces in the error log. Some looking for a 500.html template which means there was probably a server error involved. So the first step is probably creating that template to keep the logs less noisy for more debugging.

Does localwiki keep a log of internal errors somewhere or just the stack trace in the apache log?

@mivanov
LocalWiki.org member

LocalWiki doesn't keep a log other than the apache log, but one can be set up using Sentry. The way it works is all application errors get forwarded to a central logging server (also a Django app and can actually be on the same server) where you can login and see detailed error reports. See http://sentry.readthedocs.org/ or let me know if you'd like me to set it up for you. It would be good to get to the bottom of this. I have also added a 500.html template so when you deploy the next version of localwiki you don't get so much noise in the apache logs.

@philipn
LocalWiki.org member

Also lots of

[Mon Oct 08 08:20:42 2012] [error] [client 157.55.32.75] Script timed out before returning headers: localwiki.wsgi`

Never seen this elsewhere. Biggest difference between ArborWiki and other big sites is the lack of caching, so we may want to look at that at some point.

MaxClients is ~150, so that's a lot of hanging requests if that's what's going on. @mivanov would be good to have edward just point arborwiki @ our sentry server, right?

@mivanov
LocalWiki.org member

Yeah we can just point it to our sentry server.

@vielmetti

The 157.55.32.75 address is an msnbot address; I wonder if the msnbot crawler is either being way too aggressive, or if its fetching is weird in some other way.

@mivanov
LocalWiki.org member

A day later, no entries in the error logs other than one I triggered to test the logging is working. I have enabled detailed apache status reporting, and everything appears pretty normal so far. Please notify me as soon as this comes up again.

@vielmetti

Every once in a while, to the tune of a half dozen times a day, I'm seeing this in the logs complaining about no 500.html file:

[Thu Oct 11 14:23:02 2012] [error] [client 198.208.159.17] return callback(request, *param_dict)
[Thu Oct 11 14:23:02 2012] [error] [client 198.208.159.17] File "/usr/share/localwiki/env/lib/python2.7/site-packages/django/utils/decorators.py", line 93, in _wrapped_view
[Thu Oct 11 14:23:02 2012] [error] [client 198.208.159.17] response = view_func(request, *args, *
kwargs)
[Thu Oct 11 14:23:02 2012] [error] [client 198.208.159.17] File "/usr/share/localwiki/env/lib/python2.7/site-packages/django/views/defaults.py", line 30, in server_error
[Thu Oct 11 14:23:02 2012] [error] [client 198.208.159.17] t = loader.get_template(template_name) # You need to create a 500.html template.
....

I restarted the apache server, and also saw this in the beginning of the error log; I don't know where the "example.com" address is coming from.

[Thu Oct 11 14:33:21 2012] [notice] SIGUSR1 received. Doing graceful restart
[Thu Oct 11 14:33:21 2012] [error] Exception AttributeError: "'NoneType' object has no attribute 'destroy_geom'" in Exception AttributeError: "'NoneType' object has no attribute 'destroy_geom'" in
Error opening file for reading: Permission denied
[Thu Oct 11 14:33:21 2012] [error] Exception AttributeError: "'NoneType' object has no attribute 'destroy_geom'" in
Error opening file for reading: Permission denied
[Thu Oct 11 14:33:21 2012] [warn] mod_wsgi: Compiled for Python/2.7.2+.
[Thu Oct 11 14:33:21 2012] [warn] mod_wsgi: Runtime using Python/2.7.3.
[Thu Oct 11 14:33:21 2012] [notice] Apache/2.2.22 (Ubuntu) mod_wsgi/3.3 Python/2.7.3 configured -- resuming normal operations
[Thu Oct 11 14:33:25 2012] [error] client 68.102.159.151No such file or directory: mod_wsgi (pid=25710): Unable to connect to WSGI daemon process 'example.com' on '/var/run/apache2/wsgi.32714.11.1.sock' after multiple attempts.
[Thu Oct 11 14:33:26 2012] [error] client 68.102.159.151No such file or directory: mod_wsgi (pid=30405): Unable to connect to WSGI daemon process 'example.com' on '/var/run/apache2/wsgi.32714.11.1.sock' after multiple attempts.

@mivanov
LocalWiki.org member

example.com is just the default name of the Apache virtual host config file we install, found at /etc/apache2/sites-enabled/example.com

I saw the error that was causing the "you need to create a 500.html template" message and fixed it (including on your server) and installed a 500.html template so you won't see that in the logs anymore.

@vielmetti

Thanks Mike.

I see these two lines in the Apache config:

WSGIDaemonProcess example.com processes=2 threads=15 display-name=%{GROUP}
WSGIProcessGroup example.com

but when I look at the other example doc on

http://code.google.com/p/modwsgi/wiki/ConfigurationDirectives#WSGIDaemonProcess

it suggests that the "name" listed here should probably be arborwiki.org and not example.com. (or at least that's how I read it)

@mivanov
LocalWiki.org member

You could change that or leave it. This is just the name of the process group. If you change it, run
/etc/init.d/apache2 reload

@vielmetti

The system froze up again a few minutes ago. The last part of the log files look like this:

[Sat Oct 13 21:27:47 2012] [error] [client 157.55.33.98] Premature end of script headers: localwiki.wsgi
[Sat Oct 13 21:27:47 2012] [error] [client 157.55.34.28] Premature end of script headers: localwiki.wsgi
[Sat Oct 13 21:27:47 2012] [error] [client 157.55.34.28] Premature end of script headers: localwiki.wsgi
[Sat Oct 13 21:27:47 2012] [error] [client 157.55.34.28] Premature end of script headers: localwiki.wsgi
[Sat Oct 13 21:27:47 2012] [error] [client 68.42.74.233] Premature end of script headers: localwiki.wsgi, referer: http://www.google.com/url?sa=t&rct=j&q=stadium%20hardware&source=web&cd=3&ved=0CDgQFjAC&url=http%3A%2F%2Farborwiki.org%2FStadium_Hardware&ei=v3h5UPnoL-XzyAHhsYDwCw&usg=AFQjCNHqpu9SAccPQjt0XU0WwG1MCLAsyg
[Sat Oct 13 21:27:47 2012] [error] [client 157.55.34.28] Premature end of script headers: localwiki.wsgi
[Sat Oct 13 21:27:47 2012] [error] [client 157.55.34.28] Premature end of script headers: localwiki.wsgi

the 157.55 address block are MSNbot addresses for what that's worth; if we're getting too aggressively crawled by that site there should be some way to slow down the crawl rate.

@vielmetti

I added the lines

User-agent: msnbot
Crawl-delay: 1

to the robots.txt file, in the hopes of slowing down msnbot when it fires off lots of page views at once.

@vielmetti

I did the same for "bingbot", and their crawl rate has gone down to a steady one per second rather than launching 20-30 page fetches in the same second. I think this will solve the problem, at least for this ill-behaved crawler.

@mivanov
LocalWiki.org member

Unfortunately the errors you were seeing on Saturday did not register on the logging server. Which is kinda what I expected because the requests were not completed. I couldn't find anything helpful in your server logs either. It sounds like the crawl delay setting is helping for now, so that's great news. I'll keep reading up and see if there is a good solution for preventing this in the future.

@vielmetti

Closing this issue for now - if it reappears I'll reopen, but I'm convinced that I know the origin and solution for the problem.

@vielmetti vielmetti closed this
@vielmetti

Well, it ran for 17 days but then got stuck again. I did a graceful restart of apache and all is well again.

Mike, can you see if there's anything visibile on the logging server that might help?

@philipn
LocalWiki.org member

I'm seeing quite a few errors with the feeds, which may be causing it. It might also just be a lot of long-time-to-generate feed URLs that are sitting around.

Maybe we should log using this: https://github.com/disqus/django-perftools for slow requests. I don't have time to investigate right now, unfortunately. But it should give us an idea and is probably good to set up in general

@vielmetti

We had another freeze, and subsequently added this line to the Apache config. Notes:

From: Cory Kaufman-Schofield cory@corykaufman.com
Date: Thu, 15 Nov 2012 10:46:31 -0500

Googled the issue and ended up here:

https://groups.google.com/forum/?fromgroups=#!topic/modwsgi/_Ws2XJ-nlq0

I added this to /etc/apache2/mods-available/wsgi.conf and restarted apache:

WSGISocketPrefix /var/run/apache2/wsgi

Hopefully that fixes the "premature end of script headers" error-- let me know if you're still having problems and I'll research it more.

@philipn
LocalWiki.org member

Managed to catch arborwiki.org right as it was doing this.

top - 20:59:38 up 51 days,  8:39,  1 user,  load average: 0.00, 0.01, 0.05
Tasks: 104 total,   1 running, 103 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.1%us,  0.0%sy,  0.0%ni, 99.8%id,  0.2%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   6114036k total,  3534740k used,  2579296k free,   259388k buffers
Swap:  6288380k total,        0k used,  6288380k free,  1473404k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                           
   35 root      20   0     0    0    0 S    0  0.0  15:19.21 kworker/3:1                                                                       
 3318 root      20   0 17332 1292  960 R    0  0.0   0:00.02 top                                                                               
    1 root      20   0 24328 2228 1344 S    0  0.0   0:06.97 init                                                                              
    2 root      20   0     0    0    0 S    0  0.0   0:00.48 kthreadd  
root@arborwiki:/etc/apache2# ps auxww|grep apache
www-data  2583  0.0  0.1 1698636 8732 ?        Sl   19:40   0:00 /usr/sbin/apache2 -k start
www-data  2659  0.0  0.1 1108644 7964 ?        Sl   19:50   0:00 /usr/sbin/apache2 -k start
www-data  2721  0.0  0.1 952744  7476 ?        Sl   19:58   0:00 /usr/sbin/apache2 -k start
root      3326  0.0  0.0   8096   936 pts/0    S+   21:00   0:00 grep --color=auto apache
root     11812  0.0  0.0  97272  5236 ?        Ss   Nov15   0:24 /usr/sbin/apache2 -k start
www-data 13423  0.0  0.0  97288  2824 ?        S    Nov18   0:00 /usr/sbin/apache2 -k start
www-data 13426  0.0  0.4 2045876 27660 ?       Sl   Nov18   1:55 /usr/sbin/apache2 -k start
www-data 13427  0.0  0.4 2044108 27448 ?       Sl   Nov18   1:55 /usr/sbin/apache2 -k start
www-data 17420  0.0  0.3 2032528 18536 ?       Sl   Nov18   1:39 /usr/sbin/apache2 -k start
www-data 26523  0.0  0.2 2028236 14180 ?       Sl   11:10   0:09 /usr/sbin/apache2 -k start
root@arborwiki:/etc/apache2# ps auxww|grep wsgi
root      3336  0.0  0.0   8096   936 pts/0    S+   21:00   0:00 grep --color=auto wsgi
www-data 13424  0.8 11.5 1829448 703908 ?      Sl   Nov18  67:12 (wsgi:example.com -k start
www-data 13425  0.6  9.6 1632856 592224 ?      Sl   Nov18  55:09 (wsgi:example.com -k start

which all looks..normal.

[Tue Nov 20 21:28:51 2012] [error] [client 91.201.64.134]   File "/usr/share/localwiki/env/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 264, in __call__
[Tue Nov 20 21:28:51 2012] [error] [client 91.201.64.134]     logger.warning('Bad Request (UnicodeDecodeError): %s' % request.path,
[Tue Nov 20 21:28:51 2012] [error] [client 91.201.64.134] UnboundLocalError: local variable 'request' referenced before assignment
[Tue Nov 20 21:29:00 2012] [error] [client 91.201.64.134] mod_wsgi (pid=13425): Exception occurred processing WSGI script '/usr/share/localwiki/deploy/localwiki.wsgi'.
[Tue Nov 20 21:29:00 2012] [error] [client 91.201.64.134] Traceback (most recent call last):
[Tue Nov 20 21:29:00 2012] [error] [client 91.201.64.134]   File "/usr/share/localwiki/env/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 264, in __call__
[Tue Nov 20 21:29:00 2012] [error] [client 91.201.64.134]     logger.warning('Bad Request (UnicodeDecodeError): %s' % request.path,
[Tue Nov 20 21:29:00 2012] [error] [client 91.201.64.134] UnboundLocalError: local variable 'request' referenced before assignment
[Wed Nov 21 09:06:50 2012] [error] [client 108.211.200.241] mod_wsgi (pid=13425): Exception occurred processing WSGI script '/usr/share/localwiki/deploy/localwiki.wsgi'.
[Wed Nov 21 09:06:50 2012] [error] [client 108.211.200.241] IOError: failed to write data
[Wed Nov 21 16:25:28 2012] [error] [client 12.31.102.2] mod_wsgi (pid=13424): Exception occurred processing WSGI script '/usr/share/localwiki/deploy/localwiki.wsgi'.
[Wed Nov 21 16:25:28 2012] [error] [client 12.31.102.2] IOError: failed to write data
[Wed Nov 21 16:25:31 2012] [error] [client 12.31.102.2] mod_wsgi (pid=13425): Exception occurred processing WSGI script '/usr/share/localwiki/deploy/localwiki.wsgi'.
[Wed Nov 21 16:25:31 2012] [error] [client 12.31.102.2] IOError: failed to write data
[Fri Nov 23 03:22:45 2012] [error] [client 222.214.216.65] mod_wsgi (pid=13424): Exception occurred processing WSGI script '/usr/share/localwiki/deploy/localwiki.wsgi'.
[Fri Nov 23 03:22:45 2012] [error] [client 222.214.216.65] Traceback (most recent call last):
[Fri Nov 23 03:22:45 2012] [error] [client 222.214.216.65]   File "/usr/share/localwiki/env/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 264, in __call__
[Fri Nov 23 03:22:45 2012] [error] [client 222.214.216.65]     logger.warning('Bad Request (UnicodeDecodeError): %s' % request.path,
[Fri Nov 23 03:22:45 2012] [error] [client 222.214.216.65] UnboundLocalError: local variable 'request' referenced before assignment
[Fri Nov 23 03:22:45 2012] [error] [client 222.214.216.65] mod_wsgi (pid=13424): Exception occurred processing WSGI script '/usr/share/localwiki/deploy/localwiki.wsgi'.
[Fri Nov 23 03:22:45 2012] [error] [client 222.214.216.65] Traceback (most recent call last):
[Fri Nov 23 03:22:45 2012] [error] [client 222.214.216.65]   File "/usr/share/localwiki/env/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 264, in __call__
[Fri Nov 23 03:22:45 2012] [error] [client 222.214.216.65]     logger.warning('Bad Request (UnicodeDecodeError): %s' % request.path,
[Fri Nov 23 03:22:45 2012] [error] [client 222.214.216.65] UnboundLocalError: local variable 'request' referenced before assignment
[Fri Nov 23 10:46:08 2012] [error] [client 50.124.250.197] mod_wsgi (pid=13424): Exception occurred processing WSGI script '/usr/share/localwiki/deploy/localwiki.wsgi'.
[Fri Nov 23 10:46:08 2012] [error] [client 50.124.250.197] IOError: failed to write data
[Fri Nov 23 10:46:16 2012] [error] [client 50.124.250.197] mod_wsgi (pid=13424): Exception occurred processing WSGI script '/usr/share/localwiki/deploy/localwiki.wsgi'.
[Fri Nov 23 10:46:16 2012] [error] [client 50.124.250.197] IOError: failed to write data
[Fri Nov 23 19:41:43 2012] [error] [client 157.55.32.166] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:42:01 2012] [error] [client 110.85.36.46] Script timed out before returning headers: localwiki.wsgi, referer: http://arborwiki.org/Users/login/?next=/
[Fri Nov 23 19:42:01 2012] [error] [client 110.85.36.46] Script timed out before returning headers: localwiki.wsgi, referer: http://arborwiki.org/Users/login/?next=/
[Fri Nov 23 19:42:03 2012] [error] [client 98.243.173.134] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:42:06 2012] [error] [client 218.30.103.146] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:42:09 2012] [error] [client 157.55.32.166] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:42:23 2012] [error] [client 66.249.73.231] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:42:37 2012] [error] [client 95.81.248.41] Script timed out before returning headers: localwiki.wsgi, referer: http://arborwiki.org/Main_Page
[Fri Nov 23 19:42:41 2012] [error] [client 157.55.32.166] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:42:50 2012] [error] [client 173.199.115.91] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:42:51 2012] [error] [client 110.85.36.46] Script timed out before returning headers: localwiki.wsgi, referer: http://arborwiki.org/Users/login/?next=/
[Fri Nov 23 19:43:01 2012] [error] [client 98.243.173.239] Script timed out before returning headers: localwiki.wsgi, referer: http://devnet.aadl.org/city/Village_4_Theaters
[Fri Nov 23 19:43:09 2012] [error] [client 157.55.32.166] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:43:27 2012] [error] [client 95.81.248.41] Script timed out before returning headers: localwiki.wsgi, referer: http://arborwiki.org/
[Fri Nov 23 19:43:33 2012] [error] [client 65.52.111.250] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:43:37 2012] [error] [client 157.55.35.98] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:43:41 2012] [error] [client 110.85.36.46] Script timed out before returning headers: localwiki.wsgi, referer: http://arborwiki.org/Users/login/?next=/List_of_Ann_Arbor_forums
[Fri Nov 23 19:44:05 2012] [error] [client 62.31.187.155] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:44:08 2012] [error] [client 218.30.103.146] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:44:15 2012] [error] [client 173.199.115.43] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:44:27 2012] [error] [client 180.76.5.190] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:44:29 2012] [error] [client 66.87.114.143] Script timed out before returning headers: localwiki.wsgi, referer: http://www.google.com/search?hl=en&tbo=d&site=&source=hp&q=unique+designs+tattoo+ann+arbor&oq=unique+designs+tattoo+ann+arbor&gs_l=mobile-gws-hp.3..0i22.1853.23002.0.23219.32.31.0.1.1.0.2251.15842.2j5j2j1j2j2j5j3j0j2.24.0.les%3B..0.0...1ac.1.OjDaRIsu-Bg
[Fri Nov 23 19:44:31 2012] [error] [client 110.85.36.46] Script timed out before returning headers: localwiki.wsgi, referer: http://arborwiki.org/Users/login/?next=/List_of_Ann_Arbor_forums
[Fri Nov 23 19:44:48 2012] [error] [client 62.31.187.155] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:45:09 2012] [error] [client 157.55.32.166] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:45:22 2012] [error] [client 110.85.36.46] Script timed out before returning headers: localwiki.wsgi, referer: http://arborwiki.org/Users/login/?next=/List_of_Ann_Arbor_forums
[Fri Nov 23 19:45:23 2012] [error] [client 66.249.73.231] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:45:24 2012] [error] [client 178.255.215.69] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:45:25 2012] [error] [client 180.76.5.164] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:45:31 2012] [error] [client 62.31.187.155] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:45:34 2012] [error] [client 68.61.99.221] Script timed out before returning headers: localwiki.wsgi, referer: http://us.yhs4.search.yahoo.com/yhs/search?hspart=w3i&hsimp=yhs-defalttabtransfer&type=W3i_YT,191,8_4,Search,20120626,18370,0,18,0&p=by%20the%20pound
[Fri Nov 23 19:45:39 2012] [error] [client 173.199.120.99] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:45:39 2012] [error] [client 157.55.32.166] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:46:11 2012] [error] [client 157.55.32.166] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:46:14 2012] [error] [client 62.31.187.155] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:46:11 2012] [error] [client 157.55.32.166] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:46:14 2012] [error] [client 62.31.187.155] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:46:39 2012] [error] [client 157.55.32.166] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:46:42 2012] [error] [client 173.199.115.91] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:46:58 2012] [error] [client 62.31.187.155] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:47:16 2012] [error] [client 65.52.111.87] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:47:16 2012] [error] [client 157.55.35.98] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:47:23 2012] [error] [client 218.30.103.146] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:47:42 2012] [error] [client 62.31.187.155] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:47:51 2012] [error] [client 157.55.33.182] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:47:58 2012] [error] [client 76.226.172.86] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:48:02 2012] [error] [client 98.243.173.239] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:48:06 2012] [error] [client 157.55.35.44] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:48:26 2012] [error] [client 66.249.73.231] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:48:27 2012] [error] [client 62.31.187.155] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:48:28 2012] [error] [client 173.199.115.171] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:48:43 2012] [error] [client 157.55.32.166] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:48:51 2012] [error] [client 41.104.38.128] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:48:54 2012] [error] [client 98.209.214.48] Script timed out before returning headers: localwiki.wsgi, referer: http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=0CC8QFjAB&url=http%3A%2F%2Farborwiki.org%2FWater_Street_Redevelopment_Project&ei=WhawUOHVEcnkyQH2hYHIBw&usg=AFQjCNHH92kMzCJVW2VT2VcteeFRlginCQ
[Fri Nov 23 19:48:55 2012] [error] [client 91.234.146.142] Script timed out before returning headers: localwiki.wsgi, referer: http://arborwiki.org/city/index.php?title=Special:UserLogin&type=signup
[Fri Nov 23 19:49:10 2012] [error] [client 62.31.187.155] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:49:12 2012] [error] [client 157.55.35.44] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:49:23 2012] [error] [client 178.255.215.69] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:49:35 2012] [error] [client 71.227.42.151] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:49:39 2012] [error] [client 157.55.33.182] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:49:43 2012] [error] [client 98.250.33.114] Script timed out before returning headers: localwiki.wsgi, referer: http://www.google.com/url?sa=t&source=web&cd=11&ved=0CD8QFjAK&url=http%3A%2F%2Farborwiki.org%2FThe_Scrap_Box&ei=VhewUKDEN6j8yAGCkYCwBA&usg=AFQjCNEBwyFyMKAQFt-69dpBE_K3FvDOpw&rct=j
[Fri Nov 23 19:49:47 2012] [error] [client 24.2.223.54] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:49:53 2012] [error] [client 62.31.187.155] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:50:10 2012] [error] [client 157.55.32.166] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:50:36 2012] [error] [client 62.31.187.155] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:50:39 2012] [error] [client 173.199.119.43] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:50:41 2012] [error] [client 157.55.32.166] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:50:57 2012] [error] [client 178.255.215.69] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:51:07 2012] [error] [client 98.137.207.210] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:51:08 2012] [error] [client 157.55.35.44] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:51:16 2012] [error] [client 173.199.115.123] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:51:19 2012] [error] [client 62.31.187.155] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:51:26 2012] [error] [client 66.249.73.231] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:51:38 2012] [error] [client 157.55.34.32] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:52:03 2012] [error] [client 62.31.187.155] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:52:11 2012] [error] [client 157.55.35.98] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:52:19 2012] [error] [client 173.199.116.251] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:52:23 2012] [error] [client 218.30.103.146] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:52:40 2012] [error] [client 71.238.149.106] Script timed out before returning headers: localwiki.wsgi, referer: http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&cad=rja&ved=0CD8QFjAB&url=http%3A%2F%2Farborwiki.org%2FMiller_Manor&ei=cBmwULmxG4m4yQGh9YHIDQ&usg=AFQjCNF4TmM-HEm5ZisvEa3kZOVTH7Tmqg&sig2=mTttqhq_Y_7TjHCyobBlLA
[Fri Nov 23 19:52:44 2012] [error] [client 180.76.5.54] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:52:45 2012] [error] [client 100.43.83.157] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:52:46 2012] [error] [client 71.238.8.204] Script timed out before returning headers: localwiki.wsgi, referer: http://search.yahoo.com/search;_ylt=A0oG7ielF7BQBBsA3pJXNyoA?p=john%20seto%2C%20chief%20of%20police&fr2=sb-top&fr=yfp-t-701
[Fri Nov 23 19:52:47 2012] [error] [client 62.31.187.155] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:53:05 2012] [error] [client 157.55.35.98] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:53:07 2012] [error] [client 65.55.24.237] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:53:21 2012] [error] [client 174.37.213.34] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:53:23 2012] [error] [client 112.111.52.186] Script timed out before returning headers: localwiki.wsgi, referer: http://arborwiki.org/How_to_Participate_in_a_Public_Meeting
[Fri Nov 23 19:53:30 2012] [error] [client 62.31.187.155] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:53:30 2012] [error] [client 71.238.149.106] Script timed out before returning headers: localwiki.wsgi, referer: http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&cad=rja&ved=0CD8QFjAB&url=http%3A%2F%2Farborwiki.org%2FMiller_Manor&ei=cBmwULmxG4m4yQGh9YHIDQ&usg=AFQjCNF4TmM-HEm5ZisvEa3kZOVTH7Tmqg&sig2=mTttqhq_Y_7TjHCyobBlLA
[Fri Nov 23 19:53:33 2012] [error] [client 178.255.215.69] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:53:50 2012] [error] [client 65.55.24.237] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:53:51 2012] [error] [client 98.137.207.210] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:53:52 2012] [error] [client 207.46.199.19] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:54:01 2012] [error] [client 71.238.149.106] Script timed out before returning headers: localwiki.wsgi, referer: http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&cad=rja&ved=0CD8QFjAB&url=http%3A%2F%2Farborwiki.org%2FMiller_Manor&ei=cBmwULmxG4m4yQGh9YHIDQ&usg=AFQjCNF4TmM-HEm5ZisvEa3kZOVTH7Tmqg&sig2=mTttqhq_Y_7TjHCyobBlLA
[Fri Nov 23 19:54:05 2012] [error] [client 173.199.115.67] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:54:08 2012] [error] [client 65.55.24.237] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:54:13 2012] [error] [client 62.31.187.155] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:54:17 2012] [error] [client 112.111.52.186] Script timed out before returning headers: localwiki.wsgi, referer: http://arborwiki.org/
[Fri Nov 23 19:54:27 2012] [error] [client 66.249.73.231] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:54:43 2012] [error] [client 65.55.24.237] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:54:55 2012] [error] [client 62.31.187.155] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:55:02 2012] [error] [client 71.238.149.106] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:55:06 2012] [error] [client 71.238.149.106] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 19:55:08 2012] [error] [client 157.55.33.182] Script timed out before returning headers: localwiki.wsgi
[Fri Nov 23 20:02:09 2012] [error] server is within MinSpareThreads of MaxClients, consider raising the MaxClients setting
[Fri Nov 23 20:05:36 2012] [error] server reached MaxClients setting, consider raising the MaxClients setting

One thing I noticed is that logging (access log) wasn't enabled. I enabled logging so that we can correlate these errors with requests. (and checking in a fix so this will happen for all other new installs)

Restarting apache make it work again, of course.

@philipn philipn reopened this
@mivanov
LocalWiki.org member
@philipn
LocalWiki.org member
@mivanov
LocalWiki.org member
@vielmetti

Arborwiki is now running the latest and greatest Localwiki install, and with it an updated version of a number of Ubuntu libraries.

@vielmetti

Arborwiki stuck again, checking...

The console message says "restart required", so I'm going to reboot.

@philipn
LocalWiki.org member

Notes: try mikes suggestions, look at keepalive, connection timeout. Try simulating reallllly slow requests

@samrose

This really seems like something either something is eating up all of the memory, or something is not responding when expected to by the code. My first reaction would be to run any unit tests that are included with LocalWiki to see if they work. I'd also hit the server with ab and watch the logs and "top"

@mivanov
LocalWiki.org member

I'll take a more detailed look at this first thing in the morning.

@philipn
LocalWiki.org member
@vielmetti
@mivanov
LocalWiki.org member

I've implemented additional logging that should help us catch the requests that are crashing or getting stuck. I'll check up on it daily, and please let me know if you notice anything as well.

@mivanov
LocalWiki.org member

Checking up on the request/response log, there haven't been any crashed requests in the first day. A few very slow requests coming from Google's crawler (took about 8 seconds) ended with a 404, which is odd, so I'll keep an eye on that.

We have a couple of ideas for things to try (like disable persistent connections to Apache), but it's best to understand what exactly is happening, so I'm hesitant to change too many variables at once. I will try a few things in hopes of reproducing this issue sooner though.

@vielmetti

Mike -

One of the things we did earlier was throttle down the frequency of crawls coming from the Microsoft crawler, which would occasionally ask for 20 or 30 pages at once.

Can you characterize the very slow requests? Do they look normal, or is there something funky about them?

@mivanov
LocalWiki.org member

These slow ones are all requests for images, but nothing else unusual that I can see.

@vielmetti

Stuck again! I captured netstat output and restarted.

https://gist.github.com/4321065

@vielmetti

Based on http://www.simplemachines.org/community/index.php?topic=489039.0 , I conclude we're gettting DDOS'd by an Ahrefsbot bot

http://ahrefs.com/robot/

@mivanov
LocalWiki.org member

My logging captured a few crashed requests, in the span of less than 2 minutes. These are requests that were seen by the WSGI application but were never completed. (There were certainly many more that never made it to the application) The IPs seem to be MSN bot

2012-12-17 12:53:47,525 INFO GET /map/Ypsilanti_District_Library 157.55.35.87
2012-12-17 12:53:47,720 INFO GET /shakin jake woods?show=1 157.55.34.32
2012-12-17 12:53:48,719 INFO GET /state theater?show=1 157.55.34.32
2012-12-17 12:53:50,908 INFO GET /tags/1927events 157.55.35.87
2012-12-17 12:53:51,364 INFO GET /Die_Washtenaw_Post 157.55.33.80
2012-12-17 12:53:51,374 INFO GET /Lakeshore_Village 157.55.33.80
2012-12-17 12:54:56,585 INFO GET /Mountain_Jack's/_tags/ 157.55.35.87
2012-12-17 12:54:57,350 INFO GET /Ricky_Jefferson/_tags/ 157.55.35.87
2012-12-17 12:54:57,479 INFO GET /Donuts/_tags/ 157.55.33.80
2012-12-17 12:54:57,517 INFO GET /Dormitories/_tags/ 157.55.33.80
2012-12-17 12:54:58,356 INFO GET /Richard_Deitering/_tags/ 157.55.34.35
2012-12-17 12:54:58,362 INFO GET /Rick_Snyder/_tags/ 157.55.34.35
2012-12-17 12:55:02,791 INFO GET /Watling_Boulevard 173.199.119.11
13 crashed and 84576 completed requests found.

The first errors in Apache's error log start shortly after that:

[Mon Dec 17 12:59:07 2012] [error] server is within MinSpareThreads of MaxClients, consider raising the MaxClients setting
[Mon Dec 17 12:59:44 2012] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Mon Dec 17 13:00:14 2012] [error] [client 157.55.34.35] Script timed out before returning headers: localwiki.wsgi
[Mon Dec 17 13:00:14 2012] [error] [client 157.55.34.35] Script timed out before returning headers: localwiki.wsgi
[Mon Dec 17 13:00:14 2012] [error] [client 157.55.35.87] Script timed out before returning headers: localwiki.wsgi
[Mon Dec 17 13:00:14 2012] [error] [client 157.55.35.87] Script timed out before returning headers: localwiki.wsgi
@samrose
@philipn
LocalWiki.org member

How do we kill requests that take too long to complete?

@philipn
LocalWiki.org member

I've set KeepAlive off on arborwiki, which is how we're running on the sites@localwiki.org server.

@vielmetti

It just got stuck again, and I did an "apache2ctl graceful" to restart it.

@vielmetti

I had to restart again just now.

@philipn
LocalWiki.org member

Added

User-agent: AhrefsBot
disallow: / 

to the arborwiki.org robots.txt. Let's see if that does anything. May take a bit for the bot to re-find the robots.txt.

We should set things up so it's impossible to kill an instance like this, though.

@mivanov
LocalWiki.org member

Looking at this again. Not surprised KeepAlive Off didn't do anything, but there are still a few different possibilities. I'm going to watch the Apache live status to see if any requests are hanging around.

@mivanov
LocalWiki.org member

Found tons of slow requests, which can probably explain the lockups. The last number on each line is the time inside WSGI to complete the request. Ignore the hex UUID thing:

2012-12-17 17:01:44,866 INFO 581e413c5c7f405fa7b245a02961420e GET /Knit_A_Round None 157.55.36.39 200 0:00:01.948618
2012-12-17 17:01:44,882 INFO 94700449f9234f24950db47b863b8817 GET /Birthday_Deals None 76.238.198.111 200 0:00:06.752652
2012-12-17 17:01:44,902 INFO 7b04911543c040758192f29e759b1c1f GET /Westwind_Milling/_tags/ None 157.55.32.108 200 0:00:01.580250
2012-12-17 17:01:44,954 INFO dc4d1c3b44a342f8b93477bfebe3d3aa GET /Users/register/?next=/Wireless_Ypsi None 173.199.119.11 200 0:00:01.021105
2012-12-17 17:01:44,972 INFO 642b5d2623df4c08a4b011a8cbb09735 GET /Lake_Fingerle None 157.55.35.87 404 0:00:01.255855
2012-12-17 17:01:45,011 INFO 417706ab5f584189a0d9744c920c4aa1 GET /Carmen_Mendoza-King None 157.55.33.80 404 0:00:01.002264
2012-12-17 17:01:45,012 INFO c8af9b8669f247059f0b6c5bdf1a3d61 GET /Warren_Staebler None 173.199.119.11 404 0:00:01.013600
2012-12-17 17:01:45,015 INFO 402731159e984e0bbe99b95936850252 GET /map/tags/nhuronstypsilanti None 173.199.119.11 200 0:00:01.820891
2012-12-17 17:01:45,016 INFO 31ad43c9c72a472f9523eebfeafdc0fb GET /West_Circle_Drive None 173.199.119.11 404 0:00:01.059838
2012-12-17 17:01:45,016 INFO 64d2afea27bd44e2bba607ae6e9ce6c0 GET /CareOne_Rental None 157.55.33.80 404 0:00:01.001047
2012-12-17 17:01:45,017 INFO 8cbc657f7789422b87200efe52f05e6f GET /2009-10_Sign_Board_Crisis None 96.47.225.66 200 0:00:02.893055
2012-12-17 17:01:45,047 INFO 535829dfd3364ae9807c348f2f70efe6 GET /farmers' market?show=1 None 157.55.32.108 404 0:00:01.036204
2012-12-17 17:01:45,082 INFO 11b8e1dbbe4c47c782c2dfc35e2f36d4 GET /Carlton_Hertler None 157.55.33.80 200 0:00:01.626623
2012-12-17 17:01:45,090 INFO c9ad8292c8f74450b2bd83f631bee91e GET /Queen_of_Hearts/_tags/ None 157.55.35.87 200 0:00:01.135572
2012-12-17 17:01:45,092 INFO bd7cec3700f34ec8ae388db1e0c0e80b GET /Car_Repair None 157.55.33.80 200 0:00:01.890685
2012-12-17 17:01:45,092 INFO ccfc19585999404283342ba0bcd7087b GET /tags/developers None 173.199.119.11 200 0:00:01.486470
2012-12-17 17:01:45,095 INFO 03fba714c713468784a54623224d21c0 GET /tags/airportdrannarbor None 173.199.119.11 200 0:00:01.049052
2012-12-17 17:01:45,100 INFO 400c1f1c3d5c45fb90db0f0032fc1959 GET /Kyung_Shyn_Yoon None 157.55.35.87 200 0:00:01.564069
2012-12-17 17:01:45,101 INFO b4472a89ba4844a195360f2fed0249ab GET /Kung_Pao_Chicken None 157.55.36.39 200 0:00:01.697081
2012-12-17 17:01:45,104 INFO 1e1cbc618eb74c53bcedb6d95002107c GET /Rufus_Nutting None 173.199.119.11 200 0:00:01.744585
2012-12-17 17:01:45,108 INFO 6cc70d168d37442fb03173305062f4b1 GET /tags/eventslistedinsonicliving None 173.199.119.11 200 0:00:01.023020
2012-12-17 17:01:45,111 INFO 8638630b001644baa203f27922621d6c GET /Carmen_(dog) None 157.55.33.80 200 0:00:01.789540
2012-12-17 17:01:45,118 INFO b4d6c87633004636be9ecacdb70b8e6d GET /Green_Road_Park_and_Ride None 157.55.33.80 404 0:00:01.007803
2012-12-17 17:01:45,123 INFO 216408f785d74a908a5a705c723caa09 GET /Users/login/?next=/Eastside_Recreation_Center None 157.55.35.87 200 0:00:01.101591
2012-12-17 17:01:45,123 INFO e6c44b2a078044e299d1c7f34973c4fa GET /frog island brewing?show=1 None 157.55.32.108 404 0:00:01.084759
2012-12-17 17:01:46,048 INFO 0e32eca22a7a4f36a384a07b7f8715c7 GET /Bob_Seger/_tags/ None 157.55.35.87 200 0:00:01.120224
2012-12-17 17:01:46,061 INFO 4b43aa53ac61422096880523ed1af984 GET /Canada None 157.55.33.80 200 0:00:01.847494
2012-12-17 17:01:46,066 INFO ea579f28711d497a874846947fd329e3 GET /E._Royster_Harper None 157.55.33.80 200 0:00:02.172614
2012-12-17 17:01:46,081 INFO d3108a6e6dcd46e9ada5299e5786a4b5 GET /Zwerdling's_Fur_Shop None 157.55.33.80 200 0:00:01.643933
2012-12-17 17:01:46,141 INFO 2e499c36b3e44ef0abbe43f31fa5acaa GET /Christmas_trees_near_Ann_Arbor None 68.40.49.84 200 0:00:02.554145
2012-12-17 17:01:46,142 INFO 805af3514fe04cb39156ff669aa1c406 GET /Bolgos_Farms/_tags/ None 157.55.35.87 200 0:00:01.099323
2012-12-17 17:01:46,193 INFO 68c22fb5c64a4e12915992f11daa1742 GET /Washtenaw_County_Republican_Committee None 157.55.36.39 200 0:00:01.440036
2012-12-17 17:01:46,215 INFO ea733332d8bc4c5484e7b30cfb15b4e9 GET /Quasi-Con_2013/Localwiki_panel_proposal None 157.55.35.87 200 0:00:01.207490
2012-12-17 17:01:46,226 INFO e12dd3ff5cdb4e45a7daf34cda4db928 GET /Washtenaw_County_Planning_and_Environment_Department None 157.55.36.39 404 0:00:01.075936
2012-12-17 17:01:46,237 INFO 8382072c43f04705973537940e5d215f GET /Cabela's None 157.55.33.80 200 0:00:01.510975
2012-12-17 17:01:46,261 INFO be6433d11154460581e8b57d089620d2 GET /Westwind_Milling/_tags/ None 157.55.32.108 200 0:00:01.106207
2012-12-17 17:01:46,263 INFO b07429a5dc4e4335ab4d97ec9b37e19e GET /Courthouse_Square_Apartments None 157.55.33.80 200 0:00:01.396551
2012-12-17 17:01:46,658 INFO 6e9587db9bdd41628a2aff2b4140ecdd GET / None 76.238.198.111 200 0:00:05.133255
2012-12-17 17:01:46,673 INFO cb6dae3548f04a6699418a3fd09de927 GET /Washtenaw_County_Assessors None 173.199.119.11 200 0:00:01.490766
2012-12-17 17:01:46,699 INFO 7eda65283dc84ff98c00cbc1bbe31145 GET /Lunch_under_5_dollars None 141.214.17.5 200 0:00:04.622990
2012-12-17 17:01:46,743 INFO 26ca4e6eae114077979c25640df9fcf7 GET /Christmas_trees_near_Ann_Arbor None 198.228.228.152 200 0:00:02.251760
2012-12-17 17:01:46,859 INFO c1e72a365739484f971fab18e39fb3cb GET /Knit_A_Round None 157.55.36.39 200 0:00:01.484889
2012-12-17 17:01:46,863 INFO b40c50f8428147728ab58c1f205dbe63 GET /Kung_Pao_Chicken None 157.55.36.39 200 0:00:01.649359
2012-12-17 17:01:46,872 INFO 4c5b143f9e0c43299748c46b23e94275 GET /Queen_of_Hearts/_tags/ None 157.55.35.87 200 0:00:01.367747
2012-12-17 17:01:46,940 INFO 08736e4c71e64199a0388fc9e4317e6e GET /Carlton_Hertler None 157.55.33.80 200 0:00:01.579009
2012-12-17 17:01:46,948 INFO c90f288c854c4c998c00d5e53e03c48d GET /Car_Repair None 157.55.33.80 200 0:00:01.535841
2012-12-17 17:01:46,983 INFO e55d88d9ad6b4315ac11f0311d033f23 GET / None 76.238.198.111 200 0:00:04.906526
2012-12-17 17:01:47,099 INFO ae2719e9f39a43b5b571b145f5f7b066 GET /1981_Bursley_Hall_murders None 199.94.67.143 200 0:00:01.913548
2012-12-17 17:01:47,226 INFO 6175ed98656a454a846edd4d802d301c GET /Carmen_(dog) None 157.55.33.80 200 0:00:01.618061
2012-12-17 17:01:47,439 INFO a2a0ed33ee8845d1bf6d3d3525225cb7 GET /Cunningham_Drug_Store/_tags/ None 157.55.36.39 200 0:00:01.121647
2012-12-17 17:01:47,546 INFO fe2353065b6c46ce9c3775241241da78 GET /Christmas_trees_near_Ann_Arbor None 198.228.228.152 200 0:00:02.209006
2012-12-17 17:01:47,699 INFO 6b539bebf1174defb3411f61afd90440 GET /Cynthia_Wilbanks/_tags/ None 157.55.36.39 200 0:00:01.083976
2012-12-17 17:01:47,755 INFO a40ef83bdbca4cb48d4bea3851b1423d GET /Christmas_trees_near_Ann_Arbor None 198.228.228.152 200 0:00:02.414118
2012-12-17 17:01:47,808 INFO 4d9169e4c381480d927b3101e937cb77 GET /1981_Bursley_Hall_murders None 199.94.67.143 200 0:00:02.283481
2012-12-17 17:01:47,833 INFO 26274af6a1a1495f824c8646fda1ce30 GET /Focus_EMU/_edit None 100.43.83.157 200 0:00:01.456932
2012-12-17 17:01:47,934 INFO a839aea7bbc5426592bc2d1db48e62e4 GET /Kiwanis_Club_of_Ann_Arbor None 157.55.33.80 200 0:00:04.105533
2012-12-17 17:01:47,978 INFO 6174ff49398447769e911ed0371328f1 GET /Knowledge_Navigation_Center None 157.55.33.80 200 0:00:02.165214
2012-12-17 17:01:48,250 INFO 3cda1ffef7e24d3fb448e812b2ac9c10 GET /Kristen_Walter None 157.55.33.80 200 0:00:01.230014
2012-12-17 17:01:48,329 INFO c1732af873ff495db6e404a05ded5c45 GET /Washtenaw_Rd_(Ypsilanti)/_tags/ None 173.199.119.11 200 0:00:01.417388
2012-12-17 17:01:48,373 INFO 1169b7a6b5694a0593113dd322ced6da GET /tags/duckeggs None 173.199.119.11 200 0:00:01.068273
2012-12-17 17:01:48,398 INFO ddc132a41b7144afbc3a3c573da9b480 GET /Delhi?&redirected_from=delhi%20mills None 173.199.119.11 200 0:00:02.528336
2012-12-17 17:01:48,472 INFO ef14d3db48db4f0090e055404515f840 GET /King-Seeley_Corporation None 173.199.119.11 200 0:00:02.084066
2012-12-17 17:01:48,522 INFO eb8f9d6b55204cc2ba7f50af12cec779 GET /Fred_Wilpon None 69.174.87.92 200 0:00:01.611710
2012-12-17 17:01:48,762 INFO 4c9793c51df34dc68f20a5c5676c53e3 GET /Florida None 157.55.36.39 200 0:00:01.670436
2012-12-17 17:01:48,809 INFO ca36ba4d32a94613b48b702aa9be4ce2 GET /LLamasoft None 157.55.33.80 200 0:00:01.737625
2012-12-17 17:01:48,835 INFO 9f26e96a3ffc4e09adf3fd90cfe21f21 GET /LaSalle_Bank None 157.55.33.80 404 0:00:01.003657
2012-12-17 17:01:49,111 INFO 4c59890c199745e1be5f8ab24b24c001 GET /Washtenaw_County_Republican_Committee None 157.55.36.39 200 0:00:01.468911
2012-12-17 17:01:49,198 INFO 516a086eecde4a3f92c5aa5a9f7d3ecb GET /Lake_Forest_Golf_Club None 157.55.33.80 200 0:00:01.454644
2012-12-17 17:01:49,199 INFO a00e0f42f9eb464791791a9bda7ff1fe GET /Plymouth_Orchards None 65.55.52.96 200 0:00:01.286171
2012-12-17 17:01:49,294 INFO e1c6930072e142cf8bc8bc788a535c40 GET /The_Villas None 157.55.33.80 200 0:00:01.759938
2012-12-17 17:01:49,328 INFO c56f9931e3634618a80b5fcaaee4b18a GET /2009-10_Sign_Board_Crisis None 96.47.225.74 200 0:00:01.829721
2012-12-17 17:01:49,470 INFO a60fb8472ceb4965bff1740cbacd1e89 GET /Richard_Darr/_tags/ None 157.55.32.145 200 0:00:01.046599
2012-12-17 17:01:49,528 INFO 772d341f3dfd43e18d8c8e13f099aa21 GET /1800s_births None 157.55.34.35 200 0:00:01.513503
2012-12-17 17:01:49,545 INFO b7b4d1212df444fda65ce5369f0252af GET /Richard_Glenn_Ceely/_tags/ None 157.55.32.145 200 0:00:01.012448
2012-12-17 17:01:49,598 INFO 3e1bb3a1a7fe48c280742ff06a5cd22f GET /Ralph_Lange/_tags/ None 157.55.32.166 200 0:00:01.222133
2012-12-17 17:01:49,663 INFO 8464e4fb80e84e35bf56d48e66b24c65 GET /Warren_Road/_tags/ None 173.199.119.11 200 0:00:01.102825
2012-12-17 17:01:49,718 INFO 345a34d56aaa4c5daa9ef1d6fad1798b GET /1801_births None 157.55.34.35 200 0:00:01.462110
2012-12-17 17:01:49,787 INFO d713fac062164c4698b0d13c7c933806 GET /Ayse's_Cafe None 99.29.192.128 200 0:00:01.290424
2012-12-17 17:01:49,823 INFO 4046854dfcc9484091faf246b43aff44 GET /WWWW None 66.249.73.231 200 0:00:01.096636
2012-12-17 17:01:49,987 INFO 3621ff2487ab480d85300f13034734ef GET /Robert_M._Warner None 173.199.119.11 200 0:00:01.171177
2012-12-17 17:01:50,178 INFO 6000d20b9e0449bc8e66983cdb2f7139 GET /1830s_births None 157.55.33.80 200 0:00:01.135335
@mivanov
LocalWiki.org member

I set processes=4 threads=25 in the WSGI daemon config, which should give a lot more capacity and help stave off the snowball effect that is seen in the above.

@vielmetti

We haven't seen this problem in months, closing.

@vielmetti vielmetti closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.