Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

poll for jevdemon (twitter) regularly blows memory budget #525

Closed
snarfed opened this issue Oct 23, 2015 · 9 comments
Closed

poll for jevdemon (twitter) regularly blows memory budget #525

snarfed opened this issue Oct 23, 2015 · 9 comments
Labels
backfeed

Comments

@snarfed
Copy link
Owner

snarfed commented Oct 23, 2015

https://brid.gy/twitter/jevdemon . not sure why; his site http://www.evdemon.org/ runs known, and his other URL http://looselycoupledthinking.com/ just redirects to evdemon.org.

example log:

 0.1.0.2 - - [23/Oct/2015:06:45:08 -0700] "POST /_ah/queue/poll HTTP/1.1" 200 84 "http://www.brid.gy/_ah/queue/poll" "AppEngine-Google; (+http://code.google.com/appengine)"

13:44:26.0 D Params: UnicodeMultiDict([(u'last_polled', u'2015-10-23-13-05-09'), (u'source_key', u'aglzfmJy...')])
13:44:26.0 I Source: jevdemon (Twitter) jevdemon, http://www.brid.gy/twitter/jevdemon
13:44:26.0 I Last poll: http://www.brid.gy/log?start_time=1445605509&key=aglzfmJyaWQtZ3lyFQsSB1R3aXR0ZXIiCGpldmRlbW9uDA
13:44:26.0 D Using ETag None, last activity id 657363538386681857
13:44:26.0 D Fetching https://api.twitter.com/1.1/search/tweets.json?q=%22evdemon.org%22+OR+%22looselycoupledthinking.com%22&include_entities=true&result_type=recent&count=50
13:44:26.1 D Fetching https://api.twitter.com/1.1/search/tweets.json?q=%40jevdemon&include_entities=true&result_type=recent&count=100&since_id=657363538386681857
13:44:26.2 D Fetching https://api.twitter.com/1.1/search/tweets.json?q=%40CigojMilos&include_entities=true&result_type=recent&count=100&since_id=657363538386681857
13:44:26.3 D Fetching https://api.twitter.com/1.1/search/tweets.json?q=%40clemensv&include_entities=true&result_type=recent&count=100&since_id=657363538386681857
13:44:26.6 D Fetching https://api.twitter.com/1.1/search/tweets.json?q=%40afalk&include_entities=true&result_type=recent&count=100&since_id=657363538386681857
13:44:26.7 D Fetching https://api.twitter.com/1.1/search/tweets.json?q=%40silverfighter&include_entities=true&result_type=recent&count=100&since_id=657363538386681857
13:44:26.8 D Fetching https://api.twitter.com/1.1/favorites/list.json?screen_name=&include_entities=true
13:44:27.1 D Fetching https://api.twitter.com/1.1/account/verify_credentials.json
13:44:27.3 D Fetching https://api.twitter.com/1.1/statuses/user_timeline.json?include_entities=true&count=50
13:44:27.7 D Fetching https://api.twitter.com/1.1/search/tweets.json?q=%40jevdemon&include_entities=true&result_type=recent&count=100&since_id=657363538386681857
13:44:27.7 I Found 70 activities: [...]
13:44:27.7 D refetching h-feed for source jevdemon (Twitter)
13:44:27.7 D attempting to refetch h-feed for jevdemon (Twitter)
13:44:27.7 D fetching author url http://www.evdemon.org/
13:44:27.7 I Starting new HTTP connection (1): www.evdemon.org
13:44:30.2 D "GET / HTTP/1.1" 200 None
13:44:31.1 D No h-feed found, fallback to top-level h-entrys.
13:44:31.2 D skipping feed of type application/rss+xml
13:44:31.2 D skipping feed of type application/rss+xml
13:44:31.2 D fetching author's rel-feed http://www.evdemon.org/content/all
13:44:31.2 I Starting new HTTP connection (1): www.evdemon.org
13:44:32.7 D "GET /content/all HTTP/1.1" 200 None
13:44:32.7 D author's rel-feed fetched successfully http://www.evdemon.org/content/all
13:44:33.7 D No h-feed found, fallback to top-level h-entrys.
13:44:33.8 D processing permalink: http://www.evdemon.org/2015/a-port-of-openssh-to-windows-powershellwin32-openssh
13:44:33.8 D previously found relationship(s) for original http://www.evdemon.org/2015/a-port-of-openssh-to-windows-powershellwin32-openssh: [u'https://twitter.com/jevdemon/status/656640584149413888']
13:44:33.8 D fetching post permalink http://www.evdemon.org/2015/a-port-of-openssh-to-windows-powershellwin32-openssh
13:44:33.8 I Starting new HTTP connection (1): www.evdemon.org
13:44:34.7 D "GET /2015/a-port-of-openssh-to-windows-powershellwin32-openssh HTTP/1.1" 200 None
13:44:35.0 D rel-syndication links: ['https://twitter.com/jevdemon/status/656640584149413888']
13:44:35.0 D u-syndication links: ['https://twitter.com/jevdemon/status/656640584149413888']
13:44:35.0 D processing permalink: http://www.evdemon.org/2015/apple-ceo-defends-encryption-opposes-government-back-door-if-theres
13:44:35.0 D previously found relationship(s) for original http://www.evdemon.org/2015/apple-ceo-defends-encryption-opposes-government-back-door-if-theres: [u'https://twitter.com/jevdemon/status/656547904702709760']
13:44:35.0 D fetching post permalink http://www.evdemon.org/2015/apple-ceo-defends-encryption-opposes-government-back-door-if-theres
13:44:35.0 I Starting new HTTP connection (1): www.evdemon.org
13:44:36.0 D "GET /2015/apple-ceo-defends-encryption-opposes-government-back-door-if-theres HTTP/1.1" 200 None
13:44:36.2 D rel-syndication links: ['https://twitter.com/jevdemon/status/656547904702709760']
13:44:36.2 D u-syndication links: ['https://twitter.com/jevdemon/status/656547904702709760']
13:44:36.2 D processing permalink: http://www.evdemon.org/2015/learning-more-about-quill
13:44:36.2 D fetching post permalink http://www.evdemon.org/2015/learning-more-about-quill
13:44:36.3 I Starting new HTTP connection (1): www.evdemon.org
13:44:37.9 D "GET /2015/learning-more-about-quill HTTP/1.1" 200 None
13:44:38.5 D no syndication links from http://www.evdemon.org/2015/learning-more-about-quill to current source jevdemon (Twitter).
13:44:38.5 D processing permalink: http://www.evdemon.org/2015/taking-quill-out-for-a-quick-test-drive-quill-is
13:44:38.5 D fetching post permalink http://www.evdemon.org/2015/taking-quill-out-for-a-quick-test-drive-quill-is
13:44:38.6 I Starting new HTTP connection (1): www.evdemon.org
13:44:39.5 D "GET /2015/taking-quill-out-for-a-quick-test-drive-quill-is HTTP/1.1" 200 None
13:44:39.7 D no syndication links from http://www.evdemon.org/2015/taking-quill-out-for-a-quick-test-drive-quill-is to current source jevdemon (Twitter).
13:44:39.7 D processing permalink: http://www.evdemon.org/2015/if-you-cant-build-a-well-structured-monolith-what-makes-you
13:44:39.7 D previously found relationship(s) for original http://www.evdemon.org/2015/if-you-cant-build-a-well-structured-monolith-what-makes-you: [u'https://twitter.com/jevdemon/status/655946541325156352']
13:44:39.8 D fetching post permalink http://www.evdemon.org/2015/if-you-cant-build-a-well-structured-monolith-what-makes-you
13:44:39.8 I Starting new HTTP connection (1): www.evdemon.org
13:44:40.5 D "GET /2015/if-you-cant-build-a-well-structured-monolith-what-makes-you HTTP/1.1" 200 None
13:44:40.7 D rel-syndication links: ['https://twitter.com/jevdemon/status/655946541325156352']
13:44:40.7 D u-syndication links: ['https://twitter.com/jevdemon/status/655946541325156352']
13:44:40.7 D processing permalink: http://www.evdemon.org/2015/fb-admits-that-its-app-is-draining-your-iphones-battery
13:44:40.7 D previously found relationship(s) for original http://www.evdemon.org/2015/fb-admits-that-its-app-is-draining-your-iphones-battery: [u'https://twitter.com/jevdemon/status/657351311482982400']
13:44:40.8 D fetching post permalink http://www.evdemon.org/2015/fb-admits-that-its-app-is-draining-your-iphones-battery
13:44:40.8 I Starting new HTTP connection (1): www.evdemon.org
13:44:41.5 D "GET /2015/fb-admits-that-its-app-is-draining-your-iphones-battery HTTP/1.1" 200 None
13:44:41.7 D rel-syndication links: ['https://twitter.com/jevdemon/status/657351311482982400']
13:44:41.7 D u-syndication links: ['https://twitter.com/jevdemon/status/657351311482982400']
13:44:41.7 D processing permalink: http://www.evdemon.org/comic/5cada44771955d80fe50bc0b546945f0/own-your-content-or-risk-losing-it
13:44:41.7 D previously found relationship(s) for original http://www.evdemon.org/comic/5cada44771955d80fe50bc0b546945f0/own-your-content-or-risk-losing-it: [u'https://twitter.com/jevdemon/status/655443897875697665']
13:44:41.7 D fetching post permalink http://www.evdemon.org/comic/5cada44771955d80fe50bc0b546945f0/own-your-content-or-risk-losing-it
13:44:41.7 I Starting new HTTP connection (1): www.evdemon.org
13:44:42.4 D "GET /comic/5cada44771955d80fe50bc0b546945f0/own-your-content-or-risk-losing-it HTTP/1.1" 200 None
13:44:42.7 D rel-syndication links: ['https://twitter.com/jevdemon/status/655443897875697665']
13:44:42.7 D u-syndication links: ['https://twitter.com/jevdemon/status/655443897875697665']
13:44:42.7 D processing permalink: http://www.evdemon.org/2015/nice-post-on-the-mathematics-of-it-complexity-also-consider
13:44:42.7 D previously found relationship(s) for original http://www.evdemon.org/2015/nice-post-on-the-mathematics-of-it-complexity-also-consider: [u'https://twitter.com/jevdemon/status/657257235077140480']
13:44:42.7 D fetching post permalink http://www.evdemon.org/2015/nice-post-on-the-mathematics-of-it-complexity-also-consider
13:44:42.7 I Starting new HTTP connection (1): www.evdemon.org
13:44:43.5 D "GET /2015/nice-post-on-the-mathematics-of-it-complexity-also-consider HTTP/1.1" 200 None
13:44:43.8 D rel-syndication links: ['https://twitter.com/jevdemon/status/657257235077140480']
13:44:43.8 D u-syndication links: ['https://twitter.com/jevdemon/status/657257235077140480']
13:44:43.8 D processing permalink: http://www.evdemon.org/2015/if-you-spend-your-time-putting-content-into-someone-elses
13:44:43.8 D previously found relationship(s) for original http://www.evdemon.org/2015/if-you-spend-your-time-putting-content-into-someone-elses: [u'https://twitter.com/jevdemon/status/657363538386681857']
13:44:43.8 D fetching post permalink http://www.evdemon.org/2015/if-you-spend-your-time-putting-content-into-someone-elses
13:44:43.8 I Starting new HTTP connection (1): www.evdemon.org
13:44:45.6 D "GET /2015/if-you-spend-your-time-putting-content-into-someone-elses HTTP/1.1" 200 None
13:44:45.8 D rel-syndication links: ['https://twitter.com/jevdemon/status/657363538386681857']
13:44:45.8 D u-syndication links: ['https://twitter.com/jevdemon/status/657363538386681857']
13:44:45.8 D processing permalink: http://www.evdemon.org/2015/belated-follow-up-to-nyt-story-in-which-an-apparent-disgruntled
13:44:45.8 D previously found relationship(s) for original http://www.evdemon.org/2015/belated-follow-up-to-nyt-story-in-which-an-apparent-disgruntled: [u'https://twitter.com/jevdemon/status/657286124558155776']
13:44:45.8 D fetching post permalink http://www.evdemon.org/2015/belated-follow-up-to-nyt-story-in-which-an-apparent-disgruntled
13:44:45.9 I Starting new HTTP connection (1): www.evdemon.org
13:44:46.8 D "GET /2015/belated-follow-up-to-nyt-story-in-which-an-apparent-disgruntled HTTP/1.1" 200 None
13:44:47.0 D rel-syndication links: ['https://twitter.com/jevdemon/status/657286124558155776']
13:44:47.0 D u-syndication links: ['https://twitter.com/jevdemon/status/657286124558155776']
13:44:47.0 D fetching author url http://www.evdemon.org/
13:44:47.0 I Starting new HTTP connection (1): www.evdemon.org
13:44:49.4 D "GET / HTTP/1.1" 200 None
13:44:50.4 D No h-feed found, fallback to top-level h-entrys.
13:44:50.5 D skipping feed of type application/rss+xml
13:44:50.5 D skipping feed of type application/rss+xml
13:44:50.5 D fetching author's rel-feed http://www.evdemon.org/content/all
13:44:50.5 I Starting new HTTP connection (1): www.evdemon.org
13:44:51.9 D "GET /content/all HTTP/1.1" 200 None
13:44:51.9 D author's rel-feed fetched successfully http://www.evdemon.org/content/all
13:44:52.9 D No h-feed found, fallback to top-level h-entrys.
13:44:53.0 D processing permalink: http://www.evdemon.org/2015/a-port-of-openssh-to-windows-powershellwin32-openssh
13:44:53.0 D previously found relationship(s) for original http://www.evdemon.org/2015/a-port-of-openssh-to-windows-powershellwin32-openssh: [u'https://twitter.com/jevdemon/status/656640584149413888']
13:44:53.0 D fetching post permalink http://www.evdemon.org/2015/a-port-of-openssh-to-windows-powershellwin32-openssh
13:44:53.0 I Starting new HTTP connection (1): www.evdemon.org
13:44:53.7 D "GET /2015/a-port-of-openssh-to-windows-powershellwin32-openssh HTTP/1.1" 200 None
13:44:53.9 D rel-syndication links: ['https://twitter.com/jevdemon/status/656640584149413888']
13:44:53.9 D u-syndication links: ['https://twitter.com/jevdemon/status/656640584149413888']
13:44:54.0 D processing permalink: http://www.evdemon.org/2015/apple-ceo-defends-encryption-opposes-government-back-door-if-theres
13:44:54.0 D previously found relationship(s) for original http://www.evdemon.org/2015/apple-ceo-defends-encryption-opposes-government-back-door-if-theres: [u'https://twitter.com/jevdemon/status/656547904702709760']
13:44:54.0 D fetching post permalink http://www.evdemon.org/2015/apple-ceo-defends-encryption-opposes-government-back-door-if-theres
13:44:54.0 I Starting new HTTP connection (1): www.evdemon.org
13:44:54.7 D "GET /2015/apple-ceo-defends-encryption-opposes-government-back-door-if-theres HTTP/1.1" 200 None
13:44:54.9 D rel-syndication links: ['https://twitter.com/jevdemon/status/656547904702709760']
13:44:54.9 D u-syndication links: ['https://twitter.com/jevdemon/status/656547904702709760']
13:44:54.9 D processing permalink: http://www.evdemon.org/2015/learning-more-about-quill
13:44:54.9 D fetching post permalink http://www.evdemon.org/2015/learning-more-about-quill
13:44:55.0 I Starting new HTTP connection (1): www.evdemon.org
13:44:58.6 D "GET /2015/learning-more-about-quill HTTP/1.1" 200 None
13:44:59.3 D no syndication links from http://www.evdemon.org/2015/learning-more-about-quill to current source jevdemon (Twitter).
13:44:59.3 D processing permalink: http://www.evdemon.org/2015/taking-quill-out-for-a-quick-test-drive-quill-is
13:44:59.3 D fetching post permalink http://www.evdemon.org/2015/taking-quill-out-for-a-quick-test-drive-quill-is
13:44:59.3 I Starting new HTTP connection (1): www.evdemon.org
13:45:00.2 D "GET /2015/taking-quill-out-for-a-quick-test-drive-quill-is HTTP/1.1" 200 None
13:45:00.4 D no syndication links from http://www.evdemon.org/2015/taking-quill-out-for-a-quick-test-drive-quill-is to current source jevdemon (Twitter).
13:45:00.4 D processing permalink: http://www.evdemon.org/2015/if-you-cant-build-a-well-structured-monolith-what-makes-you
13:45:00.4 D previously found relationship(s) for original http://www.evdemon.org/2015/if-you-cant-build-a-well-structured-monolith-what-makes-you: [u'https://twitter.com/jevdemon/status/655946541325156352']
13:45:00.4 D fetching post permalink http://www.evdemon.org/2015/if-you-cant-build-a-well-structured-monolith-what-makes-you
13:45:00.5 I Starting new HTTP connection (1): www.evdemon.org
13:45:01.4 D "GET /2015/if-you-cant-build-a-well-structured-monolith-what-makes-you HTTP/1.1" 200 None
13:45:01.7 D rel-syndication links: ['https://twitter.com/jevdemon/status/655946541325156352']
13:45:01.7 D u-syndication links: ['https://twitter.com/jevdemon/status/655946541325156352']
13:45:01.7 D processing permalink: http://www.evdemon.org/2015/fb-admits-that-its-app-is-draining-your-iphones-battery
13:45:01.7 D previously found relationship(s) for original http://www.evdemon.org/2015/fb-admits-that-its-app-is-draining-your-iphones-battery: [u'https://twitter.com/jevdemon/status/657351311482982400']
13:45:01.7 D fetching post permalink http://www.evdemon.org/2015/fb-admits-that-its-app-is-draining-your-iphones-battery
13:45:01.8 I Starting new HTTP connection (1): www.evdemon.org
13:45:02.8 D "GET /2015/fb-admits-that-its-app-is-draining-your-iphones-battery HTTP/1.1" 200 None
13:45:03.0 D rel-syndication links: ['https://twitter.com/jevdemon/status/657351311482982400']
13:45:03.0 D u-syndication links: ['https://twitter.com/jevdemon/status/657351311482982400']
13:45:03.0 D processing permalink: http://www.evdemon.org/comic/5cada44771955d80fe50bc0b546945f0/own-your-content-or-risk-losing-it
13:45:03.0 D previously found relationship(s) for original http://www.evdemon.org/comic/5cada44771955d80fe50bc0b546945f0/own-your-content-or-risk-losing-it: [u'https://twitter.com/jevdemon/status/655443897875697665']
13:45:03.0 D fetching post permalink http://www.evdemon.org/comic/5cada44771955d80fe50bc0b546945f0/own-your-content-or-risk-losing-it
13:45:03.1 I Starting new HTTP connection (1): www.evdemon.org
13:45:03.9 D "GET /comic/5cada44771955d80fe50bc0b546945f0/own-your-content-or-risk-losing-it HTTP/1.1" 200 None
13:45:04.2 D rel-syndication links: ['https://twitter.com/jevdemon/status/655443897875697665']
13:45:04.2 D u-syndication links: ['https://twitter.com/jevdemon/status/655443897875697665']
13:45:04.2 D processing permalink: http://www.evdemon.org/2015/nice-post-on-the-mathematics-of-it-complexity-also-consider
13:45:04.2 D previously found relationship(s) for original http://www.evdemon.org/2015/nice-post-on-the-mathematics-of-it-complexity-also-consider: [u'https://twitter.com/jevdemon/status/657257235077140480']
13:45:04.2 D fetching post permalink http://www.evdemon.org/2015/nice-post-on-the-mathematics-of-it-complexity-also-consider
13:45:04.2 I Starting new HTTP connection (1): www.evdemon.org
13:45:05.5 D "GET /2015/nice-post-on-the-mathematics-of-it-complexity-also-consider HTTP/1.1" 200 None
13:45:05.7 D rel-syndication links: ['https://twitter.com/jevdemon/status/657257235077140480']
13:45:05.7 D u-syndication links: ['https://twitter.com/jevdemon/status/657257235077140480']
13:45:05.7 D processing permalink: http://www.evdemon.org/2015/if-you-spend-your-time-putting-content-into-someone-elses
13:45:05.7 D previously found relationship(s) for original http://www.evdemon.org/2015/if-you-spend-your-time-putting-content-into-someone-elses: [u'https://twitter.com/jevdemon/status/657363538386681857']
13:45:05.7 D fetching post permalink http://www.evdemon.org/2015/if-you-spend-your-time-putting-content-into-someone-elses
13:45:05.8 I Starting new HTTP connection (1): www.evdemon.org
13:45:06.6 D "GET /2015/if-you-spend-your-time-putting-content-into-someone-elses HTTP/1.1" 200 None
13:45:06.9 D rel-syndication links: ['https://twitter.com/jevdemon/status/657363538386681857']
13:45:06.9 D u-syndication links: ['https://twitter.com/jevdemon/status/657363538386681857']
13:45:06.9 D processing permalink: http://www.evdemon.org/2015/belated-follow-up-to-nyt-story-in-which-an-apparent-disgruntled
13:45:06.9 D previously found relationship(s) for original http://www.evdemon.org/2015/belated-follow-up-to-nyt-story-in-which-an-apparent-disgruntled: [u'https://twitter.com/jevdemon/status/657286124558155776']
13:45:06.9 D fetching post permalink http://www.evdemon.org/2015/belated-follow-up-to-nyt-story-in-which-an-apparent-disgruntled
13:45:06.9 I Starting new HTTP connection (1): www.evdemon.org
13:45:07.8 D "GET /2015/belated-follow-up-to-nyt-story-in-which-an-apparent-disgruntled HTTP/1.1" 200 None
13:45:08.0 D rel-syndication links: ['https://twitter.com/jevdemon/status/657286124558155776']
13:45:08.0 D u-syndication links: ['https://twitter.com/jevdemon/status/657286124558155776']
13:45:08.1 I Added poll task 17747537211711054641 with args {'countdown': 610.59640833677986}
13:45:08.2 F Exceeded soft private memory limit of 128 MB with 149 MB after servicing 161 requests total
13:45:08.2 W After handling this request, the process that handled this request was found to be using too much memory and was terminated. This is likely to cause a new process to be used for the next request to your application. If you see this message frequently, you may have a memory leak in your application.
@kylewm
Copy link
Contributor

kylewm commented Oct 23, 2015

This post has a relatively big inline data: image in it... http://www.evdemon.org/2015/learning-more-about-quill ... can't be the whole explanation, but I bet it isn't a coincidence

@snarfed
Copy link
Owner Author

snarfed commented Oct 23, 2015

holy crap, yes it does. 1MB! cc'ing @benwerd since i'm guessing he might want to cap the size of images he inlines into open graph tags like that.

@snarfed
Copy link
Owner Author

snarfed commented Oct 24, 2015

i was hoping i could give requests a response size limit so it would cut off and error out on big responses like this, but i didn't find an option like that.

requests does let you read the raw response bytes, so i could implement it myself, but I'd rather not. :/

@kylewm
Copy link
Contributor

kylewm commented Oct 24, 2015

You could check the Content-Length header before calling Response.content; I do this in my webmention handler https://github.com/kylewm/redwind/blob/master/redwind/plugins/wm_receiver.py#L229 ... That at least helps mitigate well-behaved servers that accidentally send large data.

@snarfed
Copy link
Owner Author

snarfed commented Oct 24, 2015

ooh interesting, thanks! so requests only reads the full response body when you access response.content (or .text), not before?

/me heads off to RTFM...

@snarfed
Copy link
Owner Author

snarfed commented Oct 24, 2015

aha, you opt into lazily fetching the response body with stream=True. this use case is even their example in the docs. :P http://docs.python-requests.org/en/latest/user/advanced/#body-content-workflow

(@kylewm looks like you'll probably want to add that to redwind's util.fetch_html too.)

@snarfed
Copy link
Owner Author

snarfed commented Oct 24, 2015

AAAARRGHH evdemon.org doesn't even give us Content-Length. :(:(:(

@kylewm
Copy link
Contributor

kylewm commented Oct 24, 2015

@snarfed oy, I really need to learn how to read.

@kylewm
Copy link
Contributor

kylewm commented Oct 24, 2015

heh, i guess you could use iter_content which is basically the same thing you said about dealing with raw bytes, but yeah i generally agree, would rather not.

@snarfed snarfed mentioned this issue Oct 24, 2015
Closed
@snarfed snarfed mentioned this issue Nov 2, 2015
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
backfeed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@snarfed @kylewm