Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Too many open files with PhantomJS #315

Open
ethus3h opened this issue Mar 8, 2016 · 0 comments
Open

Too many open files with PhantomJS #315

ethus3h opened this issue Mar 8, 2016 · 0 comments
Labels

Comments

@ethus3h
Copy link

ethus3h commented Mar 8, 2016

Got this error using grab-site with the PhantomJS integration:

https://twitter.com/tqbf/ on 03-07; 2,068.1 MB in 678,328 resp. at 7.4/s, 15,980 in q.; 5 con. w/ 0 ms delay; igoff
200 OK https://mobile.twitter.com/tqbf/status/684147355394834432?locale=ms
200 OK https://mobile.twitter.com/tqbf/status/684147355394834432?locale=ms
ERROR Proxy error
Traceback (most recent call last):
  File "/home/grabbot/.local/lib/python3.4/site-packages/wpull/proxy/server.py", line 57, in __call__
    yield From(session())
  File "/home/grabbot/.local/lib/python3.4/site-packages/trollius/tasks.py", line 257, in _step
    result = coro.throw(exc)
  File "/home/grabbot/.local/lib/python3.4/site-packages/wpull/proxy/server.py", line 140, in __call__
    response = yield From(session.fetch(request))
  File "/home/grabbot/.local/lib/python3.4/site-packages/trollius/tasks.py", line 259, in _step
    result = coro.send(value)
  File "/home/grabbot/.local/lib/python3.4/site-packages/wpull/http/client.py", line 87, in fetch
    self._recorder_session.request(request)
  File "/home/grabbot/.local/lib/python3.4/site-packages/wpull/recorder/demux.py", line 47, in request
    session.request(request)
  File "/home/grabbot/.local/lib/python3.4/site-packages/wpull/recorder/warc.py", line 475, in request
    self._child_session.request(request)
  File "/home/grabbot/.local/lib/python3.4/site-packages/wpull/recorder/warc.py", line 554, in request
    self._recorder.write_record(self._request_record)
  File "/home/grabbot/.local/lib/python3.4/site-packages/wpull/recorder/warc.py", line 275, in write_record
    with open(journal_filename, 'w') as file:
OSError: [Errno 24] Too many open files: '/home/grabbot/grabs/twitter.com-tqbf-2016-03-07-4106f27e/twitter.com-tqbf-2016-03-07-4106f27e-00002.warc.gz-wpullinc'
ERROR [Errno 24] Too many open files: '/home/grabbot/grabs/twitter.com-tqbf-2016-03-07-4106f27e/twitter.com-tqbf-2016-03-07-4106f27e-00002.warc.gz-wpullinc'
Finished grab 4106f27eaff4f6bb12f6afc1f408da09 https://twitter.com/tqbf/ with exit code 3
Output is in directory:
/home/grabbot/grabs/twitter.com-tqbf-2016-03-07-4106f27e
@chfoo chfoo added the bug label Mar 15, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants