Skip to content
This repository has been archived by the owner on May 30, 2023. It is now read-only.

PhantomJS 2.1.1 Occasionally but Consistently Crashes Python #14990

Closed
essandess opened this issue May 13, 2017 · 1 comment
Closed

PhantomJS 2.1.1 Occasionally but Consistently Crashes Python #14990

essandess opened this issue May 13, 2017 · 1 comment

Comments

@essandess
Copy link

We use PhantomJS for large web crawls in the project ISP Data Pollution.

We observe intermittent but persistent python crash and even system freeze issues after calling PhantomJS on the order of ten thousand GET calls. Each GET is an independent PhantomJS process called after the previous process has been quit cleanly.

I've wrapped all the calls to phantomjs methods with signal timeouts that should return control to the script if phantomjs hangs.

Here is the Python error message after a crash, caused by a call to the .quit method:

$ ~/bin/isp_data_pollution.py -mm 0
Downloading the blacklist… done.
Display formats:
Downloading: website.com; NNNNN links [in library], H(domain)= B bits [entropy]
Downloaded:  website.com: +LLL/NNNNN links [added], H(domain)= B bits [entropy]

Fatal Python error: Cannot recover from stack overflow.        

Current thread 0x00007fff993a33c0 (most recent call first):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/enum.py", line 228 in __call__
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socket.py", line 88 in _intenum_converter
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socket.py", line 539 in getaddrinfo
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socket.py", line 498 in create_connection
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 871 in connect
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 898 in send
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 963 in _send_output
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1133 in endheaders
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1182 in _send_request
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1137 in request
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 1183 in do_open
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 1211 in http_open
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 442 in _call_chain
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 482 in _open
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 464 in open
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium/webdriver/remote/remote_connection.py", line 489 in _request
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium/webdriver/remote/remote_connection.py", line 415 in execute
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium/webdriver/remote/webdriver.py", line 236 in execute
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium/webdriver/remote/webdriver.py", line 522 in quit
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium/webdriver/phantomjs/webdriver.py", line 71 in quit
  File "/Users/username/bin/isp_data_pollution.py", line 255 in phantomjs_quit
  File "/Users/username/bin/isp_data_pollution.py", line 747 in call_func
  File "/Users/username/bin/isp_data_pollution.py", line 256 in quit_session
  File "/Users/username/bin/isp_data_pollution.py", line 764 in phantomjs_hang_handler
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socket.py", line 507 in create_connection
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 871 in connect
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 898 in send
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 963 in _send_output
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1133 in endheaders
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1182 in _send_request
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1137 in request
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 1183 in do_open
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 1211 in http_open
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 442 in _call_chain
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 482 in _open
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 464 in open
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium/webdriver/remote/remote_connection.py", line 489 in _request
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium/webdriver/remote/remote_connection.py", line 415 in execute
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium/webdriver/remote/webdriver.py", line 236 in execute
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium/webdriver/remote/webdriver.py", line 522 in quit
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium/webdriver/phantomjs/webdriver.py", line 71 in quit
  File "/Users/username/bin/isp_data_pollution.py", line 255 in phantomjs_quit
  File "/Users/username/bin/isp_data_pollution.py", line 747 in call_func
  File "/Users/username/bin/isp_data_pollution.py", line 256 in quit_session
  File "/Users/username/bin/isp_data_pollution.py", line 764 in phantomjs_hang_handler
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/subprocess.py", line 491 in _eintr_retry_call
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/subprocess.py", line 1517 in _try_wait
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/subprocess.py", line 1569 in wait
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium/webdriver/common/service.py", line 163 in stop
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium/webdriver/phantomjs/webdriver.py", line 76 in quit
  File "/Users/username/bin/isp_data_pollution.py", line 255 in phantomjs_quit
  File "/Users/username/bin/isp_data_pollution.py", line 747 in call_func
  File "/Users/username/bin/isp_data_pollution.py", line 256 in quit_session
  File "/Users/username/bin/isp_data_pollution.py", line 764 in phantomjs_hang_handler
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socket.py", line 507 in create_connection
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 871 in connect
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 898 in send
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 963 in _send_output
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1133 in endheaders
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1182 in _send_request
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1137 in request
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 1183 in do_open
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 1211 in http_open
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 442 in _call_chain
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 482 in _open
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 464 in open
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium/webdriver/remote/remote_connection.py", line 489 in _request
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium/webdriver/remote/remote_connection.py", line 415 in execute
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium/webdriver/remote/webdriver.py", line 236 in execute
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium/webdriver/remote/webdriver.py", line 522 in quit
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium/webdriver/phantomjs/webdriver.py", line 71 in quit
  File "/Users/username/bin/isp_data_pollution.py", line 255 in phantomjs_quit
  File "/Users/username/bin/isp_data_pollution.py", line 747 in call_func
  File "/Users/username/bin/isp_data_pollution.py", line 256 in quit_session
  File "/Users/username/bin/isp_data_pollution.py", line 764 in phantomjs_hang_handler
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socket.py", line 507 in create_connection
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 871 in connect
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 898 in send
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 963 in _send_output
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1133 in endheaders
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1182 in _send_request
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1137 in request
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 1183 in do_open
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 1211 in http_open
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 442 in _call_chain
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 482 in _open
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 464 in open
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium/webdriver/remote/remote_connection.py", line 489 in _request
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium/webdriver/remote/remote_connection.py", line 415 in execute
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium/webdriver/remote/webdriver.py", line 236 in execute
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium/webdriver/remote/webdriver.py", line 522 in quit
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/selenium/webdriver/phantomjs/webdriver.py", line 71 in quit
  File "/Users/username/bin/isp_data_pollution.py", line 255 in phantomjs_quit
  File "/Users/username/bin/isp_data_pollution.py", line 747 in call_func
  File "/Users/username/bin/isp_data_pollution.py", line 256 in quit_session
  File "/Users/username/bin/isp_data_pollution.py", line 764 in phantomjs_hang_handler
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socket.py", line 507 in create_connection
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 871 in connect
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 898 in send
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 963 in _send_output
  ...
Abort trap: 6

Also see essandess/isp-data-pollution#26

  1. Which version of PhantomJS are you using? Tip: run phantomjs --version.

2.1.1

  1. What steps will reproduce the problem?

  2. Run isp_data_pollution.py -mm 0 -g for hours to days.

  3. Which operating system are you using?

macOS 10.12.4
Python 3.2

  1. Did you use binary PhantomJS or did you compile it from source?

Binary downloaded from http://phantomjs.org.

  1. Please provide any additional information below.

See above.
@essandess essandess mentioned this issue May 13, 2017
Closed
@essandess
Copy link
Author

Having posted this I see that my own error handler is the culprit in the stack overflow.

I'm closing and will reopen if necessary.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@essandess