You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Google Maps scraper fail on many queries (around 10k and more)
Steps to Reproduce
Work machine is Windows Server with 4 gb RAM (it's enough for 16 threads as I test it)
Load many queries (I'm loading just a links to websites)
Run and wait
Actual behavior:
Error:
Failed to save screenshot
Closing Browser
Traceback (most recent call last):
File "C:\Users\Admin\AppData\Local\Programs\Python\Python311\Lib\site-packages\bose\base_task.py", line 192, in run_task
close_driver(driver)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python311\Lib\site-packages\bose\base_task.py", line 181, in close_driver
driver.close()
File "C:\Users\Admin\AppData\Local\Programs\Python\Python311\Lib\site-packages\bose\bose_driver.py", line 335, in close
return super().close()
File "C:\Users\Admin\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium\webdriver\remote\webdriver.py", line 551, in close
self.execute(Command.CLOSE)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium\webdriver\remote\webdriver.py", line 429, in execute
self.error_handler.check_response(response)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 243, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: disconnected: Unable to receive message from renderer
(failed to check if window was closed: disconnected: not connected to DevTools)
(Session info: chrome=116.0.5845.141)
Stacktrace:
GetHandleVerifier [0x005B37C3+48947]
(No symbol) [0x00548551]
(No symbol) [0x0044C92D]
(No symbol) [0x0043E26E]
(No symbol) [0x0043D09F]
(No symbol) [0x0043D678]
(No symbol) [0x0043C695]
(No symbol) [0x00435811]
(No symbol) [0x00435AC4]
(No symbol) [0x0049D688]
(No symbol) [0x00495053]
(No symbol) [0x004716C7]
(No symbol) [0x0047284D]
GetHandleVerifier [0x007FFDF9+2458985]
GetHandleVerifier [0x0084744F+2751423]
GetHandleVerifier [0x00841361+2726609]
GetHandleVerifier [0x00630680+560624]
(No symbol) [0x0055238C]
(No symbol) [0x0054E268]
(No symbol) [0x0054E392]
(No symbol) [0x005410B7]
BaseThreadInitThunk [0x745962C4+36]
RtlSubscribeWnfStateChangeNotification [0x77191B69+1081]
RtlSubscribeWnfStateChangeNotification [0x77191B34+1028]`
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Admin\Desktop\google-maps-scraper-master\main.py", line 19, in <module>
launch_tasks(*tasks_to_be_run)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python311\Lib\site-packages\bose\launch_tasks.py", line 54, in launch_tasks
current_output = task.begin_task(current_data, task_config)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python311\Lib\site-packages\bose\base_task.py", line 219, in begin_task
final = run_task(False, 0)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python311\Lib\site-packages\bose\base_task.py", line 214, in run_task
close_driver(driver)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python311\Lib\site-packages\bose\base_task.py", line 181, in close_driver
driver.close()
File "C:\Users\Admin\AppData\Local\Programs\Python\Python311\Lib\site-packages\bose\bose_driver.py", line 335, in close
return super().close()
File "C:\Users\Admin\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium\webdriver\remote\webdriver.py", line 551, in close
self.execute(Command.CLOSE)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium\webdriver\remote\webdriver.py", line 429, in execute
self.error_handler.check_response(response)
File "C:\Users\Admin\AppData\Local\Programs\Python\Python311\Lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 243, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: disconnected: not connected to DevTools
(failed to check if window was closed: disconnected: not connected to DevTools)
(Session info: chrome=116.0.5845.141)
Stacktrace:
GetHandleVerifier [0x005B37C3+48947]
(No symbol) [0x00548551]
(No symbol) [0x0044C92D]
(No symbol) [0x0043D249]
(No symbol) [0x0043D79A]
(No symbol) [0x0043D738]
(No symbol) [0x004326FD]
(No symbol) [0x00432F8D]
(No symbol) [0x0049D288]
(No symbol) [0x00495053]
(No symbol) [0x004716C7]
(No symbol) [0x0047284D]
GetHandleVerifier [0x007FFDF9+2458985]
GetHandleVerifier [0x0084744F+2751423]
GetHandleVerifier [0x00841361+2726609]
GetHandleVerifier [0x00630680+560624]
(No symbol) [0x0055238C]
(No symbol) [0x0054E268]
(No symbol) [0x0054E392]
(No symbol) [0x005410B7]
BaseThreadInitThunk [0x745962C4+36]
RtlSubscribeWnfStateChangeNotification [0x77191B69+1081]
RtlSubscribeWnfStateChangeNotification [0x77191B34+1028]`
Reproduces how often:
Every time when I start it, but after 30 minutes or more of work
Additional context
My log looks like that:
[7080:3484:0907/152340.096:ERROR:gles2_cmd_decoder_passthrough.cc(946)] ContextResult::k
FatalFailure: fail_if_major_perf_caveat + swiftshader
[7080:3484:0907/152340.107:ERROR:gles2_cmd_decoder_passthrough.cc(946)] ContextResult::k
FatalFailure: fail_if_major_perf_caveat + swiftshader
Done: V and B Le Mans
[7080:3484:0907/152342.742:ERROR:gles2_cmd_decoder_passthrough.cc(946)] ContextResult::k
FatalFailure: fail_if_major_perf_caveat + swiftshader
[7080:3484:0907/152342.762:ERROR:gles2_cmd_decoder_passthrough.cc(946)] ContextResult::k
FatalFailure: fail_if_major_perf_caveat + swiftshader
Done: V and B La Roche Nord
Filtered 5 links from 5.
View written JSON file at output/vandb-fr-in-france.json
View written CSV file at output/vandb-fr-in-france.csv
Closing Browser
Closed Browser
View Final Screenshot at tasks/1112/final.png
View written JSON file at output/all.json
Creating Driver with window_size=1920,1080 and user_agent=Mozilla/5.0 (Windows NT 10.0)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36
DevTools listening on ws://127.0.0.1:63583/devtools/browser/54835e3a-595b-4cce-8ce0-c9d1
f0639475
Launched Browser
[6804:3312:0907/152354.671:ERROR:gles2_cmd_decoder_passthrough.cc(946)] ContextResult::k
FatalFailure: fail_if_major_perf_caveat + swiftshader
[6804:3312:0907/152354.717:ERROR:gles2_cmd_decoder_passthrough.cc(946)] ContextResult::k
FatalFailure: fail_if_major_perf_caveat + swiftshader
Fetched 5 links.
Creating Driver with window_size=1920,1080 and user_agent=Mozilla/5.0 (Windows NT 10.0)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36
The text was updated successfully, but these errors were encountered:
Description
Google Maps scraper fail on many queries (around 10k and more)
Steps to Reproduce
Work machine is Windows Server with 4 gb RAM (it's enough for 16 threads as I test it)
Actual behavior:
Error:
Reproduces how often:
Every time when I start it, but after 30 minutes or more of work
Additional context
My log looks like that:
The text was updated successfully, but these errors were encountered: