Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Got a runtime error while render() #278

Open
Areso opened this issue Mar 16, 2019 · 11 comments
Open

Got a runtime error while render() #278

Areso opened this issue Mar 16, 2019 · 11 comments

Comments

@Areso
Copy link

Areso commented Mar 16, 2019 •
edited by bisguzar 

session = HTMLSession()
r = session.get(url)
r.html.render()
my = r.html.text

My Python version is 3.7.1
Sometimes it's first loop run falls in the error, sometimes it's about tenth. Anyway, it falls with the error:

Traceback (most recent call last):
  File "C:\Software\Python\lib\site-packages\requests_html.py", line 512, in _async_render
    await page.goto(url, options={'timeout': int(timeout * 1000)})
  File "C:\Software\Python\lib\site-packages\pyppeteer\page.py", line 862, in goto
    raise error
pyppeteer.errors.TimeoutError: Navigation Timeout Exceeded: 8000 ms exceeded.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/git-projects/Stock-watcher/stock-watcher.py", line 177, in <module>
    main_loop(ld_params, ld_assets, ld_exchanges, ld_threshold_min, ld_threshold_max, ld_sms_counter_dict)
  File "C:/git-projects/Stock-watcher/stock-watcher.py", line 158, in main_loop
    price = parsing_tradingview(each_asset, loop_exchanges[myiterator])
  File "C:/git-projects/Stock-watcher/stock-watcher.py", line 114, in parsing_tradingview
    r.html.render()
  File "C:\Software\Python\lib\site-packages\requests_html.py", line 598, in render
    content, result, page = self.session.loop.run_until_complete(self._async_render(url=self.url, script=script, sleep=sleep, wait=wait, content=self.html, reload=reload, scrolldown=scrolldown, timeout=timeout, keep_page=keep_page))
  File "C:\Software\Python\lib\asyncio\base_events.py", line 573, in run_until_complete
    return future.result()
  File "C:\Software\Python\lib\site-packages\requests_html.py", line 537, in _async_render
    await page.close()
  File "C:\Software\Python\lib\site-packages\pyppeteer\page.py", line 1458, in close
    raise PageError('Protocol Error: Connectoin Closed. '
pyppeteer.errors.PageError: Protocol Error: Connectoin Closed. Most likely the page has been closed.
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "C:\Software\Python\lib\site-packages\pyppeteer\launcher.py", line 262, in killChrome
    self._cleanup_tmp_user_data_dir()
  File "C:\Software\Python\lib\site-packages\pyppeteer\launcher.py", line 154, in _cleanup_tmp_user_data_dir
    raise IOError('Unable to remove Temporary User Data')
OSError: Unable to remove Temporary User Data
@Areso Areso changed the title Got a random runtime error Got a random runtime error while render() Mar 16, 2019
@Areso Areso changed the title Got a random runtime error while render() Got a runtime error while render() Mar 16, 2019
@bisguzar
Copy link
Contributor

Can you run your script with full rights? Like root or if you are using cmd you can try open terminal as administrator. And please share traceback again when you try.

@Areso
Copy link
Author

Areso commented Mar 17, 2019 •
edited

Did it.

Future exception was never retrieved
future: <Future finished exception=NetworkError('Protocol error Target.sendMessageToTarget: Target closed.')>
pyppeteer.errors.NetworkError: Protocol error Target.sendMessageToTarget: Target closed.
Traceback (most recent call last):
  File "C:\Software\Python\lib\site-packages\requests_html.py", line 512, in _async_render
    await page.goto(url, options={'timeout': int(timeout * 1000)})
  File "C:\Software\Python\lib\site-packages\pyppeteer\page.py", line 862, in goto
    raise error
pyppeteer.errors.TimeoutError: Navigation Timeout Exceeded: 8000 ms exceeded.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "stock-watcher.py", line 177, in <module>
    main_loop(ld_params, ld_assets, ld_exchanges, ld_threshold_min, ld_threshold_max, ld_sms_counter_dict)
  File "stock-watcher.py", line 158, in main_loop
    price = parsing_tradingview(each_asset, loop_exchanges[myiterator])
  File "stock-watcher.py", line 114, in parsing_tradingview
    r.html.render()
  File "C:\Software\Python\lib\site-packages\requests_html.py", line 598, in render
    content, result, page = self.session.loop.run_until_complete(self._async_render(url=self.url, script=script, sleep=sleep, wait=wait, content=self.html, reload=reload, scrolldown=scrolldown, timeout=timeout, keep_page=keep_page))
  File "C:\Software\Python\lib\asyncio\base_events.py", line 573, in run_until_complete
    return future.result()
  File "C:\Software\Python\lib\site-packages\requests_html.py", line 537, in _async_render
    await page.close()
  File "C:\Software\Python\lib\site-packages\pyppeteer\page.py", line 1458, in close
    raise PageError('Protocol Error: Connectoin Closed. '
pyppeteer.errors.PageError: Protocol Error: Connectoin Closed. Most likely the page has been closed.
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "C:\Software\Python\lib\site-packages\pyppeteer\launcher.py", line 262, in killChrome
    self._cleanup_tmp_user_data_dir()
  File "C:\Software\Python\lib\site-packages\pyppeteer\launcher.py", line 154, in _cleanup_tmp_user_data_dir
    raise IOError('Unable to remove Temporary User Data')
OSError: Unable to remove Temporary User Data

@bisguzar
Copy link
Contributor

render() method timeout parameter (check it out from https://html.python-requests.org/). The default value is 8. Lets set it to a bigger one. Can you share the target? We can try by ourselves.

Maybe you can try this solution for OSError.

@Areso
Copy link
Author

Areso commented Mar 17, 2019 •
edited

I did the following

    session = HTMLSession()
    r = session.get(url)
    r.html.render(timeout=30)
    my = r.html.text
    r.close()
    session.close()

Still ran as administrator.
My target to render is https://www.tradingview.com/symbols/NASDAQ-TSLA

Traceback (most recent call last):
  File "stock-watcher.py", line 178, in <module>
    main_loop(ld_params, ld_assets, ld_exchanges, ld_threshold_min, ld_threshold_max, ld_sms_counter_dict)
  File "stock-watcher.py", line 159, in main_loop
    price = parsing_tradingview(each_asset, loop_exchanges[myiterator])
  File "stock-watcher.py", line 134, in parsing_tradingview
    session.close()
  File "C:\Software\Python\lib\site-packages\requests_html.py", line 736, in close
    self.loop.run_until_complete(self._browser.close())
  File "C:\Software\Python\lib\asyncio\base_events.py", line 573, in run_until_complete
    return future.result()
  File "C:\Software\Python\lib\site-packages\pyppeteer\browser.py", line 251, in close
    await self._closeCallback()  # Launcher.killChrome()
  File "C:\Software\Python\lib\site-packages\pyppeteer\launcher.py", line 262, in killChrome
    self._cleanup_tmp_user_data_dir()
  File "C:\Software\Python\lib\site-packages\pyppeteer\launcher.py", line 154, in _cleanup_tmp_user_data_dir
    raise IOError('Unable to remove Temporary User Data')
OSError: Unable to remove Temporary User Data

@Areso
Copy link
Author

Areso commented Mar 17, 2019 •
edited

How to find to which directory it downloaded Chromium? I wanna to try to delete it.

@mrr010
Copy link

mrr010 commented Mar 17, 2019 •
edited

I come across similar issues from rendering http://global.krx.co.kr/contents/GLB/05/0503/0503050600/GLB0503050600.jsp
I also got a window popup shows "Chromium has stopped working"

Traceback (most recent call last):
  File "C:\Users\mrr010\PycharmProjects\Project_LearnPython\venv\lib\site-packages\requests_html.py", line 512, in _async_render
    await page.goto(url, options={'timeout': int(timeout * 1000)})
  File "C:\Users\mrr010\PycharmProjects\Project_LearnPython\venv\lib\site-packages\pyppeteer\page.py", line 862, in goto
    raise error
pyppeteer.errors.TimeoutError: Navigation Timeout Exceeded: 8000 ms exceeded.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/mrr010/PycharmProjects/Project_LearnPython/webscraping.py", line 14, in <module>
    r.html.render()
  File "C:\Users\mrr010\PycharmProjects\Project_LearnPython\venv\lib\site-packages\requests_html.py", line 598, in render
    content, result, page = self.session.loop.run_until_complete(self._async_render(url=self.url, script=script, sleep=sleep, wait=wait, content=self.html, reload=reload, scrolldown=scrolldown, timeout=timeout, keep_page=keep_page))
  File "C:\Users\mrr010\AppData\Local\Programs\Python\Python37\lib\asyncio\base_events.py", line 568, in run_until_complete
    return future.result()
  File "C:\Users\mrr010\PycharmProjects\Project_LearnPython\venv\lib\site-packages\requests_html.py", line 537, in _async_render
    await page.close()
  File "C:\Users\mrr010\PycharmProjects\Project_LearnPython\venv\lib\site-packages\pyppeteer\page.py", line 1458, in close
    raise PageError('Protocol Error: Connectoin Closed. '
pyppeteer.errors.PageError: Protocol Error: Connectoin Closed. Most likely the page has been closed.
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "C:\Users\mrr010\PycharmProjects\Project_LearnPython\venv\lib\site-packages\pyppeteer\launcher.py", line 262, in killChrome
    self._cleanup_tmp_user_data_dir()
  File "C:\Users\mrr010\PycharmProjects\Project_LearnPython\venv\lib\site-packages\pyppeteer\launcher.py", line 154, in _cleanup_tmp_user_data_dir
    raise IOError('Unable to remove Temporary User Data')
OSError: Unable to remove Temporary User Data

then the second time i run the script, the error becomes this:

Traceback (most recent call last):
  File "C:\Users\mrr010\PycharmProjects\Project_LearnPython\venv\lib\site-packages\requests_html.py", line 512, in _async_render
    await page.goto(url, options={'timeout': int(timeout * 1000)})
  File "C:\Users\mrr010\PycharmProjects\Project_LearnPython\venv\lib\site-packages\pyppeteer\page.py", line 862, in goto
    raise error
pyppeteer.errors.TimeoutError: Navigation Timeout Exceeded: 8000 ms exceeded.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/mrr010/PycharmProjects/Project_LearnPython/webscraping.py", line 14, in <module>
    r.html.render()
  File "C:\Users\mrr010\PycharmProjects\Project_LearnPython\venv\lib\site-packages\requests_html.py", line 598, in render
    content, result, page = self.session.loop.run_until_complete(self._async_render(url=self.url, script=script, sleep=sleep, wait=wait, content=self.html, reload=reload, scrolldown=scrolldown, timeout=timeout, keep_page=keep_page))
  File "C:\Users\mrr010\AppData\Local\Programs\Python\Python37\lib\asyncio\base_events.py", line 568, in run_until_complete
    return future.result()
  File "C:\Users\mrr010\PycharmProjects\Project_LearnPython\venv\lib\site-packages\requests_html.py", line 537, in _async_render
    await page.close()
  File "C:\Users\mrr010\PycharmProjects\Project_LearnPython\venv\lib\site-packages\pyppeteer\page.py", line 1458, in close
    raise PageError('Protocol Error: Connectoin Closed. '
pyppeteer.errors.PageError: Protocol Error: Connectoin Closed. Most likely the page has been closed.

@mrr010
Copy link

mrr010 commented Mar 17, 2019

How to find to which directory it downloaded Chromium? I wanna to try to delete it.

C:\Users\Username\AppData\Local\pyppeteer\pyppeteer

i tried deleting the pyppeteer but the error still exisit

@toskip
Copy link

toskip commented Mar 27, 2019

modify pyppeteer/laucher.py, line 102

        if 'headless' not in self.options or self.options.get('headless'):
            self.chrome_args.extend([
                '--headless',
                '--disable-gpu',
                '--hide-scrollbars',
                '--mute-audio',
                #add following two lines
                '--proxy-server="direct://"',
                '--proxy-bypass-list=*'
            ])

I found the solution in puppeteer/puppeteer#2391 and it worked for me

@collinsanele
Copy link

There is definitely a problem with the render() function in requests_html. I always get permissionError: Access is denied.
Other than that, it's a great library

@razak17
Copy link

razak17 commented Mar 29, 2020

modify pyppeteer/laucher.py, line 102

        if 'headless' not in self.options or self.options.get('headless'):
            self.chrome_args.extend([
                '--headless',
                '--disable-gpu',
                '--hide-scrollbars',
                '--mute-audio',
                #add following two lines
                '--proxy-server="direct://"',
                '--proxy-bypass-list=*'
            ])

I found the solution in puppeteer/puppeteer#2391 and it worked for me

This worked for me too

@PhilipWerz
Copy link

modify pyppeteer/laucher.py, line 102

        if 'headless' not in self.options or self.options.get('headless'):
            self.chrome_args.extend([
                '--headless',
                '--disable-gpu',
                '--hide-scrollbars',
                '--mute-audio',
                #add following two lines
                '--proxy-server="direct://"',
                '--proxy-bypass-list=*'
            ])

I found the solution in puppeteer/puppeteer#2391 and it worked for me

I can't find this line in launcher.py... Any idea?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@Areso @toskip @bisguzar @collinsanele @mrr010 @razak17 @PhilipWerz