New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Session closed error when script is delayed more than 20 seconds #175

Open
zxwild opened this Issue Dec 11, 2018 · 3 comments

Comments

Projects
None yet
3 participants
@zxwild
Copy link

zxwild commented Dec 11, 2018

When a script is delayed, the pyppeteer failed with the following error:
Session closed. Most likely the page has been closed.

Here is a simple example how to reproduce it:

  1. Create browser
  2. Select page (one is already created)
  3. Request wikipedia main page
  4. Try to get an element from the page, - all is ok.
  5. Wait for 30 seconds
  6. Try to get the same element from the page and you'll receive:
    Protocol Error (Runtime.callFunctionOn): Session closed. Most likely the page has been closed.

PS: On my machine, with a 19 seconds delay this script works ok, but with 20 and more - fails.

import asyncio
from pyppeteer import launch


async def main():
    browser = await launch({
        'headless': True,
        'args': [
            '--no-sandbox',
        ],
    })

    pages = await browser.pages()
    page = pages[0]
    await page.goto('https://wikipedia.org')

    valid_element = await page.waitForSelector('.central-featured')
    print(f'valid_element pass 1: {valid_element}')

    # Here we wait for 30 seconds, but actually 20 is enough on my machine
    # in other words with a 19 seconds delay this script should work, with 20 - shouldn't
    await asyncio.sleep(30)

    valid_element = await page.waitForSelector('.central-featured')
    print(f'valid_element pass 2: {valid_element}')

    await browser.close()

asyncio.get_event_loop().run_until_complete(main())
@zxwild

This comment has been minimized.

Copy link

zxwild commented Dec 11, 2018

Looks like it's related to websockets, browser._connection._ws, which is connected with default parameters:

ping_interval=20
ping_timeout=20

And somehow keepalive_ping_task doesn't work properly.

PS:
Just noticed #160 at least it helps to increase 20 seconds limit to 15 minutes at least in my case.
More comments are there.

Possible solutions:

  • a fix from #160 with pings removing
  • dropdown websockets to "==6.0" (with some instability it seems)
@Lucifaer

This comment has been minimized.

Copy link

Lucifaer commented Dec 13, 2018

Good job!
I have found this problem too, and I rewrite my js code to python code like this:

async def scroll_page(page):
    cur_dist = 0
    height = await page.evaluate("() => document.body.scrollHeight")
    while True:
        if cur_dist < height:
            await page.evaluate("window.scrollBy(0, 500);")
            await asyncio.sleep(0.1)
            cur_dist += 500
        else:
            break

and the top solution seems works for me, but it doesn't solve the problem at all.
And your solution fundamentally solved the problem. Thanks! 👍

@stolati

This comment has been minimized.

Copy link

stolati commented Dec 20, 2018

For those who want to hack before the patch arrives.

def patch_pyppeteer():
    import pyppeteer.connection
    original_method = pyppeteer.connection.websockets.client.connect

    def new_method(*args, **kwargs):
        kwargs['ping_interval'] = None
        kwargs['ping_timeout'] = None
        return original_method(*args, **kwargs)

    pyppeteer.connection.websockets.client.connect = new_method
patch_pyppeteer()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment