Conditional wait

[AlvinSartorTrityum]

Hi! Thanks for the plugin, it's very useful.

I generally use the PageMethod("wait_for_load_state", "networkidle") to know when a page has done loading, but I encountered a site that keeps doing requests in the background, so I get a timeout error.

On normal playwright I'd handle the situation like this:

  await page.goto(url);
  await Promise.race([
    page.waitForLoadState("networkidle"),
    page.waitForTimeout(25),
  ]);

So to wait for networkidle OR 25 seconds.

Is it possible to do so with scrapy-playwright?

[elacuesta]

Seems like this could be addressed by #54. I'll try to take over that one to add the missing tasks (docs & tests), but I can't make any promises about the timeline. In the meantime, please try the branch directly to see if it solves your problem.

[elacuesta]

#54 was just merged and released as 0.0.19.

I think this situation could be handled with

yield scrapy.Request(
    url="https://example.org",
    meta={
        "playwright": True,
        "playwright_page_goto_kwargs": {
            "wait_until": "networkidle",
            "timeout": 25000,  # milliseconds
        },
    },
)

[AlvinSartorTrityum]

#54 was just merged and released as 0.0.19.

I think this situation could be handled with

yield scrapy.Request(
    url="https://example.org",
    meta={
        "playwright": True,
        "playwright_page_goto_kwargs": {
            "wait_until": "networkidle",
            "timeout": 25000,  # milliseconds
        },
    },
)

Thanks! Exactly what I needed!