Does spider support click?

[squirrelfish]

Some websites don't have direct URLs for their target sections. You gotta click through the entry URLs to get there.

[yetval]

Hey! Yep, this works. Use a browser session and pass a page_action on your Request. It runs after the page loads, so you can click your way from the entry URL to the section you actually want, then Scrapling reads the resulting HTML for you.

from scrapling.spiders import Spider, Request, Response
from scrapling.fetchers import AsyncDynamicSession


async def click_through(page):
    await page.click("a.entry-link")
    await page.wait_for_load_state("networkidle")


class MySpider(Spider):
    name = "clicker"
    start_urls = ["https://example.com/entry"]

    def configure_sessions(self, manager):
        manager.add("browser", AsyncDynamicSession(headless=True))

    async def parse(self, response: Response):
        yield Request(
            response.url,
            sid="browser",
            page_action=click_through,
            callback=self.parse_target,
        )

    async def parse_target(self, response: Response):
        yield {"title": response.css("h1::text").get()}

Works the same whether the click loads a new page or just swaps content in place. Chain as many clicks or waits as you need inside page_action. Hope it helps!

[squirrelfish]

This is ok, but I have a new question, how do I click to turn the page ( js onclick, no direct url)

[yetval]

Yeah, for onclick pagination you need a browser fetcher (StealthyFetcher or DynamicFetcher). Plain Fetcher is HTTP only so it can't fire the click.

Pass a page_action function. It runs after the page loads and gives you a Playwright page to click around with.

from scrapling.fetchers import StealthyFetcher

def click_next(page):
    page.locator("a.next-page").click()
    page.wait_for_load_state("networkidle")

page = StealthyFetcher.fetch(url, headless=True, page_action=click_next)
items = page.css("div.item")

For more than one page, loop inside the function (it only runs once per fetch) and collect the HTML each round:

htmls = []
def paginate(page):
    for _ in range(10):
        page.wait_for_selector("div.item")
        htmls.append(page.content())
        nxt = page.locator("a.next-page")
        if not nxt.count() or not nxt.is_visible():
            break
        nxt.click()
        page.wait_for_load_state("networkidle")

StealthyFetcher.fetch(url, headless=True, page_action=paginate)

If the site just swaps content without a real navigation, replace wait_for_load_state with page.wait_for_selector(...) on something new that appears.

[yetval]

Hope this helps!