AJAX Pagination

[Aswath1709]

I’m trying to scrape a website that loads more products via AJAX when I click next. The base URL stays the same , how can I extract all items?

[dosubot[bot]]

ScrapeGraphAI can handle AJAX-loaded content by using browser automation via the ChromiumLoader class, which supports JavaScript rendering and scrolling—both essential for scraping dynamic pages where the URL doesn't change. You can use FetchNode or FetchNodeLevelK with ChromiumLoader to fetch content after all items are loaded. For many sites, enabling scrolling (with the ascrape_playwright_scroll method) will trigger AJAX loading and let you extract all products once the page is fully rendered.

If the site requires clicking a "next" button (not just scrolling), ScrapeGraphAI doesn't have a built-in node for simulating clicks, so you may need to extend ChromiumLoader or use Playwright directly to automate those clicks and collect the content after each interaction.

Make sure Playwright is installed and configure your FetchNode or graph to use JS support and scrolling as needed. Example usage and configuration details for these nodes are in the docs and codebase, but there’s no out-of-the-box solution for generic AJAX pagination via button clicks yet. For most infinite scroll or lazy-load cases, scrolling with ChromiumLoader is the recommended approach (ChromiumLoader details).

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}