guides scraping with playwright

Scraping with Playwright

The rest of the Guides explain how a single surface gives you away. This group is the other direction: a concrete task, in order, with the code that does it and the specific thing that breaks it.

Every page here is written against a real browser rather than a generic recipe. The launch is a two-line change from stock Playwright, the browser returned is a real Playwright Browser with no wrapped subset to learn, and each how-to carries at least one mistake we made first, measured and fixed rather than assumed.

Start with how to scrape without getting blocked for the model that orders the rest, then pick the task you have.

Get past blocking

• How to scrape without getting blocked - Blocking is five independent layers; fix them cheapest first, looking real beats rotating agents.
• How to avoid bot detection with Playwright - navigator.webdriver patches are level one; a patched engine reaches the rest, with runnable code.
• How to scrape a site that blocks headless browsers - Headless is rarely the block; fix the GPU, font and screen tells instead.
• How to run Playwright in Docker without getting detected - Run in Docker with a real GPU, font set and screen, not six tells.
• How to rotate proxies when scraping with Playwright - A proxy per session, SOCKS5 auth, DNS through the proxy, exit IP matching timezone.
• How to scrape geotargeted content with Playwright - Timezone, locale, number format and geolocation must all agree with the proxy exit IP.
• How to rate limit your own Playwright scraper - Request velocity is a scored signal; throttle with a minimum gap, jitter, concurrency caps.
• Handle 403 and 429 backoff mid-scrape in Playwright - Read 403 and 429 off the response event, honor Retry-After, back off, keep the identity.
• How to retry failed requests when scraping Playwright - Retry with a total time-and-attempt budget; aggressive retries raise a velocity signal.

Page mechanics: waiting, pagination and crawl flow

• How to wait for content to load in Playwright - Why networkidle stalls on long-poll and websockets, and when wait_for_selector or wait_for_function fits.
• Wait for a specific API response in Playwright - Use page.expect_response with a URL predicate, read the JSON at source, and waiting's one limit.
• How to scrape infinite scroll pages with Playwright - A loop that waits for content growth, not a fixed sleep; when to stop, how to dedupe.
• Scrape load-more button pages with Playwright - Re-query the button each round, wait for the count to grow, vary the click timing.
• How to scrape paginated pages with Playwright - Numbered and next-page pagination without the context-destroyed crash: re-query after every page turn.
• Scrape nested pagination with Playwright - Two cursors: keep the outer position across inner navigations, pace the tree, one identity.
• Scrape an SPA that changes URL via history API - history.pushState with no page load: wait on route and DOM markers, capture the view XHR.
• Crawl list pages to detail pages with Playwright - Two phases: collect card links, visit each detail URL, re-associate data, pace the fan-out.
• How to extract links and build a crawl frontier in Playwright - Resolve absolute URLs, strip tracking params, filter same-origin, and dedup with a visited set.
• How to scrape a sitemap.xml with Playwright - Walk the index-to-urlset tree, decompress the .xml.gz leaves, recrawl only changed lastmod URLs.

Forms, widgets and page controls

• Scrape search results by driving a form in Playwright - Fire the input and change events, get past a JS-gated submit, tell results from zero-results.
• Scrape autocomplete and typeahead inputs with Playwright - These inputs ignore fill(); type per character to fire the debounced XHR, wait for the listbox.
• Scrape date-picker calendars with Playwright - Page between months, skip disabled cells, click to commit, sweep a range under one identity.
• Scrape a multi-step wizard flow with Playwright - Complete each step in order, carry the per-step tokens, keep one identity for the whole flow.
• How to handle cookie consent banners in Playwright - The accept button sits in a cross-origin iframe; use frame_locator, not force=True.
• How to handle popups and modals in Playwright - Separate in-page modals, native dialogs and new tabs, and use the right API for each.
• How to scrape iframe content with Playwright - Use frame_locator for same-origin frames; cross-origin iframes fail for a process-isolation reason.
• How to scrape shadow DOM content with Playwright - Locators pierce open shadow roots automatically; closed roots stay unreachable by design.
• Scrape a map-based search with Playwright - Capture the bounding-box marker XHR, pan and zoom, grid the viewport to cover the area.
• How to scrape map-based local results with Playwright - Drive the viewport, capture the bounds-keyed XHR per step, tile an area through an in-region proxy.

Extracting content and structured data

• How to extract JSON-LD structured data with Playwright - Parse JSON-LD and @graph instead of fragile selectors; an empty result reads as a blocked page.
• How to extract Open Graph and meta tags with Playwright - Read the rendered head after JS, resolve relative og:image URLs, apply og:title fallbacks.
• How to scrape HTML tables with Playwright - Pull the whole table in one evaluate_all call, extract before navigating so no rows drop.
• How to extract clean article text with Playwright - Wait for the body, run a readability pass over page.content() to drop nav, ads and chrome.
• How to scrape news article text with Playwright - Pull headline, author and date from JSON-LD, isolate the article node, strip boilerplate.
• Extract data from canvas charts with Playwright - Read the data XHR or the chart library's JS state, not the noised pixels.
• How to capture XHR and API responses in Playwright - Capture JSON with page.on('response') and page.route() instead of parsing HTML.
• How to scrape RSS and Atom feeds with Playwright - Find the feed URL, fetch the XML through the browser, and parse both schemas.

Files, images and media

• How to download files with Playwright - Use expect_download and save_as, keeping the transfer on the browser's proxied exit.
• How to upload files with Playwright - Use set_input_files or expect_file_chooser; the driver fires trusted events, then verify it landed.
• How to download images in bulk with Playwright - Resolve lazy srcset and data-src to full-res URLs, then fetch the bytes inside the page session.
• How to scrape image galleries with Playwright - Scroll to trigger lazy tiles, take the widest srcset, open the lightbox for full-res.
• Scrape lazy-loaded images with Playwright - Read the data-src attribute off the DOM: no scrolling, no downloads. Scroll fallback included.
• How to scrape video listings and metadata with Playwright - Pull exact duration, views and upload date from JSON-LD, fall back to overlays, page the grid.
• How to take full-page screenshots with Playwright - Captured pixels are the true rendered image while the same page's canvas fingerprint is substituted.
• How to generate a PDF with Playwright and Firefox - page.pdf() is Chromium-only; why it is unavailable, plus two capture routes that run on Firefox.

Products, prices and shopping

• How to scrape e-commerce product pages with Playwright - Price and stock live in a variant XHR; select each variant and cross-check the Product JSON-LD.
• How to scrape product reviews with Playwright - Reviews load from a separate endpoint behind a tab; drive the widget, dedupe on review id.
• How to scrape reviews and ratings with Playwright - Read star ratings from aria-label or CSS width, expand Read more, page the load more XHR.
• How to track product prices with Playwright - Wait for the async price widget, keep one fingerprint per item, diff a saved daily time series.
• How to track product stock and restocks with Playwright - Poll the per-variant availability XHR, diff the in-stock boolean, pace to read as one shopper.
• How to scrape deals and coupon codes with Playwright - Codes cloaked behind a reveal button: fire a trusted click, capture from tab, clipboard or XHR.
• How to scrape location-based store prices with Playwright - Drive the store or ZIP picker, verify the location cookie stuck, then align proxy and timezone.
• How to scrape restaurant menu data with Playwright - Read JSON-LD first, fall back to the menu tab XHR, flatten category, size and variant prices.

Listings, travel and live data by vertical

• How to scrape real estate listings with Playwright - Portals re-fetch from a map-bounds XHR and hide price and beds in detail-page JSON-LD.
• How to scrape apartment rentals with Playwright - Trigger the unit-availability XHR, read the per-floorplan price table, key rows on move-in date.
• How to scrape vacation rental listings with Playwright - Drive the date and guest inputs, read the fee-inclusive total from the pricing XHR, iterate windows.
• How to scrape hotel room prices with Playwright - Drive the date and occupancy form, read the rate XHR: rates exist only when you query.
• How to scrape flight prices with Playwright - Wait for the results-complete signal instead of networkidle, then read the settled fare matrix.
• How to scrape flexible-date fare calendars with Playwright - Wait for each month's price XHR before reading the grid, page forward across a seed-stable sweep.
• How to scrape event and ticket listings with Playwright - Wait for the availability XHR, normalise event times to the source timezone, step the calendar.
• How to scrape car listings with Playwright - Drive the faceted filter sidebar, wait on the results XHR, pull VIN and specs, then dedupe.
• How to scrape classifieds listings with Playwright - Align proxy region and timezone, harvest cards from the rotating feed, drive a contact-reveal click.
• How to scrape business directory listings with Playwright - Drive the search form, reveal click-gated phone and email, walk filtered pagination as one visitor.
• How to scrape job postings with Playwright - Drive the faceted search filters, wait for the results XHR, and read the JobPosting JSON-LD.
• How to scrape cryptocurrency prices with Playwright - Read WebSocket frames, not the flickering DOM node, and hold the feed open with a stable identity.
• How to scrape stock and financial data with Playwright - Scrape fundamentals from tables, capture the live quote stream at the WebSocket or polling XHR.
• How to scrape sports scores and stats with Playwright - Read the score feed over a WebSocket, click each stat tab to trigger its XHR, one long session.
• How to scrape forum and community threads with Playwright - Walk the reply tree depth-first, expand collapsed branches over XHR, strip quoted text.
• How to scrape social media profiles with Playwright - Extract each virtualized batch before it scrolls out, parse abbreviated counts, reuse a session.

Output, storage and data cleaning

• How to scrape to CSV with Playwright - Correct delimiter and newline escaping, a UTF-8 BOM spreadsheets read, crash-safe incremental appends.
• How to scrape to JSON Lines with Playwright - Why NDJSON, not one big array, is crash-safe for a long crawl, plus the flush rule.
• How to export scraped data to Excel with Playwright - Export with openpyxl without turning SKUs, barcodes and lot codes into numbers or dates.
• How to scrape into a SQLite database with Playwright - Upsert on a natural key, one transaction per page, idempotent instead of piling up duplicates.
• How to scrape into a pandas DataFrame with Playwright - Feed read_html page.content() for a rendered logged-in table, then fix the dtypes it guesses wrong.
• How to clean scraped prices and dates with Playwright - Parse locale prices and relative dates into typed numbers and UTC timestamps.

Sessions, scheduling and running at scale

• How to scrape data behind a login with Playwright - Log in once with a fixed seed, save storage_state, and reuse it every run.
• Resume an interrupted scrape with Playwright - Write a durable checkpoint, skip completed work, re-validate the boundary, reload the seeded identity.
• Incremental scraping: only new items since last run - A high-water mark that stops at the first already-seen id, handling out-of-order inserts and edits.
• How to use invisible_playwright in Docker - Install in Docker, fetch the patched Firefox at build time, and verify the fingerprint survives.
• Can you run invisible_playwright serverless? - Why the patched-Firefox download weight is the real blocker, and the container or worker pattern.
• Run invisible_playwright headful on a server with Xvfb - Run a real headful window on a headless Linux server with Xvfb, when headless already suffices.
• Run invisible_playwright in GitHub Actions CI - A headless CI recipe caching the engine, and why a datacenter runner IP is the limit.
• Run invisible_playwright in a Jupyter notebook - Why the sync API raises inside a kernel, and the await-in-a-cell fix for interactive sessions.
• Run invisible_playwright in Celery task workers - One browser per worker reused across tasks, the seed passed as a task argument for retries.
• Use invisible_playwright in an Airflow DAG - One browser per task run, the seed stored as a param for reproducible retries.
• Schedule invisible_playwright scrapes with cron - A scrape firing at the same clock minute daily is detectable; jitter the schedule and pacing.
• Wrap invisible_playwright in a FastAPI service - One browser kept alive by lifespan, a semaphore to bound concurrency, the cost of one server IP.
• How to scrape pages in parallel with Playwright - Run many workers with asyncio.gather, one identity per worker; a shared fingerprint is the tell.
• Run invisible_playwright concurrently with asyncio - Bound parallel pages with a Semaphore, give each worker its own seed; concurrency is a speed lever.
• Combine invisible_playwright with httpx for speed - Clear the fingerprint-gated entry, export storage_state, hand cookies to httpx for cheap follow-up.
• Use BeautifulSoup with invisible_playwright - The browser does the fingerprint-real fetch, BeautifulSoup parses page.content(); BS4 does not evade detection.
• Block images to speed up scraping (and when not to) - Blocking images with route.abort() cuts bandwidth, but a no-image waterfall is itself a tell.
• Run stealth Playwright tests with pytest fixtures - Wire pytest-asyncio fixtures: a session-scoped browser, a per-test seeded context, realness assertions.