DataContractsScraping
title: Data Contracts for Scraping Pipelines radar_quadrant: Techniques radar_ring: Assess radar_position: inner created: 2026-05-22 last_updated: 2026-05-22 related: ["APIFirstArchitecture", "LambdaUnitTesting"]
Data Contracts for Scraping Pipelines is a technique that applies schema validation at the scraper's output boundary using Pydantic models, treating scraped data the same way an external API boundary is treated — with explicit type enforcement that surfaces breakage immediately rather than propagating silent failures downstream.
-
Define the contract first: Write a Pydantic model specifying field names, types, and constraints before writing the scraper (
price: float,url: HttpUrl,in_stock: bool) - Scrape raw data: BeautifulSoup, Playwright, or any extraction library produces unvalidated dicts
-
Validate at the boundary: Each record passes through
Model.model_validate(raw_data)at ingestion; invalid records raise immediately with a precise field-level error - Store only valid data: Only records that pass validation reach the database or downstream pipeline
Without a contract, scraper breakage (site HTML structure changes) manifests as silent None values or type errors discovered hours later in downstream queries. With a contract, the failure is immediate, localised, and actionable: the field that changed, the value that failed, and the record that caused it.
This applies the data contracts discipline — popularised at scale by Uber and LinkedIn between data producer and consumer teams — to single-developer scraping projects with no additional tooling beyond Pydantic.
Data Contracts for Scraping Pipelines sits at Techniques → Assess inner. Pydantic is already standard in the Python ecosystem; the only adoption cost is the upfront discipline of writing the schema before the scraper. Inner position reflects immediate applicability to any Python data ingestion project and zero new dependencies. The technique connects to API-first Architecture (Assess inner) — same principle of defining the interface before the implementation. The remaining gate before Trial is a scraping project where the contract catches at least one real schema drift in production.
