GitHubSQLitePipeline
title: Git-backed SQLite Data Pipeline radar_quadrant: Techniques radar_ring: Assess radar_position: inner
The Git-backed SQLite data pipeline is a technique for storing time-series scraped data without a dedicated database server. A scraper runs on a schedule via GitHub Actions, writes its output to a SQLite file in the repository, and commits and pushes the updated file back. GitHub serves as the storage layer: version-controlled, zero-cost, with a full history of every change.
The pattern was popularised by Simon Willison, who uses it extensively for personal data journalism — tracking GitHub star counts, news headlines, API snapshots, and similar time-series datasets. The SQLite file can be queried locally with any SQLite client, analysed with DuckDB, or published as a browsable web interface via Datasette deployed to Cloudflare Pages or Vercel.
Placed in Techniques / Assess / inner.
The technique eliminates the three operational concerns that make small-scale scraping impractical: scheduled compute (GitHub Actions free tier), persistent storage (the repo itself), and versioning (every commit is a diff of what changed). The result is a complete data pipeline — schedule, compute, storage, audit trail, and optionally a read interface — with no infrastructure to maintain.
The version-controlled diff property is an underappreciated advantage: if a scraped site changes its schema, the commit history shows exactly when the change happened and what values flipped. This is the Data Contracts for Scraping Pipelines blip applied at the storage layer rather than the schema layer.
Practical constraints: GitHub's 100MB file size limit caps dataset scale; frequently-updated large files accumulate git history that must be managed (periodic squash commits or --force pushes to a data branch). The technique is best suited for datasets that grow slowly or where only the latest snapshot matters.
Inner position reflects zero infrastructure cost, direct applicability to any scraping or API-polling workflow, and clear composability with DuckDB Vector Search (query the file), Scrapy (populate it), and Data Contracts for Scraping Pipelines (validate the schema before writing).
Trial gate: a scheduled GitHub Actions scraper committing to a SQLite file in a repository, with at least one week of history and a verified query against the accumulated data.
