Ingest California state bills via Open States

[ThatXliner]

Closes #158.

Billion could query state legislation on demand through the existing Open States client and tRPC router, but never stored it — so CA SB 243 (companion chatbots, chaptered Oct 2025) was invisible in a feed that carried federal bills fine. This adds durable ingestion into the normal Bill pipeline.

Identity

billNumber is "CA SB 243 (2025-2026)", sourceWebsite is openstates.org. Every part is load-bearing: the unique constraint is (billNumber, sourceWebsite), SB 243 exists in most states, and SB 243 exists in every California session as an unrelated bill. Changing the format silently duplicates every stored row.

Bill.congress stays null — it is a federal field, and the session lives in the bill number. Chamber uses the state vocabulary: CA maps lower to Assembly, not House.

Quota

The free Open States tier allows a few hundred requests/day. The walk asks /bills for sponsorships, abstracts, actions and versions inline, so one request returns 20 fully hydrated bills; bill text comes from leginfo and does not touch the quota. At one request per bill a backfill would take weeks — that is what shelved the CourtListener scraper.

--bulk-dir <path> imports an unzipped session-CSV export through the identical normalize→upsert path for a zero-quota backfill, so a backfilled bill and its later incremental refresh are one row rather than two. It takes a local directory rather than downloading, because those archives sit behind a site login, not the API key — automating it would mean storing account credentials in the scraper. The import deliberately does not move the cursor: an export has no position in the update feed.

Otherwise this mirrors the federal walk — ascending updated_since from a durable cursor, retry queue, same three-way upsertContent outcome contract. updated_since is date-granular so the cursor rounds back to its own day; a re-offered unchanged bill is a no-op, whereas rounding forward drops everything updated later that day.

Acceptance criteria

• pnpm --filter @acme/scraper run start open-states --max-items 1 validates env and caps the run
• SB 243 normalizes with correct jurisdiction, session, chamber, sponsor, status, source link, and abstract (fixture test)
• Re-running unchanged is idempotent — content hash gates enrichment
• Incremental runs pick up amended/passed/vetoed/chaptered bills
• State bill IDs cannot collide across states, sessions, or with congress.gov
• Unit tests cover normalization, status mapping, identifier stability, missing optional fields
• Docs no longer describe state-bill ingestion as planned
• Votes are not ingested — Bill has no column for them and adding one is a schema change beyond this issue. The openStates router already serves votes live for the detail screen.

Verification

tsc --noEmit clean on scraper/api/env. 124 scraper tests pass, including the live-query-format regression test. CLI help, env validation, and all three flag guards exercised.

Live verification and remaining limitation:

1. Live API run completed. A real default-tier key fetched CA SB 243, which exposed and fixed the API query format: include must be repeated rather than comma-joined. The bill then completed the full local pipeline with a Big Mac FLUX fallback, and a second run produced zero new entries. The local database contains exactly one bill with its brief, dual lens, and header image.
2. Bulk CSV column names are unverified — the archives are login-gated, so no real export was inspected. The reader maps columns by name with aliases and raises BulkExportShapeError listing the headers it actually found rather than importing shifted data. Expect to adjust the mapping on first real use.

Also note --max-items is per state, not per run. Identical with one state today; worth knowing before adding a second.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
billion-nextjs	Ready	Preview	Aug 14, 2026 12:20am

[ThatXliner]

Rebased onto main (clean, no conflicts) now that #279 has merged and deployed.

Added one commit: state bills now fill Bill.lastActionAt, the column #279 introduced as the sort key for every "recent" listing.

This matters more than it looks. State bills share the feed with federal ones, so a CA bill chaptered last week has to rank against a House bill passed last week — which only holds if both sources fill the column from the same kind of event. Left unset, every state bill would fall back to its introduction date and sit permanently below federal bills that moved the same week. The normalizer already assembles actions in the shape the shared latestActionDate helper reads, so it is a one-line write plus a test asserting SB 243 resolves to its chaptered date (2025-10-13) rather than its introduction.

123 scraper tests pass, tsc --noEmit clean, CI green, mergeable.

Still not merged on purpose. The blocker from the original description stands: OPEN_STATES_API_KEY is not in any local env file, so this has never made a real API call. The normalization is covered by fixture tests against the documented response shape, and the bulk CSV column mapping is written against a format nobody has been able to inspect (the archives are login-gated). Both want one real run before this goes to production — unlike #279, whose behavior I could verify directly against the live database.