Skip to content

[Data Source] Build Durham City Council OnBase agenda and legislation scraper #166

Description

@ThatXliner

Summary

Add Durham, NC City Council meetings, work sessions, agendas, items, attachments, minutes, ordinances/resolutions, and action/vote text.

Source assessment

Durham does not appear to use the Legistar API Billion already supports. Its official record is Hyland OnBase Agenda Online, so this needs a source adapter rather than a JURISDICTIONS entry.

Scope

  • Identify the public JSON/XHR requests used by OnBase; avoid brittle rendered-page scraping where possible.
  • Enumerate Council Meetings and Work Sessions by date.
  • Capture body, time/location, agenda status, item number/title, attachments, minutes, action/result, and canonical URL.
  • Parse official PDF/HTML attachments only as needed; retain URL/checksum.
  • Map to the existing local-government meeting/matter/vote contract via an adapter.
  • Add polite concurrency, cache, retries, idempotency, and source-structure versioning.
  • Test one regular meeting, one work session, amended agendas, and missing minutes.
  • Contact the City Clerk only if no stable public request path exists.

Acceptance criteria

  • Backfill 12 months; normalized items include official attachments and minutes/actions when present.
  • Re-runs do not duplicate data; amendments update safely.
  • Every record cites Durham; failures are isolated and actionable.

Outreach Tracker

  • Track: Data Source
  • Stage: Researching
  • Playbook: A — no outreach unless endpoint stability requires clarification
  • Time sensitivity: Medium
  • Next action: capture the network contract for one 2026 regular meeting.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: businessFunding, legal, partnerships, opsarea: scrapersapps/scraper (data source scrapers)enhancementNew feature or requestsourceRelated to news/information sources e.g. scrapers

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions