Skip to content

Tool to scrape City of Vancouver website for rezoning updates & publish to Slack

License

Notifications You must be signed in to change notification settings

rgwood/RezoningScraper

Repository files navigation

RezoningScraper

A small project written for Abundant Housing Vancouver.

Scrapes the City of Vancouver's website for rezoning and development applications, then notifies people of any changes via Slack. It's a simple standalone app using SQLite as a data store, runs on any major OS. Just copy the app (1 file, no dependencies) to a server and run it with a cron job, no further steps needed.

image

image

How to use

Download a binary from the releases page (no external dependencies needed) or build it from source (requires the .NET 6 SDK). Then run it:

❯ .\RezoningScraper.exe --help
RezoningScraper
  A tool to detect new+modified postings on Vancouver's shapeyourcity.ca website.
  Data is stored in a local SQLite database next to the executable.

Usage:
  RezoningScraper [options]

Options:
  --slack-webhook-url <slack-webhook-url>  A Slack Incoming Webhook URL. If specified, RezoningScraper will post info about new+modified rezonings to this
                                           address.
  --save-to-db                             Whether to save the API results to database. [default: True]
  --version                                Show version information
  -?, -h, --help                           Show help and usage information

License

Public domain. Do whatever you like with this code, no attribution needed.

To Do

  • GitHub Actions for build+test
  • --dry-run flag that doesn't update the local DB
  • Add retries to the token fetching. It times out fairly often
  • Some kind of development cache thingy. The API is slowwwww and that makes iterative development painful
  • Discord integration?
  • Twitter integration
  • Display more project fields?
  • This is rough, there's gotta be a better way to query JSON
  • Consider archiving old versions of projects
  • Hook up to Sentry for error reporting
  • Script deployment+setup (just rsync and cron lol) to a remote server
  • Strip line breaks from project titles - CoV does that sometimes and it breaks the Slack link format.
  • Consider excluding published -> archived state transitions from Slack. Lots of noise, not particularly useful

About

Tool to scrape City of Vancouver website for rezoning updates & publish to Slack

Resources

License

Stars

Watchers

Forks

Packages

No packages published