Skip to content
Download data from NSW Transport OpenAPI to get stats on delays on the Sydney network
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
notebooks
src
tools
.gitignore
LICENSE
README.md
requirements.txt

README.md

sydney-transport-tracker

Automated scripts for downloading, analysing, and publishing insights about the public transport in Sydney, Australia.

Getting Started

This is currently developed on Ubuntu. The target platform is Linux (Raspbian).

Installing

This requires python3. See requirements.txt for packages.

Connecting to the Open Data API

A .env file must be created in src/data/, containing API keys in the format:

DELAYS_API_KEY=XXXXXX
TIMETABLE_API_KEY=XXXXXX

It is generated from Transport for NSW Open Data API. You will have to register to get the key.

It is recommended that you use separate keys for delays download and timetable download. This is to ensure that they never conflict and call the API too quickly and get rejected. It also allows you to easily track usage from the scripts on the Open Data API dashboard.

Connecting to Twitter

A .env file must be created in src/features/, containing keys in the format:

CONSUMER_KEY=AAAA
CONSUMER_SECRET=BBBB
ACCESS_TOKEN=CCCC
ACCESS_TOKEN_SECRET=DDDD

These are created from the Twitter development platform.

Connecting to Dropbox

Previous day's data can be zipped and uploaded using the scripts in tools/.

Follow the Usage guide for dropbox_upload.sh from the Dropbox-Uploader Github.

Run

chmod +x upload_yesterday.sh
./upload_yesterday.sh 

To zip and upload all of yesterday's downloads to Dropbox. This will delete the zip file, but not the original directory.

Deployment

The target platform is a Raspberry Pi running Raspbian. These steps assume you have cloned this directory to the home directory.

Create .env files as described in the previous section, and setup Dropbox.

Get all packages by running

pip3 install -r requirements.txt

Set up cron jobs by typing

sudo crontab -e

The following task schedule is suggested:

*/2 * * * * python3 /home/pi/sydney-transport-tracker/src/data/train_delay_download.py
1 4 * * * python3 /home/pi/sydney-transport-tracker/src/data/train_timetable_download.py
1 9 * * * python3 /home/pi/sydney-transport-tracker/src/features/tweet_time_period.py 07:00 09:00
31 18 * * * python3 /home/pi/sydney-transport-tracker/src/features/tweet_time_period.py 16:00 18:30
1 1 * * * /home/pi/sydney-transport-tracker/tools/upload_yesterday.sh

This will:

  • Save current delay information every two minutes /data/raw/YYYYmmdd/HHMMSS.pickle.
  • Save timetable information and unzip it once per day to /data/raw/YYYYmmdd/
  • Analyse delays and tweet them for AM and PM peak times
  • Back up yesterday's raw downloaded data at 1:01 each morning

Do not use /etc/rc.local. I find that it sometimes starts the script while pi is still in UTC which will ruin your day.

Acknowledgments

You can’t perform that action at this time.