This repository holds Collideoscope-specific code that doesn't belong in the main FixMyStreet repository.
This includes the Perl code for processing Stats19 data into a FixMyStreet database, plus all the cobrand code specific to Collideoscope.
In order to import Stats19 data from scratch, you need to run through the following steps. (NB if you are updating Stats19 data skip to step 6b):
-
Check out this repository alongside the FMS repository that runs your site. (Note, if you're using the FMS provided vagrant box, you'll need to edit the
Vagrantfile
in order to share the collideoscope folder with vagrant too.) -
Set up your perl paths so that you can find the collideoscope code from within the FMS environment. For me (in vagrant), it was sufficient to do:
$> export PERL5LIB=/home/vagrant/collideoscope/perllib:$PERL5LIB
-
cd
in to the collideoscope directory. -
Run the
stats19
script from\bin
to download the zipfile of data:$> bin/stats19 download
-
Unzip it:
$> bin/stats19 unzip
-
a) This will create a
vehicles.csv
,casualties.csv
andaccidents.csv
indata/stats19
to complement the existing collection of static data files. Except they won't be called that, as they get named based on the years of data they contain, so rename them to fix that.b) If you're updating a single year of Stats19 data, download the casualties, vehicles, and accidents files for the appropriate year from data.gov.uk and unzip them manually. Then copy or symlink the CSVs into
data/stats19
asvehicles.csv
,casualties.csv
, andaccidents.csv
. -
"Deploy" the data, meaning load it into the interim SQLite database from the collection of CSV files.
$> bin/stats19 deploy
This will take a long time (several hours on my machine) -
Finally, you can import the data into your FMS database:
$> bin/stats19 import
If you're running this locally, soon enough you'll trip over mapit's rate limiting, because the script calls MapIt for each report. You can kill the script at any time and use the data you've got (a few reports are probably enough for development), or you can wait 3 minutes and re-run until you get enough to meet your needs - the script will skip existing reports. Alternatively, connect to the mySociety VPN to lift the MapIt rate limit. -
Don't forget to update the
LATEST_STATS19_UPDATE
constant inSmidsy.pm
.
-
Check out this repo, so it’s alongside your
fixmystreet
repo:cd fixmystreet git clone --recursive git@github.com:mysociety/collideoscope.git ../collideoscope
-
Update your Vagrantfile so that it has access to the
../collideoscope
directory. Once you’re finished, the “synced_folder” lines should look like this:config.vm.synced_folder ".", "/home/vagrant/fixmystreet", :owner => "vagrant", :group => "vagrant" config.vm.synced_folder "../collideoscope", "/home/vagrant/collideoscope", :owner => "vagrant", :group => "vagrant"
-
Then start up your FixMyStreet vagrant VM:
vagrant up && vagrant ssh
-
Inside the vagrant VM, it’s time to set up the new cobrand:
cd /home/vagrant/fixmystreet ../collideoscope/bin/make_po ../collideoscope/bin/create-cobrand-symlinks script/update
-
Then start your FixMyStreet server process as usual. Collideoscope will be available at the
smidsy.<whatever>
subdomain of whatever local development domain you use, eg:smidsy.127.0.0.1.xip.io:3000
. -
If you want to populate your database with some dummy data, run the fixture script from inside the VM:
cd /home/vagrant/fixmystreet ../collideoscope/bin/fixture --empty --commit