Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
app
lib
test
tooling
.bowerrc
.editorconfig
.gitattributes
.gitignore
.jshintrc
.travis.yml
Gruntfile.js
bower.json
karma-e2e.conf.js
karma.conf.js
package.json
readme.md

readme.md

Why do this?

  • This project is specific to Toronto Open Data
  • Open Data api from the city is super slow, you can't use it to build anything meaningful because it'll probably slow down even further
  • 4+ years worth of data is enough to analyze
  • as a citizen , it is simply interesting and responsible to analyze what our city does and how it performs

DEV Stack

  • .env is used for environment variables (MONGOURL,etc)
  • Front End: AngularJS + Bootstrap
  • NodeJS (hapi for api endpoints ,requests, Promises, Grunt)
  • MongoDB (mongoose)
  • Outbound connection to scrape data from opendata site

Architecture

  • Toronto Open data api is slow , seriously slow, this project provides much faster data access
  • Data for 311 requests is available from 2009 onwards
  • Scraping is possibly by running importdata311.js , for each 25 days (maximum their api provides)
  • AngularJS based ui with bootstrap for data
  • Grunt tasks
    • grunt hapi task will start the api server
    • grunt tasks provide building, checking, running livereload, jslint, etc

Notes about data

  • "status_notes": "In progress - The request is being investigated, assessed and/or responded to; additional work may be required, if applicable.",
  • "status_notes": "Completed - The request has been concluded.",
  • "status_notes": "In progress - The request has been scheduled.",
  • status is open or closed
  • agency_responsible is always 311 Toronto
  • most are "service_name": "Road - Pot hole", or "...... Graffiti Complaint"
  • lat/lng and address can be null - Weird, where did the work occur

ENV stuff

Mandatory:

Model (mongoose) for issues (properly indexes for GEO lookup)


  var IssueSchema = Schema( 
	{ 
		status: String,
		service_request_id: Number,
		status_notes: String,
		service_name: String,
		desc: String,
		agency: String,
		service_notice: String,
		requested_datetime: Date,
		updated_datetime: Date,
		expected_datetime: Date,
		address: String,
		address_id: Number,
		zipcode: String,
		media_url: String,
		loc: {lat: Number,  lng: Number }

	});

Raw Access to Toronto Open data


https://secure.toronto.ca/webwizard/ws/requests.json?start_date=2012-02-01T00:00:00Z&end_date=2012-02-07T00:00:00Z&jurisdiction_id=toronto.ca&status=open

{
	"service_requests":[]
}

RESPONSE OBJECT

 {
"service_requests": [
{
"service_request_id": 101001356540,
"status": "open",
"status_notes": "In progress - The request has been scheduled.",
"service_name": "Road - Pot hole",
"service_code": null,
"description": null,
"agency_responsible": "311 Toronto",
"service_notice": null,
"requested_datetime": "2012-02-05T20:29:00-05:00",
"updated_datetime": null,
"expected_datetime": "2012-02-10T20:30:00-05:00",
"address": "296 BRUNSWICK AVE",
"address_id": 8417961,
"zipcode": null,
"long": -79.407597678,
"lat": 43.665497385,
"media_url": null
},
{
"service_request_id": 101001353098,
"status": "open",
"status_notes": "In progress - The request has been scheduled.",
"service_name": "Road - Pot hole",
"service_code": null,
"description": null,
"agency_responsible": "311 Toronto",
"service_notice": null,
"requested_datetime": "2012-02-02T07:50:00-05:00",
"updated_datetime": null,
"expected_datetime": "2012-02-07T07:50:00-05:00",
"address": "police #86479, toronto (Ms.)",
"address_id": null,
"zipcode": null,
"long": null,
"lat": null,
"media_url": null
}
]
}