Quality of Life Dashboard v2
A joint project between the City of Charlotte, Mecklenburg County, and UNCC. The first release of the dashboard is now in the v1 branch.
The current (master) version is ready for production, but we decided to add a new calculation at the last minute requiring new data deliverables and code before we launch. You can see a reasonably current verision of the pre-launch site here. Our launch date is looking like May 2015.
The dashboard uses D3 for visualizations. The good news is besides all the cool features in the new version has it is less than half the size over the wire and loads twice as fast. The bad news is IE8 absolutely will not work. I'm sorry/you're welcome.
Here's a handy YouTube Tutorial on customizing the Dashboard for your area of interest if reading isn't your bag.
We hope you find this project useful. Patches are always welcome!
Installing node is a piece of cake. On Windows, download the install file and click Next until it stops asking you to click Next. On MacOS I have no idea, but there's probably a draggy-droppy involved. Any Linux distro will have nodejs in its repos, so do a
sudo pacman -S nodejs or
sudo apt-get install nodejs or whatever package management magic you system employs.
To make sure everything went well here, when you're done installing pull up a terminal (that's DOS for you Windows types) and type
node --version. You should get some like
Install topojson, gulp and bower
npm install -g gulp topojson bower
Because Windows is the only desktop OS that doesn't come with a C compiler, you'll need to get one to get topojson to install. Fortunately the free Express version of Visual Studio works fine. There is more information about this here. Yay Windows.
Clone or download/unzip the project
If you have git installed, just:
git clone https://github.com/tobinbradley/Mecklenburg-County-Quality-of-Life-Dashboard.git
If you don't have git installed, grab the zip file instead. The rest of the commands you see below will need to be run in your project root folder.
Install the node and bower dependencies
We need to install our project dependencies. Those dependencies are specified in package.json and bower.json respectively. We will also run a gulp task to swap in a simple search.
npm install bower install
Swap in the simple search
The default search autocomplete is for Mecklenburg and won't work for anyplace else. This command will swap it out with a simple search for your neighborhood ID's. The advanced search will be renamed search.js.advanced if you want some examples of how to add additional search services.
Fire it up!
The default gulp task starts BrowserSync and launches your current web browser to view the site. Live reload is enabled, so changes will automatically refresh in your browser.
Customizing the Dashboard
Data in the dashboard comes in three pieces:
- The neighborhood geography topojson with your neighborhood id.
- The metric data. You will need 1 to 4 files depending on the calculation and what you want to show.
- The metric metadata.
The first thing you'll need to do is convert your neighborhood layer to topojson. There are a lot of options during conversion, which you can peruse here. You should start out with an Esri shapefile or geojson file projected to WGS84 (EPSG:4326). You can preview, manipulate, and project your data with QGIS.
Here's a good general call to create your topojson file, but do play around with the options if you want more or less line generalization:
topojson -o geography.topo.json -s 7e-11 --id-property=id_field your_shapefile.shp
id_field being the field in the shapefile you want to use for your neighborhood identifier. Copy that file into
dist/data. Make note of what your shapefile was named - you'll need that information when you update
Metric files are simple CSV files named to reflect what they are. The are stored in
src/data/metric. The format is always the same:
y_XXXX representing the year of that data column. Null values are empty, like:
Depending on what you want to do with the metric, you will name your files slightly different things. There are three types of metrics, with the type being set in
config.js. Each type will look for different file names.
- sum: This is for raw variables, like population. It will look for a file name beginning with r, like
- mean: This is for normalized variables, like population density. It will look for a file name beginning with n, like
- weighted: This will normalize data or perform a weighted average and will look for two files - a raw file beginning with r, like
r[X].csv, and a weighting variable beginning with d, like
d[X].csv. Suppose you wanted to have a weighted average for people per acre (population density). If you specify normalize, it'll look for a raw population in
rX.csvand the number of acres for the denominator in
dX.csvand produce a weighted average.
The CSV's are converted to JSON by running
Tip 1: CSV column case sensitivity
Spreadsheet software often likes to capitalize the first letter in the
y_XXXX field. That will turn your life into a furious ball of nothing. You can all your files at once on Linux like so:
for f in *.csv; do sed -e 's/\(.*\)/\L\1/' $f > out/$f; done
Tip 2: Sort on the ID
Your CSV files should be sorted on the metric ID. On Linux to auto do this to all your files:
export LC_NUMERIC=C for f in *.csv; do sort -n $f > out/$f; done
Tip 3: Beware the hanging zero
Some identifiers like Census tracts can have a hanging zero, like 541.10. If you are manipulating your data files in spreadsheet software, make sure your ID column is being treated as a string and not a number or it will get dropped and your life will be destroyed.
Metadata files are in markdown and are named for the metric, like
m1.md, and are located in
src/data/meta. Your metadata file needs to maintain a structure with h2 and h3's laid out like this:
## Title of Metric Median age of poodles ### Whis is this important? Because we like poodles. ### About the Data I hang out at dog parks and type stuff in my phone. Circa 1986. ### Additional Resources Dog pound yo.
When processing the HTML from the markdown, the h2 and h3 tags are used in a really awful lefty-righty kind of way. If you insert another h2 or h3, you will be boned. Same if you drop one. If you want to format the metadata differently, you must edit
src/scripts/functions/metadata.js to meet your needs. It won't be hard, but not doing it and jacking with the markdown will lead to disappointment.
After you add metadata, run
gulp datagen to convert the markdown to HTML.
Tip: Don't add additional H2 (##) or H3 (###) tags
Because we're using those as choppers for layout, adding more of those will screw up your formatting.
Tip: Beware weird, non-web safe characters characters
Don't edit Markdown in Word. You're welcome.
src/scripts/config.js has knobs you will need to turn to set up the dashboard for your area. It is all well documented there. Each metric has a JSON description with a few required and many optional properties.
- metric: The unique metric number.
- type: The calculation performed: sum, mean, or weighted. This effects the files fetched. See the data diagram for more information.
- category: The category of the metric.
- title: Descriptive title (for the select list).
- [optional] accuracy: Set to
Trueif the metric has an accuracy file.
- [optional] label: Metric unit information, like "square miles".
- [optional] decimals: Number of decimals to display (default is 0).
- [optional] prefix: Prefix for the number, like "$".
- [optional] suffix: Suffix for the number, like "%".
- [optional] raw_label: Label for raw number if there is one (also makes it visible).
- [optional] scale: A numeric array of custom data scale breaks, ommitting the top and bottom bounds. Must match the number of color breaks. For example, if you specified 5 color breaks, your array might be
[100, 2000, 4000, 10000]. Non-specified scales get a runtime-computed linear scale.
Note the weighted type doesn't work yet. It doesn't break, it just does the same thing as mean.
Odds and ends
You will have a few additional things to fiddle with, all of which you'll find in either
src/report.html. Different app titles, logos, making some custom charts for your report - that kind of thing. It'll be pretty easy. At the bottom of report.html you will find example chart templates which will require a bit more thought, but all of the chart properties are logically laid out in data attributes on the canvas elements.
Build for deployment
Copy the contents of the
dist folder to your production web server and you're good to go.
There are a series of unit tests for the calculations to make sure the outputs are correct and that nulls are handled appropriately. These execute via QUnit and can be fired via gulp.