Tracking Financial Vulnerability in the UK

This repo contains the code for the data tool Tracking Financial Vulnerability in the UK. This tool allows users to find how financially vulnerable a parliamentary constituency (PC), region, or country is, as measured by the Financial Vulnerability Index (FVI) and the six components of that index. The tool features quarterly data going back to 2017 Q3 and is updated on a quarterly basis.

The feature uses Mapbox to power the map and tippecanoe and some d3 cartography modules to generate the mbtiles for the map.

Data inputs

The following files are sourced as inputs for the tool:

• index_data.csv - a csv containing the values of the FVI and each of the six components for each quarter for each PC, region, country, and the UK as a whole. This file powers all of the charts in the data tool.
• pcs_bboxes.json and regions_bboxes.json - two json files with the bounding boxes for each PC and region/country. This is used in the map to zoom into the selected PC, region or country.
• search_data_v2.csv - a csv mapping UK postcodes to the PC they fall in. This is used for the search functionality. Important: this file is quite large (over 50MB). See Issue #8 filed on GitHub for a better way to implement the location search.

Data processing scripts

1. makeFinalDatasets.R - this script ingests the source files and outputs two geojson files (one for PCs and one for regions/countries) that includes the most recent FVI value, adds a numeric unique ID field to each geography so it can be used in Mapbox, and reprojects the source shapefiles to WGS84 / EPSG:4326, the projection required by Mapbox. It also generates a csv file mapping each geography to its FVI and component values for each quarter which is used in the charts in the tool. Finally, it creates a csv file mapping UK postcodes to PCs that is used for the searchbox.
   • Inputs: parliamentary_constituencies/uk_shape.shp, regions/uk_region.shp, UK_Index_PCs.xls, UK_Index_region.xls, UK_Index.xls, 2021_abb_geocrosswalk/2021_abb_geocrosswalk.csv
   • Outputs: index_data.csv, pcs.geojson, regions.geojson, search_data_v2.csv
2. makeMaptiles.sh - this script generates the mbtiles for Mapbox to ensure the data are available for all of the zoom levels we need. The mbtiles set has the most recent FVI values bound to each geography. It also adds a top-level numeric identifier for each PC or region/country which is needed to allow individual geographies to be highlighted when mousedover on the map.
   • Inputs: pcs.geojson, regions.geojson
   • Outputs: pcs-id.mbtiles, regions-id.mbtiles
3. reshapeGeojsons.py - this script returns a json file with the calculated bounding box for each PC and region/country. This is needed to tell Mapbox where to shift the map to after the user selects a PC or region/country.
   • Inputs: pcs-id.json, regions-id.json
   • Outputs: pcs_bboxes.json, regions_bboxes.json

How to update

1. First run makeFinalDatasets.R from within the \data directory. The outputs generated by this script are taken as inputs to the other data processing script. Save the input xls and csv files, which are provided by the research team, in a subdirectory inside your \data directory called \source. Save the two shapefiles in a subdirectory inside your \data directory called \shapefiles. Within shapefiles, create two subdirectories, one for \parliamentary_constituencies and one for \regions and save the shapefiles in the appropriate location. Also, be sure to update the most_recent date on line 28 with the date of the most recent quarter of data. Although the update process is well oiled, changes in the .xls files might generate errors or NA values –i.e., "Yorkshire and the Humber" instead of "Yorkshire and The Humber." Double-check the data before updating.
2. Run makeMaptiles.sh from within the \data directory. Upload the two .mbtiles files to Mapbox to the tilesets specified in this script. (Note: reshapeGeojsons.py shouldn't need to be rerun unless the boundaries of the PCs, regions or countries has changed.)
3. Politico is hosting a version of this tool using an iframe. If the update is only data-related, there's no need to do anything. Urban Institute and Politico's versions share data and css files. If changes are required in the JS/D3 or in the html, those need to be made in index.html and index_politico.html, and in main.js and politico.js. The index files are almost identical. The only difference is that index_politico.js should add <script src="js/pym.min.js"></script> in line 380 and <script src="js/politico.js"></script> should replace <script src="js/main.js"></script>. Both main.js and politico.js also have minimal differences. The height for the map is calculated differently in both cases (find more about it in line 13 in both scripts) and politico.js includes code to make the pym.js work.
   • In line 246, dispatch.on("resize.map", function(containerWidth) includes containerWidth
   • In line 1648, pymChild.sendHeight() is called
   • In line 1837, var pymChild = new pym.Child({ renderCallback: redraw });
   • In line 1839, the function redraw has to include containerWidth as its only parameter, that should be passed as a parameter in the dispatch call after this.

Hosting the staging version

For clarity and order, host the staging code inside the features/tpm/uk-debt-updates folder. There, create a new folder YYYYMM and clone the repo.