"air" is a project to visualize live air quality data provided by the Tokyo Metropolitan Government. The main components of the project are:
- a scraper to extract air data from www.kankyo.metro.tokyo.jp
- a postgres database to store the data
- an express.js server to serve this data and other static files to the client
- a client app that interpolates the data and renders an animated wind map
building and launching
Clone the project and install libraries from npm:
Install postgres and create a database, something like:
CREATE DATABASE air WITH OWNER = postgres ENCODING = 'UTF8' TABLESPACE = pg_default LC_COLLATE = 'en_US.UTF-8' LC_CTYPE = 'en_US.UTF-8' CONNECTION LIMIT = -1;
Launch the server:
node server.js <port> <postgres-connection-string> <air-data-url>
node server.js 8080 postgres://postgres:12345@localhost:5432/air <air-data-url>
Finally, point the browser at the server:
Building this project required solutions to some interesting problems. Here are a few:
- Live air data is available as Shift_JIS encoded HTML. Node.js does not natively support Shift_JIS, so the iconv library is used to perform the conversion to UTF-8.
- Geographic data of Tokyo was sourced directly from the Ministry of Land, Infrastructure, Transport and Tourism, as an 80MB XML file. This data was transformed to a 300KB TopoJSON file, small enough for browsers to download and render as SVG with D3.
- Roughly 50 sampling stations provide hourly wind and pollutant data. Inverse Distance Weighting interpolation is used to construct a wind vector field that covers all of Tokyo. IDW produces strange artifacts and is considered obsolete, but it is very simple and was easy to extend to perform vector interpolation.
- The browser interpolates each point (x, y) using the n-closest sampling stations. To determine the n-closest neighbors, the client constructs a k-d tree, which greatly improves the performance.
- Pollutant visualization overlays use Thin Plate Spline interpolation. TPS is definitely the wrong method to use for natural phenomenon such as air pollutants, but it produces a smoother surface than IDW.
- The SVG map of Tokyo is overlaid with an HTML5 Canvas, where the animation is drawn. The animation renderer needs to know where the borders of Tokyo are rendered by the SVG engine, but this pixel-for-pixel information is difficult to obtain directly from the SVG elements. To workaround this problem, Tokyo's polygons are re-rendered to a detached Canvas element, and the Canvas' pixels operate as a mask to distinguish points that lie inside the map to those outside. The field mask occupies the red color channel, and the display mask is encoded in the green color channel.
- I used when.js in the browser because it was a fun experiment.