Data Flow

Overview

You can regenerate this diagram by pasting the linked code into Excalidraw.

Note: Production is the only environment with the serializer.

Steps

Step 1: Scraping

Scraper code

S3 bucket link: https://minio.scottylabs.org/browser/cmumaps

Data sources:

• OSM Scraper uses the Overpass API to scrape outside graph from OpenStreetMa.

• ESIM Scraper scrapes the CMU Building ArcGIS layer.

• FMS Scraper scrapes the svgs from the CMU FMS website.

  • Backup link: https://www.cmu.edu/enterprise-space/database/index.html

Step 2: Generation

The generator takes in the scraped data and the serialized data as input and generates

• Rescale the svgs to fit in a 1920x1080 rectangle and converts them to floorplans.json file.

• Generates the inside graph from the floorplans.json file.

Step 3: Deserialization

The S3 bucket json files are deserialized locally.

Step 4: Visualization

The visualizer is used to place the data in geo-coordinates. It can also be used to add new data, such as connections between rooms and POIs.

Step 5: Serialization

The updated data are serialized to the S3 bucket.

Step 6: Deployment

The S3 bucket files are deserialized in staging and production.