Data Pipeline and Active Projects

UN Goal Timelines

Our UN Goal Timeline Development uses Apache eCharts with the Google Data Commons javascript API.

Help us add more timelines Locate in Google UN Data Commons Nav - Click then widen your screen to view navigation for the UN Goals. You can display goal links in our left navigation by turning on "Dev Mode" under Settings in the upper right.

Projects

TO DO: Combine our Earthscape Tabulator with our chartJS timeline display by adding to earthscape.js.

TO DO: Python with UN Goal topics Google Sheet for larger timeline datasets to also be used by RealityStream.

Pre-saving .csv files using python in our CoLabs is easier to collaborate on than Observable Data Loaders.

See our Data Loader notes on Weather API and DuckDB Parquet display sample

Making Updates

Our Development Folders (docs) and Built Output (dist)

After creating your webroot and pulling down data-commons, edit pages and build your static site to update the local "dist" folder.

cd data-commons
yarn build	

Visit the following to view built: http://localhost:8887/data-commons/dist and source http://localhost:8887/data-commons/docs
The "dist" folder is only deployed live by Loren to avoid file conflicts.

UN Sustainable Development Goals

UN Data Commons: SDG Data by Goal The UN Sustainable Development Goals (SDG) data resides in the Google Data Commons API.

Blog: Google Data Commons API for UN Global Goals

About the 17 Global Goals - Latest Goal News - World's Largest Lesson

UN Comtrade Data Pull and Intergovernmental Panel on Climate Change (IPCC) - The UN body for climate change data assessments.

Install and Build Locally

This is an Observable Framework project, however you can avoid building for most contributions.

If you do need to build, run when you start to update dependencies defined in package.json: This is the same as running yarn install

yarn

If you don't have yarn yet, install it with npm install --global yarn You can check if you have yarn with yarn --version

Build the data-commons project locally This pulls from APIs and outputs from "docs" to static files in "dist". You won't be deploying changes to "dist" because numerous merge conflicts occur. Contact Loren to build and deploy your changes to dist.

yarn build

The following is an alternative to using yarn build with yarn dev You can use this to view the content of dist locally at: http://localhost:8887/data-commons/dist

This setup allows you to view multiple repos in one webroot. If you haven't yet, start an http server in your webroot, external to the data-commons folder.

cd ../ && python -m http.server 8887

If you haven't yet, pull down the localsite repo. This provides navigation when you view src files in the "docs" folder.

git clone https://github.com/modelearth/localsite localsite &&
cd data-commons

Then visit the following to view: http://localhost:8887/data-commons/dist http://localhost:8887/data-commons/docs

Turn on GitHub Pages for any repos you fork so we can preview changes at [youraccount].github.io.

For more, see https://observablehq.com/framework/getting-started.

Project structure

In our project, folders for components and data reside in multiple "goal" folders. Add the localsite repo external to the data-commons repo. Your webroot setup with 2 folders:

localsite
data-commons
├─ README.md
├─ docs
│  ├─ space
│  │  ├─ components
│  │  │ 	└─ timeline.js     # an importable module
│  │  ├─ data
│  │  │ 	└─ events.json     # a static data file
│  │  └─ launches.csv.js       # a data loader
│  ├─ jobs
│  ├─ transit
│  ├─ innovation
│  ├─ education
│  ├─ economy
│  ├─ index.html               # a localsite page visible in docs
│  └─ index.md                 # a dist page
├─ dist
├─ observablehq.config.ts      # the project config file
├─ package.json
└─ .gitignore

docs - This is the “source root” — where your source files live. Pages go here. Each page is a Markdown file. Observable Framework uses file-based routing, which means that the name of the file controls where the page is served. You can create as many pages as you like. Use folders to organize your pages.

docs/index.md - This is the home page for your site. You can have as many additional pages as you’d like, but you should always have a home page, too.

docs/data - You can put data loaders or static data files anywhere in your source root, but we recommend putting them here.

docs/components - You can put shared JavaScript modules anywhere in your source root, but we recommend putting them here. This helps you pull code out of Markdown files and into JavaScript modules, making it easier to reuse code across pages, write tests and run linters, and even share code with vanilla web applications.

observablehq.config.ts - This is the project configuration file, such as the pages and sections in the sidebar navigation, and the project’s title.

Command reference

Command	Description
yarn install	Install or reinstall dependencies. Same as just yarn
yarn dev	Start local preview server
yarn build	Build your static site, generating ./dist
yarn deploy	Deploy your project to Observable
yarn clean	Clear the local data loader cache
yarn observable	Run commands like observable help