🎉 Add a new charts-based data API

[larsyencken]

Add a new draft charts-based API for data fetching.

Motivation

We would like to offer simple code snippets that give you the data powering a chart, but we want to prototype what that could look like before committing to baking data in a special way or making a special worker.

The most likely outcome, if we like this code, is that we will later migrate it to a CloudFlare worker and serve the API from there.

A Python charts-based API

from owid.catalog import charts

# get all available charts
slugs = charts.list_charts()

# get a nice data frame, with extra metadata in `df.attrs['metadata']`
df = charts.get_data('life-expectancy')

Technical approach

• For list_charts()
  • Scrape the Charts and explorers page for chart links:
• For get_data()
  • Scrape the JSON config from the chart page
  • Fetch data and metadata for every indicator from the indicator-based data API
  • Throw a LicenseError if the data is not redistributable
  • Make a merged data frame
  • Resolve yearIsDay
  • Stuff indicator metadata into df.attrs['metadata']
  • For single-indicator charts, use chart slug as value column name
• Call a bunch of stuff we want to replace internal, and prefix it all with underscore (e.g. _Indicator) to demonstrate that people shouldn't rely on it

Before merge

• Wait for:
  • Data download restrictions are not properly respected in prod #2631
• Get first round of feedback
• Update docs
• Update changelog

Post merge

• Cut a new owid-catalog release

[owidbot]

Quick links (staging server):

Site	Admin	Wizard

Login: ssh owid@staging-site-catalog-charts

Chart diff:

No new or modified charts. Details

Edited: 2024-05-10 18:20:09 UTC
Execution time: 1.59 seconds

[Marigold]

Very nice! I just had a couple of minor suggestions. In general, it's very user-friendly.