Roadmap: data engineering 2023

[larsyencken]

↑ Engineering | 2024 →

2023 Q4

Must have

• ⏳ Support data pages in our work
  • Tracking issue: scaling data pages owid-grapher#2379
  • metadata: Discourage the use of display.name in ETL metadata #1903
  • ⏩ Support data pages for our most important metrics #1964

Nice to have

• An admin link to Github to edit or create a YAML metadata file

✅ 2023 Q3

See below

• Use our new data API client-side
  • Decide how to measure site performance
  • ❓ Trial it on a subset of our work
  • Merge change to use API client-side
  • Check how site performance changed
• Kill data_values and backporting
  • Make variables read-only in the admin
  • Insert a warning in the importers repo and warn data managers
  • Migrate all the covid scripts to the ETL
  • Decide on whether to keep a once-off set of backports
  • Turn off backporting and delete backporting code
  • Remove the data_values table
• Support derived charts in visualisation
  • Full export of ETL indicators and DAG into MySQL
  • Support for basic relationships in the ETL and MySQL (see: proposal)

✅ 2023 Q2

See below

• Publish a new API from the ETL
  • For MySQL indicators
• Sunset data import via the Grapher admin owid-grapher#2132
• Prepare for a new data science API
  • 🔨 Remove parquet autogeneration #1252
• Improve indicator visibility and understanding
  • ✨ metadata handling improvements #1251
  • 🔨 Support strict primary key enforcement #1275
  • 🔨 raise strict mode error for non-unique index #1279
  • Update walkthrough to properly set and reset indexes #1283

Not this year

• Full model of the ETL in MySQL
  • MySQL contains the full dag
  • Every MySQL variable has an ETL link
• ❓ Make it easier to see how big data updates are and plan them better
  • ❓ Draft technical catalog explaining indicators and dependencies and downstream usage (e.g. charts)
  • ❓ Create a better dashboard of how datasets are used (in the admin)
• ❗ Admin support for ETL metadata editing
• Update the data science API
  • Create a baking step for the new API
  • Update owid-catalog to support the new API
• Bake new data download options