Before going forward with a full database for all the inputs, it's worth putting together a small database for a subset of the inputs, just to test out the approach and the tooling.
This should consist of:
- a set of csv files containing the data
- an in-memory database that reads in the csv files, with all the necessary validation steps. I think we've identified SQLAlchemy as an appropriate tool
- retrieval of data from the database into xarray and/or pandas
I'd probably go with the commodities section of the database to start with as it's relatively self contained.