Input data

Every input data set has a corresponding Python module to adjust the unprocessed data provided directly by the upstream data source to the common iTEM structure.

These modules are not invoked directly, but through the function ~historical.process, e.g.

# Process upstream data for data set T009; return the results
process(9)

~historical.process loads and makes use of dataset-specific configuration, checks, and additional code from the corresponding submodule, while automating common cleaning steps. See:

the function documentation for a complete description of these steps.
the green [source] link next to each function (e.g. .T001.process) to access and inspect the source code for the dataset-specific cleaning steps.

This pattern reduces duplicated code in each dataset-specific submodule, while remaining flexible to upstream data formats.

HOWTO add upstream data sources or sets

Add the data set entry to sources.yaml.
Copy, rename, and modify an existing module, e.g. T012.py.
Extend the tests to ensure this data set is tested.
Update the docstrings in the code and this documentation.

Common code

item.historical

item.historical.REGION

item.historical.SOURCES