Versions | |
Documentation and Support | |
Open Source | |
Coding Standards | |
Development Status | |
Funding |
The cdm_reader_mapper
toolbox is a python tool designed for both:
- to read data files compliant with a user specified data model
* map observed variables and its associated metadata from a data modelor models combination to the C3S CDS Common Data Model (CDM) format
It was developed to read the IMMA (International Maritime Meteorological Archive) data format, but it has been enhanced to account for meteorological data formats in the case of:
- Data that is stored in a human-readable manner: “ASCII” format.
- Data that is organized in single line reports
- Reports that have a coherent internal structure and can be modelised.
- Reports that have a fixed width or field delimited types
- Reports that can be organized in sections, in which case each section can be of different types (fixed width of delimited)
The cdm_reader_mapper
toolbox is currently NOT working with python 3.11.9
. For more information see dask
Issue #11038 and PR #11035. This issue should be fixed within a newer release.
If you want to contribute, I recommend cloning the repository and installing the package in development mode, e.g.
git clone https://github.com/glamod/cdm_reader_mapper
cd cdm_reader_mapper
pip install -e .
This will install the package but you can still edit it and you don't need the package in your PYTHONPATH
By default, cdm_reader_mapper
outputs logging information to stdout
. To tell cdm_reader_mapper
to output logs to a file, set the CDM_LOG_FILE
environment variable before loading cdm_reader_mapper
.
import os
os.environ["CDM_LOG_FILE"] = "log_file.log"
import cdm_reader_mapper as cdm
This will set the file log_file.log
as the output for all logging information from cdm_reader_mapper
, including the initial logging on loading of the package.
Read imma data with the cdm.read() and copy the data attributes:
import cdm_reader_mapper as cdm
data = cdm.tests.read_imma1_buoys_nosupp()
imma_data = cdm.read(filepath, data_model="imma1", sections=["core", "c1", "c98"])
data_raw = imma_data.data.copy()
attributes = imma_data.attrs.copy()
Map this data to a CDM build for the same deck (in this case deck 704: US Marine Metereological Journal collection of data):
name_of_model = "icoads_r3000_d704"
cdm_dict = cdm.map_model(
name_of_model,
data_raw,
attributes,
cdm_subset=None,
log_level="DEBUG",
)
For more details on how to use the reader
tool see the following jupyter notebooks. For more details on how to use the mapper
tool see the following jupyter notebook.
For a detailed guide on how to build a cdm and write the output of the cdm.map_model() function in ascii see the user guide.