Read colocated files made outside of pyaerocom

[lewisblake]

Close #857

The idea here this that while an entire experiment can be set up to only produce json files by setting only_json=True, a specific obs-network can not. In this approach, we define only_json and coldata_dir as keys in a obs config, and assuming both are provided, proceed with the standard ColdataToJsonEngine processing. Outside of pyaerocom, work will need to be done to take colocated data objects that partners provide, and convert them into a format (i.e., with appropriate metadata) that pyaerocom expects.

[codecov]

Codecov Report

Attention: Patch coverage is 36.36364% with 7 lines in your changes are missing coverage. Please review.

Project coverage is 78.62%. Comparing base (9d2cc92) to head (275c996).
Report is 44 commits behind head on main-dev.

Files	Patch %	Lines
pyaerocom/aeroval/experiment_processor.py	22.22%	7 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           main-dev    #1060      +/-   ##
============================================
+ Coverage     78.53%   78.62%   +0.08%     
============================================
  Files           120      125       +5     
  Lines         19939    19975      +36     
============================================
+ Hits          15660    15706      +46     
+ Misses         4279     4269      -10     

Flag	Coverage Δ
unittests	78.62% <36.36%> (+0.08%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[lewisblake]

Example config of usage. For the observation network, you must state that only_json=True and provide a coldata_dir. All netcdf files in that coldata_dir will be processed at once. Hence, to use this feature across multiple observations and networks, they should be broken up into directories by model, similar to the current structure of /coldata.

#!/usr/bin/env python3

import os

# BASE FILTERS
ALTITUDE_FILTER = {"altitude": [0, 1000]}

MODEL_AUX_VARS = {
    "vmrco2": dict(vars_required=["mmrco2"], fun="mmr_to_vmr"),
    "vmrch4": dict(vars_required=["mmrch4"], fun="mmr_to_vmr"),
}
# Setup for models used in analysis
MODELS = { 
    "IFS-Reanalysis-GHG": dict(
        model_id="ECMWF_I1DG",
        obs_vars=["vmrco2", "vmrch4"],
        model_read_aux=MODEL_AUX_VARS,
    ),
}

OBS_GROUNDBASED = {

    "ICOS": dict(
        obs_id="ICOS",
        obs_vars=["vmrco2", "vmrch4"],
        obs_vert_type="Surface",
        only_json = True,
        coldata_dir= "/lustre/storeB/users/lewisb/Python/Evaluations/aeroval/coldata/cams2-82/ICOS/IFS-Reanalysis-GHG"
    ),

}

# Setup for supported satellite evaluations
OBS_SAT = {}

OBS_CFG = {**OBS_GROUNDBASED, **OBS_SAT}

DEFAULT_RESAMPLE_CONSTRAINTS = dict(monthly=dict(daily=21), daily=dict(hourly=18))

CFG = dict(
    model_cfg=MODELS,
    obs_cfg=OBS_CFG,
    json_basedir=os.path.abspath("../../data"),
    coldata_basedir=os.path.abspath("../../coldata"),
    io_aux_file=os.path.abspath("../eval_py/gridded_io_aux.py"),
    reanalyse_existing=True,
    only_json=False,
    add_model_maps=True,
    only_model_maps=False,
    clear_existing_json=False,
    raise_exceptions=False,
    # Regional filter for analysis
    filter_name="ALL-wMOUNTAINS",
    # colocation frequency (no statistics in higher resolution can be computed)
    ts_type="monthly",
    map_zoom="World",
    freqs=["daily", "monthly"],
    periods=[
        "2018-2021",
        "2018",
        "2019",
        "2020",
        "2021",
    ],
    main_freq="monthly",
    stats_tseries_base_freq="daily",
    zeros_to_nan=False,
    min_num_obs=DEFAULT_RESAMPLE_CONSTRAINTS,
    colocate_time=False,
    obs_remove_outliers=True,
    model_remove_outliers=False,
    harmonise_units=True,
    regions_how="htap",
    annual_stats_constrained=False,
    proj_id="testing",
    exp_id="fix857",
    exp_name="Read colocated files",
    exp_descr=("Read colocated files #857"),
    exp_pi="Lewis",
    public=True,
    # directory where colocated data files are supposed to be stored
    weighted_stats=True,
    var_order_menu=["vmrco2", "vmrch4"],
)

if __name__ == "__main__":

    from pyaerocom.aeroval import EvalSetup, ExperimentProcessor

    stp = EvalSetup(**CFG)

    ana = ExperimentProcessor(stp)

    ana.exp_output.delete_experiment_data(also_coldata=True)
    res = ana.run()

[jgriesfeller]

You might want to correct the typo I mentioned.
Otherwise ready to merge.