CHANGELOG.md

Changelog

All notable changes to OpenGHG will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Unreleased

Fixed

• Ensured explicitly integrated CO2 site and satellite footprints are standardised and modelled through the integrated-footprint pathway, while preserving the time-resolved default for CO2 footprints. PR #1698
• Added a dimension dtype checker for H_back dimension to "timedelta64[ns]". Bug occured for dimension "resolution" and dtype "resolution".PR #1671
• Added config-driven metadata key handling for transformed flux and boundary condition data: removed hard-coded required-key lookups.PR #1686

Updated

• Removed file and retrieved-data hashing from standardisation. Repeated inputs now follow the normal if_exists overlap policy, and data can be standardised again after its datasource is deleted. The deprecated force argument remains accepted but is ignored; passing force=True emits a DeprecationWarning, and callers should use if_exists to choose update behaviour. Issue #1676
• Updated the standardisation of AGAGE format data so that rows containing nans are no longer dropped. PR #1634
• Updated copilot instructions and added repo graph using graphify.PR #1684

Added

• Added support for passing in-memory xarray.Dataset objects to supported standardisation and transformation parsers, including object-store retrieval and forward ModelScenario coverage.
• Added a tutorial for adding CO2 satellite data, including how integrated footprints differ from time-resolved footprints in OpenGHG. PR #1698
• Added ability to save output of plot_comparison using save_path and ability to pass additional figure modifications.PR #1672
• Added more tests and explanation in the tutorial regarding if_exists behaviour.PR #1693

[0.19.0] - 2026-06-11

Fixed

• Converted negative surface observation uncertainty values to NaN before optional averaging. PR #1544
• Updated the value of atol and removed rtol from check_coord_alignment to process 6km file. PR #1588
• Fixed EDGAR parsing for monthly sectoral files by normalising latitude/longitude coordinates to lat/lon and raising a clear error when the EDGAR version cannot be inferred from the filename. PR #1187
• Updated CAMS boundary-condition transforms to support CO2 vertical coordinates via height_above_reference_ellipsoid, pass through the requested domain during interpolation, and correctly parse CAMS filenames with input-observation suffixes such as surface_inst. PR #1601
• Fixed unit parsing ambiguities by keeping the Pint registry case-sensitive for SI abbreviations and adding explicit case-variant aliases for coordinate and time units (e.g. Degrees_North, Hours). PR #1599
• Fixed convert_to_slice to use abs(input) when computing the relative tolerance range, ensuring that negative inlet values (eg. for sites below sea level) are correctly matched during data retrieval. PR #1605
• Fixed "xarray fails to decode time" by using pandas datetime conversion and storing as np.datetime64[ns].PR #1608
• Clarified datasource update/versioning behavior for if_exists, save_current, and overlap handling, including fixes for copied-version metadata and non-overlapping combine updates. PR #1614
• Fixed handling of irregular fp time reindexing and missing "calibration_scale", also added the ability to detect "mf_mod_high_res" for plot_comparison.PR #1611
• Fixed object store search and retrieve results to include datasource-managed metadata without mutating datasource records, while keeping raw metastore descriptor metadata authoritative. PR #1652

Updated

• Refactored Datasource to use VersionedZarrStore directly, removed LocalZarrStore and openghg.store.storage, and moved storage utilities into openghg.storage. PR #1618
• Updated dependency versions: black (26.5.1), mypy (2.1.0), pre-commit (4.6.0), filelock (3.29.0), h5netcdf (1.8.1), msgpack-types (0.7.0), rapidfuzz (3.14.5), nbsphinx (0.9.8), sphinxcontrib-bibtex (2.7.0), types-paramiko, types-requests (2.33.0.20260518). Also added a change in dependabot.yml to group all major package version changes into one PR. PR #1636

Added

• Option to set location of openghg log via an environment variable OPENGHG_LOG_PATH. PR #1607
• Added the ability to standardise GEMINI data using standardise_column.PR #1501

[0.18.0] - 2026-02-18

Added

• Added feature to allow meteorological data in relation to a site to be retrieved from the Copernicus Climate Data Store (CDS) and stored in an object store (data_type="site_met"). PR #1421
• Implemented update method for Zarr storage, which will be used by Datasource to allow updating existing data in the object store. Also added methods for aligning a dataset to data stored by Zarr. PR #1532
• Added version parameter to all get_* retrieve functions (get_obs_surface, get_obs_column, get_flux, get_bc, get_footprint) to enable retrieval of specific data versions instead of always defaulting to the latest version. PR #1549
• Implemented if_exists = "combine" option for adding data to Datasource. This option will now be available when standardising data, so data can be updated. PR #1548
• Added the ability to standardise and retrieve 6km footprint data with ability to pass inner_domain. PR #1580
• Progress towards PR #1447
• Adds search methods to create Sparql queries to search the ICOS Carbon Portal and returns results as Pandas dataframes. PR #1447
• Adds methods for directly downloading ICOS text files for ATC time series and ATC flask time series as well as downloading netCDT time series. PR #1447

Updated

• BaseStore.clear_datasources method is removed because this was incomplete and the functionality should be covered by the DataManager class instead. PR #1533
• Tidied Datasource by moving date range methods to util and reorganising methods. PR #1545
• Updated clean_string function to retain "." characters by default PR #1546
• Search for "inlet" (and other related keys) is updated to always look for a slice even when single values are specified. This is to make sure the numerical value, rather than the string equivalent of the value, is matched and for this reason the default tolerance for the slice is set to be small (1e-6 relative to the value itself). This allows a search of e.g. inlet="10m" to be matched to stored metadata={"inlet": "10.0m", ...}. PR #1558
• Updated pip installation, build and publishing steps to use uv. Added dynamic versioning ability using tags. PR #1498

Fixed

• Fixed ValueError when searching for footprints with special inlet keywords like "column" (for satellite data) or "multiple" (for aggregated data). The convert_to_slice function now passes through these special keywords unchanged instead of attempting to extract float values.
• Updated the version of black formatter in pre-commit-config.yaml as well ran the formatter on the repo. PR #1567
• Units in attributes of variables of cams bc data in transform function.PR #1535

[0.17.0] - 2025-11-15

Added

• Added transform function for boundary conditions along with parser to standardise CAMS N2O and CH4 boundary conditions.PR #1496
• Added new submodule openghg.storage with classes for storing xr.Datasets. These classes will be used to reimplement LocalZarrStore. PR #1391
• Check for negative uncertainty values (these are converted to NaN). PR #1480

Updated

• Refactored individual read_file methods on data_type classes in openghg.store to use a central read_file method on BaseStore parent class (openghg.store.base). This centralises the read_file functionality and ensures the steps for each subclass are consistent. Each data_type class now requires a format_inputs method to apply the formatting steps to the input kwargs when standardising. PR #1415
• Allow a data_type to be specified when defining the required attributes. This is a placeholder at the moment as only "surface" details are included but this allows required attributes to specified and checked for other data_types as well. PR #1443
• Removed pinning of icoscp from 0.17.0 and adding details of how to use the new authentication method to the tutorials. Note this also required explicit inclusion of numpy>=2.0 otherwise this get downgraded to numpy<2.0 based on icoscp current stated requirements. PR #1447
• Added chunking_schema for Flux data type to make sure the chunks created a < maximum size accepted by Codec (Codec does not support buffers of > 2147483647 bytes). PR #1434
• Added version pins for key dependencies (numpy, pandas, xarray, scipy, matplotlib, netcdf4, h5netcdf, tinydb, toml, rich, msgpack) to enable better management through dependabot. Upper bounds use <= with latest stable versions rather than < with next major version. Also added explicit pin for h5py < 3.15 to resolve Python 3.10 test failures. PR #1493
• Refactored parse_agage function to integrate instrument metadata validation and species label definition logic within the get_dataset implementation in agage.py. PR #1510
• Updated parse_openghg for column data to recognise "data_owner" and "data_owner_email" from the file. PR #1519
• Refactored LocalZarrStore to use new VersionedZarrStore class from openghg.storage submodule. PR #1468

Fixed

• Bug causing regrid_uniform_cc to fail for data arrays with time dimension. PR #1482
• Typo that caused chunking to be skipped in most cases. PR #1467
• Bug causing error due to resampling non-numeric data. PR #1478
• Fixed bug in merge_and_extend_dict to make sure repeated values in left and right produce unique values in output and to ensure left and right are not modified in place. PR #1477
• Fixed retrieve_all() function where the value to fetch latest version was overridden in the loop. PR #1522

[0.16.0] - 2025-08-29

Added

• Support for TCCON data: implemented a dedicated parser and integrated with standardise_footprint, get_obs_column, and related workflow steps. Renamed _surface_obs_resampler_dict to _obs_resampler_dict in openghg/data_processing/_resampling.py for generalisation beyond surface data. [PR #1407] (#1407)
• The standardise_flux, standardise_bc, standardise_eulerian and standardise_column can now all accept a list of input netcdf files for filepath (rather than just a single file). This pre-processes the data and concatenates the files when opening them. PR #1393
• retrieve method to ObjectStore and related methods that allow metadata from the metastore and datasources to be combined on retrieval. PR #1408
• Ability to pass the non configured path to the store argument directly for get_* functions. Also added the ability add the store to config using openghg --register-store command. PR #1389
• Add basic schema for EulerianModel data type. This currently checks appropriate coordinates and types are included. PR #1414
• Unit tracking with pint to ModelScenario. PR #1417
• Schema for ObsColumn data type. This currently checks appropriate data variables (time, column, averaging kernel and apriori values) are included of appropriate types. At the moment, this also expects a vertical dimension of "lev". PR #1409

Updated

• Updated temporary path creation to have user specific folder.PR #1396
• Removed unused code from Datasource and LocalZarrStore and changed the name of LocalZarrStore's update method to overwrite. See the PR for full details of code removed.PR #1404
• Allow precision_filepath to be passed directly when specifying the "gcwerks" format for standardise_surface. Previous method of passing a tuple for filepath is still supported but will be deprecated. PR #1405
• Updated workflow of pypi release to follow trusted publisher management from pypi.PR #1402
• Allowed mf_variability to be read in from AGAGE-style obs files. PR #1416

Fixed

• Bug with chunking when standardising PARIS and FLEXPART CO2 footprints. PR #1399
• Fixed bug to process deltao2n2 flask level 2 corso data. PR #1394
• Typo in variability calculation causing the wrong values to be filled. PR #1424
• Issue with importlib.resources.files and editable installs. PR #1429
• Added minimum version of xarray of 2025.04.0 due to changes put in within PR #1417 (use of AlignmentError from xarray package) and flox dependency. PR #1440
• Added section on standardising column satellite data in the tutorials. Added references to missing standardise_column and parse_tccon api in doc. PR#1516

[0.15.0] - 2025-07-02

Added

• Allow to provide a kwargs dict to resampler function via drop_na that will be used by xarray.Dataset.dropna PR #1314
• Improved the check for nans in surface_obs_resampler function: drop data for times where any of f"{species}" or "inlet" variables are nan PR #1314. This removed the improvements of surface_obs_resampler from PR #1298.
• Fews fix for footprints standardisation : add unify_chunks in openghg/store/storage/_localzarrstore.py to prevent chunking errors while checking if chunks match; change name of function check_function_open_nc to footprint_open_nc_fn in openghg/standardise/footprints/_acrg_org.py; rewrite footprint_open_nc_fn to allow selection of the month (for badly formatted footprints) and footprints (same reason); and add a "release_height" variable to old format footprints (format="acrg_org") for compatibility with new footprint format (format="paris") PR #1287
• Option to compute modelled obs (and "fp x flux") by flux sector/source in ModelScenario.footprints_data_merge. PR #1330
• Option to return "fp x flux" from ModelScenario.footprints_data_merge. PR #1328
• Function to compute baseline sensitivities for NESW. This is used in calc_modelled_baseline and will be useful for OpenGHG inversions. PR #1326
• Added support for converting calibration_scale before plotting in the plot_timeseries function.PR #1361
• Method to update attributes of stored data. PR #1375
• Added "tag" keyword as an option when standardising data. This allows a list of user-specified tags to be included. This allows users to search and connect data which includes the chosen tags. PR #1354
• ObjectStore class, which manages access to the metastore and to datasources. PR #1379

Updated

• Updated ModelScenario to work with the new PARIS footprint format for time-resolved footprints. PR #1324
• Updated the package release pyproject.toml and removed the setup.py to make sure PEP621 is followed. PR #1345
• Updated '_scale_convert' to 'convert' function from openghg_calscales package. PR #1349
• Renamed optional_metadata to info_metadata within standardise_* functions so this is more descriptive of how these keys are currently used PR #1377

Fixed

• Added unit of xch4 data var as units attribute to mf inside get_obs_column. PR #1360
• Added missing reference to mf_mod while plotting in the tutorial.PR #1365
• Made call to .load in combine_datasets optional. PR #1371
• Fixed bug where force keyword was not being used correctly for standardise_surface and wasn't allowing the same data to be added again. PR #1374

[0.14.0] - 2025-04-16

Added

• Added new tutorial for satellite ModelScenario processing.PR #1304
• Improved the check for nans in surface_obs_resampler function: drop data for times where any of f"{species}" or "inlet" variables are nan, or when both f"{species}_variability" and f"{species}_repeatability" are nans.PR #1298
• Added a "keep_variables" parameter in get_obs_surface to choose which variables we want to keep when retrieving data. This can be use to prevent resampling functions to try to resample unused variables filled with nans or string PR #1283
• Added a new resampling feature for obs where a f"{species}_variability" variable is present but not f"{species}_number_of_observation" PR #1275
• Added ability to retrieve ICOS combined Obspack .nc data. PR #1212
• Added ability to process ModelScenario for Observation and Footprint satellite data. Added platform keyword to split the process and added ability to pass satellite as argument.#PR 1244
• The platform keyword can now be used with surface data and can be passed to the standardise_surface function (e.g. "surface-insitu", "surface-flask"). This can be used to (a) separate data into different datasources based on platform when storing and (b) when deciding whether to resample data when aligning using ModelScenario methods. PR #1278, PR #1279 and PR #1289.
• Added ability to reindex footprint data to obs data with tolerance of 1ms with method="nearest".#PR 1264
• Added ability to standardise CORSO radiocarbon data, added new parser named parse_icos_corso to handle data modifications.PR #1285
• tox testing setup. #PR 1268
• Added abilty to parse reformatted NAME co2 footprints (PARIS format) to 'paris.py' PR #1319

Updated

• For new object stores, a config file copied into this by default. If no config file is detected the internal defaults for the config are used instead. A custom config file can still be created as needed. PR #1260

Fixed

• Possible circular import due to get_metakeys; the metakey config functionality was moved to the store module. PR #1318
• Changed type definition from xr.Coordinates to xarray.core.coordinates.PR #1316
• Bugs of resampling functions : delete all variables in the obs data that are filled of nan, test the emptiness of the dataset, and delete "flag" variable (removed in [PR #1283] #1283), all that before resampling to prevent errors PR #1275
• Fixed bug where period="varies" could not be used or set when determining the time period associated with the input data. #PR 1259 and PR #1267
• Dropped exposure_id variable for GOSAT data to avoid change in dimension size error raised from to_zarr. PR #1243 PR #1257
• Drop id coordinate for GOSAT data to avoid merging errors PR #1257
• Fixed bugs in ModelScenario for satellite data e.g. requiring max_level as argument #PR 1261
• Fixed get_* functions if passed with start_date or end_date in format of ""dd:mm:yy T 00:00:0000" can still fetch the relevant data.PR #1273
• Fixed numcodecs version to be less than 0.16 to avoid ci runner failing while importing zarr.PR #1296

[0.13.0] - 2025-03-10

Added

• New datapack submodule to allow output obspacks to be created. This includes the create_obspack function which takes an input search file and produces an obspack within a defined structure from this. PR #1117

Updated

• Unpinned numpy so that we can now use numpy 2.0. PR #1235
• When combining obs and footprint data in ModelScenario, allow resample_to to be set to None so that the data is aligned but not resampled. This is also turned on by default by passing the platform keyword and setting to any name which contains "flask" to the relevant ModelScenario methods. PR #1236
• Extracted align_obs_and_other from ModelScenario.align_obs_footprint into analyse._alignment. PR #1234

Fixed

• Bug where attributes were not preserved during some resampling operations. PR #1233

[0.12.0] - 2025-02-27

Updated

• Update standardise_column inputs to include more explicit keywords around selection of satellite points. This includes adding the obs_region keyword to describe an area selected for satellite points (not necessarily the same as domain) and updating the definition of selection to be linked to any additional selection filters included for the satellite data. #PR 1217
• Update standardise_footprint inputs to include more explicit keywords around selection of satellite points. This includes adding the obs_region keyword to describe an area selected for satellite points (not necessarily the same as domain) and updating the definition of selection to be linked to any additional selection filters included for the satellite data. #PR 1218
• Output of parsers changed from nested dictionary to list of MetadataAndData objects. PR #1199

Added

• Added parser to process and add "NIWA" network data to the object store. PR #1208
• Improved resampling of variability when number of observations is present. Also added methods for customising resampling, and a Registry class to "register" functions. PR #1156
• Allow parsers to return a list of MetadataAndData directly. PR #1222

Fixed

• Changed icos_data_level to data_level in ObsSurface.store_data to fix bug where ICOS data was not distinguished by data level. PR #1211
• Fixed permissions for file locks PR #1221

[0.11.1] - 2025-01-10

Fixed

• Added align_metadata_attributes to retrieve_remote and shifted function defination to standardise/meta. PR #1197
• Added icos flags to handle data that is flagged bad for remote icos data.PR #1200
• Pinned Zarr to 2.18.3 as github runners are picking zarr 3.0 which is still in significant development state.PR #1205
• Added exist_ok = true argument to create_config_folder and removed ObjectStoreError call.PR #1198

[0.11.0] - 2024-12-16

Fixed

• Fixed options used with xr.Dataset.to_zarr in reponse to updates in xarray. PR #1160
• Added xfail for cfchecker tests due to broken link. PR #1178
• Removed duplicate code from read_file method in BoundaryConditions and EulerianModel. PR #1192

Added

• Check for file lock permissions with helpful error message. File locks are now created with rw permissions for user and group. PR #1168
• Removed parsers that are unused. PR #1129
• Added data_owner and inlet_height_magl as attributes to parse_icos. PR #1147
• Align the dataset(s) when opening the data while standardising footprints data to prevent error due to misalign coordinates. PR #1164
• Moved sync_surface_metadata to ObsSurface.read_file function so this is applied for all input data regardless of source_format. PR #1138
• Added parse_co2_games parser splitting multiple model from one file into separate datasources. PR #1170
• Added dobj_url as attribute to icos retrieve_atmospheric function. PR #1174
• Added parser for BoundaryConditions class. PR #1180
• Added parser for Eulerian Model class. PR #1181

Updated

• Removed serverless/cloud code since it is not being used. PR #1177
• Minimum version of python to 3.10. PR #1175
• Updated ICOS standardise function to reflect changes in ASCII file format. PR #1140
• Added rename_vars option to get_obs_surface to allow variable names based around species to be returned. PR #1130
• Added option to get_obs_column to return the column data directly rather than converting to mole fractions. PR #1131
• Made calculations in ModelScenario._calc_modelled_obs_HiTRes more efficient. PR #1062
• Updated type hints and typing imports for Python 3.10 PR #1193

[0.10.1] - 2024-09-27

Fixed

• Bug where search_surface couldn't accept a dictionary argument for data_level. PR #1133
• GIT_TAG variable passed to build step of release_conda and also environment activation is executed for publishing to conda step. PR #1135

[0.10.0] - 2024-09-24

Updated

• Updated parse_* functions for surface data type to accept filepath rather than data_filepath. This maps better to the standardise_surface input and makes this consistent with the other data types. PR #1101
• Separated data variable formatting from assign attributes function into dataset_formatter function.- PR #1102
• Required and optional keys for "column" data type were updated to reflect the two sources of this data (site, satellite) PR #1104
• Ensure metadata keywords for NOAA obspack are consistent with wider definitions including renaming data_source to dataset_source (data_source would be "internal"). PR #1110
• Updated data type classes to dynamically select inputs to pass to parse function and to include any required/optional keys not passed to the parse function within the metadata. PR #1111
• When adding new data sources, updated how lookup keys add optional keys. This used to only extract these from the optional_metadata input but this now allows keys to be added through any metadata. PR #1112
• Formalising metadata data merging logic within new util.metadata_util functions. PR #1113
• Splited build and publish steps in the workflow and check for - and . in the tags for build and publish. PR #759

Fixed

• Bug where a datasource's folder in the data directory was not deleted by Datasource.delete_all_data(). This was causing check_zarr_store in util/_user.py to give a false negative. PR #1126
• Bug where an input filepath list to standardise_surface was only storing the last file hash. This allowed for some files to bypass the check for the same files depending on where they were in the original filepath list. PR #1100
• Bug where filepath needed to be a Path object when storing the file hash values. PR #1108
• Catch an AttributeError when trying synchronise attributes and metadata and a user passes a bool - PR #1029
• Mypy issue fixed for util.download_data() function based on updates described requests Issue 465 and included in urllib3 PR 159. This allowed the decode_content flag to be set directly rather than needing to patch the method. PR #1118

[0.9.0] - 2024-08-14

Added

• In get_obs_surface, if inlet is passed a slice and multiple search results are found, they will be combined into a single ObsData object with a "inlet" data variable. PR #1066
• Packaging and release documentation. PR #961
• Options to search metastore by "negative lookup" and by "test functions"; the latter is used to implement searching by a slice object to find a range of values - PR #1064
• Code to combine multiple data objects - PR #1063
• A new object store config file to allow customisation of metadata keys used to store data in unique Datasources - PR #1041
• Adds data_level and data_sublevel as additional keys which can be used to distinguish observation surface data - PR #1051
• New keywords data_level and data_sublevel as additional keys which can be used to distinguish observation surface data - PR #1051
• The dataset_source keyword previously only used for retrieved data is now available when using standardise_surface as well. This allows an origin key for the dataset to be included e.g. "InGOS", "European ObsPack". PR #1083
• Footprint parser for "paris" footprint format which includes the new format used for both the NAME and FLEXPART LPDM models - PR #1070
• Added AGAGE as a source format in the standardise_surface function, and associated parser functions and tests. Reads in output files from Matt Rigby's agage-archive - PR #912
• Added feature of file sorting at standardise level before processing and remove filepaths input option - PR #1074
• Utility functions to combine multiple "data objects" (e.g. ObsData, or anything with .data and .metadata attributes) - PR #1063

Updated

• Updated base to offset in resample due to xarray deprecation. PR #1073
• Updated get_obs_column to output mole fraction details. This involves using the apriori level data above a maximum level and applying a correction to the column data (aligned with this process within acrg code). PR #1050
• The data_source keyword is now included as "internal" when using standardise_surface to distinguish this from data retrieved from external sources (e.g. "icos", "noaa_obspack"). PR #1083
• Added interactive timeseries plots in Search and Plotting tutorial. PR #953
• Pinned the icoscp version within requirements to 0.1.17 based on new authentication requirements. PR #1084
• The icos_data_level metadata keyword is now retired and replaced with data_level when using the retrieve.icos.retrieve_atmopsheric workflow to access data from the ICOS Carbon Portal. PR #1087
• Removing 'station_long_name' and 'data_type' as required keys from the metadata config file as these do not need to be used as keys to distinguish datasources when adding new data. PR #1088
• The sort flag can now be passed via the SearchResults.retrieve interfaces to choose whether the data is returned sorted along the time axis. PR #1090

Fixed

• Bug in test when checking customised chunks were stored correctly in the zarr store. dask v2024.8 now changed the chunk shape after this was sorted was test was updated to ensure this didn't sort when retrieving the data. PR #1090
• Error reporting for BaseStore context manager. PR #1059
• Formatting of inlet (and related keys) in search, so that float values of inlet can be retrieved - PR #1057
• Test for zarr compression test_bytes_stored_compression that was failing due to a slight mismatch between actual and expected values. The test now uses a bound on relative error - PR #1065
• Typo and possible performance issue in analysis._scenario.combine_datasets - PR #1047
• Pinned numpy to < 2.0 and netcdf4 to <= 1.6.5. Numpy 2.0 release caused some minor bugs in OpenGHG, and netCDF4's updates to numpy 2.0 were also causing tests to fail - PR #1043
• Fixed bug where slightly different latitude and longitude values were being standardised and not aligned later down the line. These are now all fixed to the openghg_defs domain definitions where applicable upon standardisation. PR #1049

[0.8.2] - 2024-06-06

Fixed

• Updated incorrect import for data_manager within tutorial. This now shows the import from openghg.dataobjects not openghg.store - PR #1007
• Issue causing missing data when standardising multiple files in a loop - PR #1032

Added

• Added ability to process CRF data as flux_timeseries datatype (one dimensional data) - PR #870

[0.8.1] - 2024-05-17

Added

• Ability to convert from an old style NetCDF object store to the new Zarr based store format - PR #967
• Updated parse_edgar function to handle processing of v8.0 Edgar datasets. PR #965
• Argument time_resolved is added as phase 1 change for high_time_resolution, also metadata is updated and added deprecation warning. - PR #968
• Added ability to pass additional tags as optional metadata through standardising_* and transform_flux_data functions. PR #981
• Renaming high_time_resolution argument to time_resolved in metadata as more appropriate description for footprints going forward and added deprecation warning. - PR #968
• Added explicit backwards compatability when searching previous object stores containing the high_time_resolution keyword rather than time_resolved - PR #990
• Added ability to pass additional tags as optional metadata through standardise_* and transform_flux_data functions. PR #981
• Check added object stores listed in the configuration which are in the previous format with a warning raised for this PR #962

Fixed

• source can be passed to transform_flux_data with the EDGAR parser; date isn't stored with the transformed EDGAR data, since this is used for choosing what data to add, but doesn't describe all of the data in the object store. Fixed bug due to string split over two lines in logging message - #PR 1010
• Fixed problem where the zarr store check raised an error for empty stores, preventing new zarr stores from being created - #PR 993
• Retrieval of level 1 data from the ICOS Carbon Portal now no longer tries to retrieve a large number of CSV files - #PR 868
• Added check for duplicate object store path being added under different store name, if detected raises ValueError. - PR #904
• Added check to verify if obs and footprint have overlapping time coordinates when creating a ModelScenario object, if not then raise ValueError - PR #954
• Added fix to make sure data could be returned within a date range when the data had been added non-sequentially to an object store - PR #997
• Replace references to old supplementary_data repository with openghg_defs - PR #999
• Added call to synonyms for species while standardising - PR #984

[0.8.0] - 2024-03-19

This version brings a breaking change with the move to use the Zarr file format for data storage. This means that object stores created with previous versions of OpenGHG will need to be repopulated to use this version. You should notice improvements in time taken for standardisation, memory consumption and disk usage. With the use of Zarr comes the ability for the user to control the way which data is processed and stored. Please see the documentation for more on this.

Added

• Added option to pass store argument to ModelScenario init method. PR #928

Fixed

• Issue caused when passing a list of files to be processed. If OpenGHG had seen some of the files before it would refuse to process any of them - PR #890

Changed

• Moved to store data in Zarr stores, this should reduce both the size of the object store and memory consumption whilst processing and retrieving data - PR #803
• standardise_footprint was updated to allow a source_format input to be specified. This currently only supports "acrg_org" type but can be expanded upon PR #914.
• Internal format for "footprint" data type was updated to rename meteorological variable names to standard names PR #918.
• Standardise_footprint now uses the meterological model input as a distinguishing keyword when adding data. PR #955.
• Meterological model input renamed from metmodel to met_model PR #957.
• Updated internal naming and input data_type to use "flux" rather than "emissions" consistently. - PR #827

Added

• More more explanation regarding use of search_* and get_* function in tutorial 1 PR #952

[0.7.1] - 2024-03-01

Fixed

• Bug fix for conversion of species parameter with its synonym value inside get_obs_surface_local. PR #871
• Missing requirement for filelock package added to conda environment file PR #857
• Missing store argument adding to search function allow searching within specific object stores PR #859
• Bug fix for allowing a period to be specified when this cannot be inferred from the input data PR #899

Fixed

• Bug fix for passing calibration_scale as optional parameter to the parser function. PR #872

[0.7.0] - 2023-12-15

Added

• Added DeprecationWarning to the functions parse_cranfield and parse_btt. - PR #792
• Added environment-dev.yaml file for developer conda environment - PR #769
• Added generic standardise function that accepts a bucket as an argument, and used this to refactor standardise_surface etc, and tests that standardise data - PR #760
• Added MetaStore abstract base class as interface for metastore classes, and a ClassicMetaStore subclass implements the same bucket/key structure as the previous metastore. All references to TinyDB are now in the objectstore module, meaning that there is only one place where code needs to change to use a different backend with the metastore - PR #771
• Added compression to Datasource.save and modified Datasource.load to take advantage of lazy loading via xarray.open_dataset - PR #755
• Added progress bars using rich package - PR #718
• Added config for Black to pyproject.toml - PR #822
• Added force option to retrieve_atmospheric and ObsSurface.store_data so that retrieved hashes can be ignored - PR #819
• Added SafetyCachingMiddleware to metastore, which caches writes and only saves them to disk if the underlying file has not changed. This is to prevent errors when concurrent writes are made to the metastore. PR #836

Fixed

• Bug fix for sampling period attribute having a value of "NOT_SET" and combining the observation and footprint data. Previously this was raising a ValueError. PR #808
• Bug where radon was not fetched using retrieve_atmospheric from icos data. - PR #794
• Bug with CRDS parse function where data for all species was being dropped if only one species was missing - PR #829
• Datetime processing has been updated to be compatible with Pandas 2.0: the date_parser argument of read_csv was deprecated in favour of date_format. PR #816
• Updated ICOS retrieval functionality to match new metadata retrieved from ICOS Carbon Portal - PR #806
• Added "parse_intem" function to parse intem emissions files - PR #804

Changed

• Datasource UUIDs are no longer stored in the storage class and are now only stored in the metadata store - PR #752
• Support dropped for Python 3.8 - PR #818. OpenGHG now supports Python >= 3.9.

[0.6.2] - 2023-08-07

Fixed

• Bug where the object store path being written to JSON led to an invalid path being given to some users - PR #741

Changed

• Added read-only opening of the metadata store of each storage class when searching. This is done using a mode argument pased to the load_metastore function - PR #763

[0.6.1] - 2023-08-04

Added

• Added rich package to printing out SearchResults object in a table format. If using an editable install please update your environment to match requirements.txt / environment.yml - PR #696

Fixed

• Bug in get_readable_buckets: missing check for tutorial store - PR #729
• Bug when adding high time resolution footprints to object store: they were not being distinguished from low resolution footprints - PR #720
• Bug due to object_store key not being present in Datasource metadata - PR #725
• Bug in DataManager where a string was interpreted as a list when processing metadata keys to be deleted - PR #713

[0.6.0] - 2023-07-18

Added

• Multiple object stores are now supported. Any number of stores may be accessed and read from and written to - PR #664
• Added standardise_column() wrapper function - PR #569
• The 'height_name' definition from the openghg/supplementary_data repository for each site can now be accessed, used and interpreted - PR #648
• Allow metadata within metastore to be updated for existing data sources for the latest version, start and end dates of the data - PR #652
• Allow mismatches between values in metadata and data attributes to be updated to either match to the metadata or attribute collated details - PR #682

Changed

• Configuration file format has been updated to support multiple object stores, note directing users to upgrade has been added
• The openghg --quickstart functionality has been updated to allow multiple objects stores to be added and migrate users from the previous version of the configuration file
• Synchronise metadata within metastore to align with data sources when updating (including latest version, start and end dates of the data) - PR #652 and PR #664
• The name of the DataHandler class has been changed to DataManager to better reflect its function.

Removed

• Reading multi-site AQMesh data is no longer possible. This may be reintroduced if required.

[0.5.1] - 2023-05-10

Fixed

• Fix for the sampling period of data files being read incorrectly - PR #584
• Fix for overlapping dateranges being created when adding new data to a Datasource. This introduced errors when keys were being removed and updated - PR #570
• Incorrect data being retrieved by the ICOS retrieval function - PR #611
• Error raised on attempt to delete object store after it wasn't created by some tests - PR #626
• Very small (nanosecond) changes in period between measurements resulting in error due to pandas.infer_period not being able to return a period - PR #634
• Processing of ObsPack resulted in errors due to limited metadata read and data overwrite, temporary fix in place for now - PR #642

Changed

• Mock for openghg_defs data to remove external dependency for tests - PR #582
• Removed use of environment variable for test store, moved to mock - PR # 580
• Temporary pinning of pandas < 2.0 due to changes that introduced errors - PR #619
• ObsData.plot_timeseries now uses openghg.plotting.plot_timeseries to avoid duplication in efforts/code - PR #624
• openghg.util._user.get_user_config_path now creates openghg.conf in ~/.openghg - PR #690

[0.5.0] - 2023-03-14

Added

• New tutorial on changing object store path using command line
• New tutorial on adding data from EDGAR database
• Ability to use separate
• New keywords to allow metadata for emissions data to be more specific
• New command-line tool to setup user configuration file using openghg --quickstart
• New installation guide for users and developers
• Added check for 0-dimension time coordinates in some NetCDFs

Fixed

• Fix for retrieval of different ICOS datasets
• Fix for user configuration file not being written out correctly

Removed

• Supplementary site and species data moved to separate supplementary_data repository
• Unused test data files

Changed

• Print statements changed to logging
• Updated version of mypy used to 0.991
• Converted all tutorial notebooks to restructured text files

[0.4.0] - 2022-11-12

Added

• A new DataHandler class for modification and deletion of metadata and added a matching tutorial.
• Move to a new user configuration file.
• A new ObsColumn class for handing satellite data

Fixed

• Fixes for search parameters such as inlet height, sampling height
• Removed date parameter from Flux data to improve searchability
• Inlet and height are now aliases for each other for footprints and obs_surface
• Check for tuple of data and precisions file on processing of GCWERKS data

Changed

• Documentation overhaul to ensure all functions are visible in either the user or developer documentation
• The OPENGHG_PATH environment variable check has been deprecated in favour of the user config file.

[0.3.1] - 2022-07-19

Fixed

• Removed old footprints_data_merge workflow which is superceded by ModelScenario.
• Tidied and updated tutorial notebooks
• Updated the conda build recipe

Removed

• Removed unused jobs submodule

[0.3.0] - 2022-08-31

Added

• Full ICOS and CEDA archive pulling capabilities in the cloud and locally.
• Adds logging to logfile, stored at ~/openghg_log when run locally.
• Improved schema checking for different file formats, including footprint files.

Fixed

• Fixes the clean_string function of openghg.util to allow dashes as these are commonly used in species names.
• Improves local routing functionality within cloud functions

Changed

• OpenGHG now only supports Python >= 3.8

[0.2.1] - 2022-07-27

Added

• Adds improved get_obs_surface behaviour for cloud usage.
• Added shortcut routing for serverless functions, see openghg.cloud.call_function. This differentiates between running on the hub or the cloud, where cloud is classed as running within a serverless function environment.
• Added new running_locally function in addition to the running_in_cloud or running_on_hub to allower easier checks with the openghg.client functions.

Fixed

• Fix to metadata storage for different sampling period formats
• Fix to SearchResults where environment checks were resulting in attemping to access a local object store even though we were in a Hub environment.

[0.2.0] - 2022-07-19

Added

• New openghg.client functions for our cloud platform. Standardisation, searching and retrieval of data is passed to either the local or cloud function call depending on platform setup.

Fixed

• Extra checks added to reading of emissions NetCDF datasets that have a single time value. Previously an error occurred due to performing a len on an unsized object.

Removed

• The openghg.client.process functions have been removed. These have been replaced with openghg.client.standardise functions.

Changed

• A split in the tutorials for cloud and local platforms. We've updated all the tutorials to better cover the differences in running OpenGHG with different setups.

[0.1.0] - 2022-05-18

Added

• Standardise measuremnts taken from many different sources - Tutorial 1 - Adding observation data
• Rank observations to ensure use of the best measurements - Tutorial 2 - Ranking observations
• Compare observations with emissions - Tutorial 3 - Comparing observations to emissions
• Create workflows using high time resolution CO2 data - Tutorial 4 - Working with high time resolution CO2
• Search for data in the OpenGHG data store and create plots to compare measurements - Tutorial 5 - Searching and plotting
• Retrieve and explore NOAA ObsPack data - Tutorial 6 - Exploring the NOAA ObsPack data
• Pull data from the ICOS Carbon Portal and the CEDA archive - Tutorial 7 - Retrieving data from ICOS and CEDA