imi-1.2.1

What's Changed

We recommend updating to the new version of the IMI, as it includes a number of updates, bug fixes, and new features. We ask that users read bugfixes and features with bolded text to see if any bugs in prior versions of the IMI may have affected their inversions.

Bugfixes

• Important: after consultation with SRON we have removed the previously used albedo filters for processing the TROPOMI retrieval, see the linked pull request for more information. drop TROPOMI albedo filters, and Rocky 8 on Cannon by @nicholasbalasus
• bugfix to catch errors in TROPOMI variables by @sabourbaray in #128
• Bugfix/reduce obs array size by @laestrada in #135
• bugfix to only turn on extra observation switches for prior and posterior runs by @sabourbaray in #131
• Bugfix/fix harvard geos chem env files by @nicholasbalasus in #138
• Bugfix: Fix DOFS calculation by @laestrada in #140

New Features

• Important: We have rolled out a new default boundary conditions version,v2023-06. v2023-04 used the legacy CH4 Boundary Conditions repository. v2023-06 has two important updates: (1) removal of a misapplied stratospheric correction in the legacy repository and (2) removal of albedo filters for boundary condition generation with TROPOMI correction applied. We recommend that all users switch to the new boundary condition version by setting BCversion: "v2023-04" in the config.yml file.
• feature allow specification of partition for sbatch commands by @laestrada in #132 -- this is for ease of use on local clusters
• Enable nc4 compression by @laestrada in #143 This update greatly reduces the file size of the output directory generated by the IMI.

New Contributors

• @sabourbaray made their first contribution in #128
• v2023-06-BCs
  Full Changelog: imi-1.2.0...imi-1.2.1

imi-1.2.0

What's Changed

We recommend updating to the new version of the IMI, as it includes a number of updates, bug fixes, and new features. We ask that users read bugfixes with bolded text to see if any bugs in prior versions of the IMI may have affected their inversions.

Bugfixes

• *Important* Bugfix/tropomi data download -- Recent reprocessing of TROPOMI processor to v020400/020500 (in mid March 2023) has caused duplicate files with previous processor versions to be downloaded by the IMI. This could erroneously double the number of observations used in your inversion. We have fixed this issue in this version update. We recommend updating to IMI 1.2.0 for all future inversions. For past inversions, check for repeated orbit numbers in your tropomi data directory. If you find duplicates, delete files that are not v020400 or 020500, delete the inversion and posterior run directories and rerun the inversion and posterior setup and run steps. No need to rerun the jacobian, or spinup steps.
• *Important*Bugfix/IMI Boundary conditions -- The IMI uses tropomi smoothed boundary conditions to simulate background concentrations outside of the domain. We found a bug that caused the tropomi adjustment to be limited to 30ppb, which is insufficient for post 2019 inversions. We removed this 30ppb cap and regenerated the BCs on march 8, 2023. If you ran an inversion prior to this we recommend rerunning the IMI. Additionally, we have since added stable IMI BC versions, which can be specified in the IMI configuration file. See the relevant feature below. v2023-04 includes these updates. However, we plan to release v2023-06 soon, which includes a bugfix removing a misapplied stratospheric correction and transitions to using the scripts in src/write_BCs instead of the CH4-boundary-condition-scripts repository
• Bugfix/preview estimate -- previously the IMI Preview did not effectively take into account domain size when estimating cost of the inversion
• Bugfix/restart path -- update restart paths to use absolute paths instead of relative in jacobian and posterior directories
• Bugfix/merra2 compatibility -- fix paths to met data for .5 degree inversion option
• Bugfix/save viz output data -- fix issue where individual tropomi data was not saved properly (only relevant for visualizations in visualization_notebook)
• Bugfix/get num statevector elements update approach to getting the number of state vector elements in a less resource intensive way
• Add .readthedocs.yaml file to configure ReadTheDocs builds

New Features

• Feature/clustering options -- provide automated clustering options for reducing the state vector dimension. This can greatly reduce the cost of the IMI. Read the linked documentation for more information and see the relevant imi config file documentation. PRs include #88 and #102
• Feature/offshore emissions -- include offshore emissions in state vector based on emissions threshold (see relevant config file documentation)
• Feature/resource allocation settings -- add configuration settings for allocation of resources to simulations and jacobian simulations. See the config file documentation for more info.
• Feature/Add Boundary conditions scripts to IMI source code -- bring the imi BC file generation code into the IMI. Additionally, this update unifies how we process the tropomi data for the BCs and in the inversion. Additionally we implement several improvements to the BC generation code, including:
  • Removal of a 30ppb cap on TROPOMI bias corrections. This affected 2019-2022 where the average correction was much higher than 30ppb and could lead to high prior simulation bias from TROPOMI observations
  • Infilling missing values using the latitudinal mean for +/- 15 days. For latitudes with no TROPOMI observations, the closest available latitudinal average is used
  • Outliers in the .01 percentile are now replaced with the next highest values (.02 percentile). To deal with potentially nonphysical values
  • Script for bias correction was rewritten in python from the original R script.
• Add BCversion variable to config file -- we will be publishing discrete IMI Boundary conditions versions (the latest one is v2023-06). This will allow users to specify a fixed BC version to use in their configuration files via a new configuration variable eg. BCversion: v2023-06.
• Feature/Check for duplicate TROPOMI files -- Print error if there are any duplicate tropomi orbit files being used in inversion.
• Update cannon env -- update imi conda env for cannon cluster
• Add templates for github issues and pull requests
• Display IMI logo and favicon on the imi.readthedocs.io site

Contributors

• @djvaron
• @laestrada
• @msulprizio
• @nicholasbalasus
• @yantosca

Full Changelog: imi-1.1.0...imi-1.2.0

imi-1.1.0

What's Changed

• Average TROPOMI observations into superobservations see (pull request) [https://github.com//pull/57]
• Update the IMI to use GEOS-Chem 14.0.2 see (pull request) [https://github.com//pull/83]
• Add additional nested domains. Now includes: AF, AS, EU, ME, NA, OC, RU, SA
  *** Additionally a bug was found in the boundary conditions files affecting transport. This bug affected files from February 2022 onwards. The bug is now fixed and the updated boundary conditions are now uploaded to s3 and the washu server. We recommend redownloading and rerunning the inversions for any time periods past February 2022. ***

Full Changelog: imi-1.0.0...imi-1.1.0

IMI 1.0.0 Release

What's Changed

• Feature: script to find corrupted met files on aws by @laestrada in #38
• Bugfix/download landcover files in create shapefile notebook by @laestrada in #39
• Update main with bugfixes. by @laestrada in #40
• Bugfix/corrupt tropomi_files: Handle corrupt tropomi files by @laestrada in #43
• Feature/safe mode: implement safe mode to prevent overwriting files by @laestrada in #44
• feature: add timers for each step in the IMI by @laestrada in #47
• Cleanup IMI configuration file, setup script, and run script by @msulprizio in #48
• Update IMI to use GEOS-Chem 13.4.1 by @msulprizio in #51

Full Changelog: imi-1.0.0-beta.1...imi-1.0.0

imi 1.0.0-beta.2 release

What's Changed

• Feature: script to find corrupted met files on aws by @laestrada in #38
• Bugfix/download landcover files in create shapefile notebook by @laestrada in #39
• Update main with bugfixes. by @laestrada in #40
• Bugfix/corrupt tropomi_files: Handle corrupt tropomi files by @laestrada in #43
• Feature/safe mode: implement safe mode to prevent overwriting files by @laestrada in #44
• feature: add timers for each step in the IMI by @laestrada in #47
• Cleanup IMI configuration file, setup script, and run script by @msulprizio in #48
• Update IMI to use GEOS-Chem 13.4.1 by @msulprizio in #51

Full Changelog: imi-1.0.0-beta.1...imi-1.0.0-beta.2

imi 1.0.0-beta.1 release

This is the beta release for the integrated methane inversion workflow (IMI).