imi-1.2.0

What's Changed

We recommend updating to the new version of the IMI, as it includes a number of updates, bug fixes, and new features. We ask that users read bugfixes with bolded text to see if any bugs in prior versions of the IMI may have affected their inversions.

Bugfixes

• *Important* Bugfix/tropomi data download -- Recent reprocessing of TROPOMI processor to v020400/020500 (in mid March 2023) has caused duplicate files with previous processor versions to be downloaded by the IMI. This could erroneously double the number of observations used in your inversion. We have fixed this issue in this version update. We recommend updating to IMI 1.2.0 for all future inversions. For past inversions, check for repeated orbit numbers in your tropomi data directory. If you find duplicates, delete files that are not v020400 or 020500, delete the inversion and posterior run directories and rerun the inversion and posterior setup and run steps. No need to rerun the jacobian, or spinup steps.
• *Important*Bugfix/IMI Boundary conditions -- The IMI uses tropomi smoothed boundary conditions to simulate background concentrations outside of the domain. We found a bug that caused the tropomi adjustment to be limited to 30ppb, which is insufficient for post 2019 inversions. We removed this 30ppb cap and regenerated the BCs on march 8, 2023. If you ran an inversion prior to this we recommend rerunning the IMI. Additionally, we have since added stable IMI BC versions, which can be specified in the IMI configuration file. See the relevant feature below. v2023-04 includes these updates. However, we plan to release v2023-06 soon, which includes a bugfix removing a misapplied stratospheric correction and transitions to using the scripts in src/write_BCs instead of the CH4-boundary-condition-scripts repository
• Bugfix/preview estimate -- previously the IMI Preview did not effectively take into account domain size when estimating cost of the inversion
• Bugfix/restart path -- update restart paths to use absolute paths instead of relative in jacobian and posterior directories
• Bugfix/merra2 compatibility -- fix paths to met data for .5 degree inversion option
• Bugfix/save viz output data -- fix issue where individual tropomi data was not saved properly (only relevant for visualizations in visualization_notebook)
• Bugfix/get num statevector elements update approach to getting the number of state vector elements in a less resource intensive way
• Add .readthedocs.yaml file to configure ReadTheDocs builds

New Features

• Feature/clustering options -- provide automated clustering options for reducing the state vector dimension. This can greatly reduce the cost of the IMI. Read the linked documentation for more information and see the relevant imi config file documentation. PRs include #88 and #102
• Feature/offshore emissions -- include offshore emissions in state vector based on emissions threshold (see relevant config file documentation)
• Feature/resource allocation settings -- add configuration settings for allocation of resources to simulations and jacobian simulations. See the config file documentation for more info.
• Feature/Add Boundary conditions scripts to IMI source code -- bring the imi BC file generation code into the IMI. Additionally, this update unifies how we process the tropomi data for the BCs and in the inversion. Additionally we implement several improvements to the BC generation code, including:
  • Removal of a 30ppb cap on TROPOMI bias corrections. This affected 2019-2022 where the average correction was much higher than 30ppb and could lead to high prior simulation bias from TROPOMI observations
  • Infilling missing values using the latitudinal mean for +/- 15 days. For latitudes with no TROPOMI observations, the closest available latitudinal average is used
  • Outliers in the .01 percentile are now replaced with the next highest values (.02 percentile). To deal with potentially nonphysical values
  • Script for bias correction was rewritten in python from the original R script.
• Add BCversion variable to config file -- we will be publishing discrete IMI Boundary conditions versions (the latest one is v2023-06). This will allow users to specify a fixed BC version to use in their configuration files via a new configuration variable eg. BCversion: v2023-06.
• Feature/Check for duplicate TROPOMI files -- Print error if there are any duplicate tropomi orbit files being used in inversion.
• Update cannon env -- update imi conda env for cannon cluster
• Add templates for github issues and pull requests
• Display IMI logo and favicon on the imi.readthedocs.io site

Contributors

• @djvaron
• @laestrada
• @msulprizio
• @nicholasbalasus
• @yantosca

Full Changelog: imi-1.1.0...imi-1.2.0