Skip to content
Switch branches/tags

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

ESGF Publisher

v5.1.0b6 is the latest Pre-release version of the ESGF Publisher. We recommend everyone to upgrade and test this version for CMIP6 data.


Release notes:


  • CRITICAL: corrected File record ID format to include |data_node to conform to prior specification
  • Support for data root specifications that include the project string in the root
  • Bug fixes: citiaton case for command line project path, support tilde for homedir in cmor path property in config file


  • Update to support input4MIPs project
  • Added --version argument
  • Additonal arguments for esgunpublish
  • Halt publishing if a file listed in the mapfile isn’t found by autocurator (it will not through an error as)


  • Refactored classses for improved code reuse (among other features)
  • --no-auth mode for testing with esgf-docker publisher API


  • Cleans up output (verbose, silent) modes for some of the functionality


  • Corrects error with activity check


  • Properly updates File record latest=false when replacing a version
  • Fixes issue of Globus/GridFTP url defaults: the default is for no urls if the settings aren't populated

Legacy version v3., v4. (python3 pre-release)

Requirements: most requirements are listed in requirements.txt. Other requirements must be deployed via conda, include cdms2, cdtime and cmor (for CMIP6, other projects coming soon).

Installation: the most convenient method to install is via pip. To capture all dependencies, consider installation of the predetermined conda environment:

29 October 2019 version 3.7.3

  • Bugfixes and version compatibility updates

22 March 2019 version 3.7.0

  • Remove calls to esgfetchini and the esgprep requirement. Install esgprep separately with pip.

13 March 2019 version 3.6.2

  • Skip THREDDS reinit during esgsetup - solves installation issue for successful status code return
  • PrePARE to run on each input file rather than first in dataset
  • Fix some package versions, eg. requests, SQLAlchemy
  • Version information from esgpublish

18 December 2018 version 3.5.6

  • process files in sorted order
  • bugfix for certs location config
  • new config option to skip validation of standard names

26 June 2018 - version 3.5.2

  • REST API is default for both publish and unpublish ; support to use the ESGF CA bundle
  • Switching data specs versions during operations
  • esgtest_publish supports download test and skip-unpublish modes of operation
  • new processing modes:
    • --skip-aggregation
    • --commit-every allows for more frequent database commits to aid in reducing memory footprints

Note - logging controlled now by the [DEFAULT] section of esg.ini. If you have upgraded the publisher with a previous version of the file, you may notice that the old setting is WARNING, in which case, INFO level messages may not appear. Thus, the publisher may run successfully without any output.

16 August 2017 - version 3.2.6

  • Support for CMIP6
  • integration of PrePARE
  • PID creation via RabbitMQ
  • cdf2cim included
  • handler for non-netCDF and skip variable scan for multiple cases

29 August 2016 - version 3.1.0

  • Changes to support CMIP6
    • Configuration option to extract global attributes
    • Check for CMOR version in datafiles
    • CF checker integration
    • CMIP6-CV check integration (requires CMOR install)
  • Included documentation (requires sphinx, sphinx_rtd_theme)

03 August 2016 - version 3.0.1

  • inclusion of esgprep utility - see for more information
  • Use of project-specific esg.ini files read from the default location
  • Support for multiple mapfiles within a single directory
  • support for comma or space-delimited facets

Previous versions:

06 July 2015 - version 2.13.0

  • Support for multithreaded checksumming
  • Version pick up in esgscan and use in esgpublish
  • esgfind_excludes to find files in the thredds exclude property
  • Add optional facet values with esgadd_facetvalues for published datasets
  • ACME file handler to support publication of netCDF + other formats in single data set

14 January 2015 - version 2.12.2

  • Warning message for esgscan_directory failues
  • --nodbwrite flag to enable "dry run" for fileO validation without writing to postgreSQL database
  • Additional utility scripts: - metadata syncronization checks for PGSQL, Thredds and SOLR, - used for replica publication (at LLNL/PCMDI)