GFS Reforecast V2 Grib File Downloader
This is a small package to download version 2 reforecasts from the global forecast system (GFS). The data can be accessed via the NOAA CDC data server (National Oceanographic and Atmospheric Agency). The package provides a handy way to access and download a specific subset across all reforecasts.
The GFSV2 python package provides a set of functions and two executables
GFSV2_bulk for convenient data processing.
Some more details about the most recent changes can be found in the CHANGELOG.md file.
Update (I blamed my script but it was the the HDD)
For those who came across this package in September 2017: I've written a warning that there are some unknown problems. Everyhing looks fine! The problem was caused by the hard drive where my scripts were running. Due to serious segfaults the downloader crashed relatively randomly arrr.
While (trying to) bucktrack this issue I've added some additional settings in
curl section in the config files (optional), see lisa.conf
or config/default.conf files, section
Furthermore, an overall sleep time (see section
[main] sleeptime has been
added (optional). These options should help to avoid getting blacklisted if
several dozens of requests are made. However, as the rules are unknown
ftp.cdc.noaa.gov) you might see what works for you (depending on internet
speed, number of requests, ...).
[curl] curllog: string, logfile. If given, pycurl drops some download logs there.
[curl] timeout: integer. Custom pycurl timeout.
[curl] retries: integer.
0(only one shot), else number of retries if download fails.
[curl] sleeptime: time in seconds to sleep if download failed (before try to download again, only has an effect if
retries > 0).
[main] sleeptime: time in seconds to sleep after each download (each file which has to be downloaded; not the time to wait between two retries).
The github repository contains the small python package which should be ready for installation. You can simply install the package by calling:
pip install git+https://github.com/retostauffer/PyGFSV2.git
setup.py script should automatically take care of the dependencies
wgrib2 which is used for subsetting (if configuration files with subset
settings are used in
GFSV2_bulk; see below). After installation you can try
the installation (no subsetting) by calling:
GFSV2_get --step 12 24 --level 700 850 --param tmp_pres --date 2005-01-01
Requires the following python packages:
pycurl(and standard libs like
- If subsetting is used (see
wgrib2executable has to be callable (see CPC wgrib2 readme).
GFSV2_get executable can be used to download specific manually defined
subsets of the reforecast. This can be handy during the development/specification
process. However, if you would like to download larger amounts of data please
have a look at the
GFSV2_bulk description below!
Basic usage (shows usage help):
There are two required input arguments to
GFSV2_get. One is the
input, the other one is the
-d/--date input. Both can contain one or more
--param specifies which parameters or variables you would like to
--date the dates of the model initialization. A list/description
of all available parameters can be found here.
--date arguments have to be of type
YYYY-mm-dd. As an example
tmax_2m for January 1 2014:
GFSV2_get --param tmax_2m --date 2014-01-01
GFSV2_get -p tmax_2m -d 2014-01-01
In addition, a
-s/--steps argument can be given. The
steps are the
forecast lead times (e.g.,
24 for the forecast 24 hours after initialization).
Several steps can be defined:
GFSV2_get -p tmax_2m -d 2014-02-01 -s 24 48
By default only
sprd files (ensemble mean and ensemble spread)
will be downloaded. If the individual members and the control run are required
-m/--members flag has to be set:
GFSV2_get -p tmax_2m -d 2014-02-05 -s 24 -m
Downloading two different parameters for two different dates, only +24h forecast:
- ``GFSV2_get -p tmax_2m cape_sfc -d 2014-02-01 2014-02-02 -s 24
For pressure level variables (e.g,
(see here, Table 1) the
level can be given in advance to download specific levels only. Level specification
in millibars or hecto pascal. An example:
GFSV2_get -p tmp_pres -l 500 -d 2014-03-10
NOTE: The download will be skipped if the output grib file already exists!
This can be crucial if you download a data set, re-specify the settings and try
to run the script again. As the file already exists (even if containing different
subset of data) the data won't be downloaded again. In this case simply delete
the local grib files in
data/YYYY/mm and re-run the job.
This is the bulk download version of the
GFSV2_get executable explained
above. Rather than providing a set of input arguments only one argument is
allowed and required:
-c/--config is the path to a config
file. All specifications can be set in this config file. The config file allows
- Specific ouptut path/file name.
- Date range for which data should be downloaded (
onlyflag (download data for date range if and only if the date is in month
- A spatial subset (required
wgrib2installed). Data will be subsetted after downloading with respect to the specification (
- Parameters which have to be downloaded (each one can have it's own level/members specification).
The default config file (used by
be found here and can be used as a template
to specify the data set you need. A second template config can be found
in the repository called lisa.conf. After you have made a copy
and changed the config specification simply call:
GFSV2_bulk --config your_config_file.conf