Skip to content

New function to download subsets in ecco_download.py, and associated tutorial #67

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Dec 23, 2023
Merged

New function to download subsets in ecco_download.py, and associated tutorial #67

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a979f48
New function to download subsets, and associated tutorial
Dec 23, 2023
f2bfb85
Tutorials for ECCO and subset downloads, formatting corrections
Dec 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4,472 changes: 4,472 additions & 0 deletions ...ESS/Downloading_ECCO_datasets_from_PODAAC/Tutorial_Python3_Downloading_ECCO_Subsets.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

152 changes: 91 additions & 61 deletions ...from_PODAAC/Tutorial_Python3_Jupyter_Notebook_Downloading_ECCO_Datasets_from_PODAAC.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

813 changes: 785 additions & 28 deletions ECCO-ACCESS/Downloading_ECCO_datasets_from_PODAAC/ecco_download.py
View file
Open in desktop

Large diffs are not rendered by default.

92 changes: 92 additions & 0 deletions ECCO-ACCESS/Downloading_ECCO_datasets_from_PODAAC/wget_download_fromlist.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
#!/bin/bash

# Andrew Delman, December 2023
#
# This script takes PO.DAAC Opendap URLs listed in a text file
# and downloads them to the current directory, or another directory
# specified by option -P.
# For example:
# ./wget_download_fromlist.sh -i urls_download.txt \
# -P /ECCOv4_downloads/ -n Caribbean \
# -u username -p password
# downloads the files from URLs listed in ./urls_download.txt,
# to the directory /ECCOv4_downloads/, and appends the
# identifier 'Caribbean' to each of the downloaded file names.
# Input options can be specified either using the -i -P -n -u -p tags
# shown above, or sequentially in that order without the tags.
# However, option and positional/sequential inputs can not be combined
# when this script is called.
#
# Note: if NASA Earthdata user authentication is already stored in
# the user's .netrc file, then -u and -p can be omitted,
# and storing the authentication in .netrc is recommended for frequent users.
# See https://wiki.earthdata.nasa.gov/display/EL/How+To+Access+Data+With+cURL+And+Wget
# for a step-by-step guide to set this up.


# default arguments
download_dir="./"
append_id=""

# positional argument support
if [ "$1" != "-i" ]; then
download_dir="$2"
append_id="$3"
username="$4"
password="$5"
fi

# if input options given, assign to string variables
while getopts ":i:P:n:u:p:" option; do
case $option in
i) # text file specifying URLs to download
url_listfile="$OPTARG";;
P) # directory (with path) to download files to
download_dir="$OPTARG";;
n) # identifier to append to file names
append_id="$OPTARG";;
u) # Earthdata username
username="$OPTARG";;
p) # Earthdata password
password="$OPTARG";;
esac
done

# if -i option not supplied then assume $1 is URL file list
if [ -z ${url_listfile+x} ]; then
url_listfile="$1"
fi

# if no input arguments supplied, return error message and exit
if [ -z ${1+x} ]; then
echo "Error: no URL file list supplied. No files downloaded."
exit
fi

# create download directory if it does not already exist
mkdir -p "$download_dir"


# read URLs from URL text file and download to $download_dir,
# with file names as the name of the granule plus $append_id
while IFS= read -r line; do
no_paths=${line##*/granules/}
after_dap=${no_paths#*.dap.}
filename=${no_paths%.dap.nc*}
if [ "${after_dap:0:2}" = "nc" ]; then
filename=$filename"_"$append_id".nc"
else
echo "Downloaded file type uncertain; may not be NetCDF"
fi
if [ "${download_dir:(-1)}" = "/" ]; then
path_filename=$download_dir$filename
else
path_filename=$download_dir"/"$filename
fi
if [ -n "$username" ] && [ -n "$password" ]; then
wget -nv -nc -c -O $path_filename \
--user="$username" --password="$password" $line
else
wget -nv -nc -c -O $path_filename $line
fi
done < $url_listfile
Binary file added ECCO-ACCESS/misc/IBTrACS.NA.2004-2006.nc
View file
Open in desktop
Binary file not shown.
161 changes: 38 additions & 123 deletions Intro_to_PO_Tutorials/Geostrophic_balance.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
"metadata": {},
"source": [
"# Part 1: Geostrophic balance\n",
"Andrew Delman, updated 2023-01-23.\n",
"Andrew Delman, updated 2023-12-22.\n",
"\n",
"## Objectives\n",
"\n",
Expand Down Expand Up @@ -118,21 +118,26 @@
"Help on function ecco_podaac_download in module ecco_download:\n",
"\n",
"ecco_podaac_download(ShortName, StartDate, EndDate, download_root_dir=None, n_workers=6, force_redownload=False)\n",
" This routine downloads ECCO datasets from PO.DAAC. It is adapted from the Jupyter notebooks created by Jack McNelis and Ian Fenty (https://github.com/ECCO-GROUP/ECCO-ACCESS/blob/master/PODAAC/Downloading_ECCO_datasets_from_PODAAC/README.md) and modified by Andrew Delman (https://ecco-v4-python-tutorial.readthedocs.io).\n",
" This routine downloads ECCO datasets from PO.DAAC. It is adapted from the Jupyter notebooks \n",
" created by Jack McNelis and Ian Fenty (https://github.com/ECCO-GROUP/ECCO-ACCESS/blob/master/PODAAC/Downloading_ECCO_datasets_from_PODAAC/README.md)\n",
" and modified by Andrew Delman (https://ecco-v4-python-tutorial.readthedocs.io).\n",
" \n",
" Parameters\n",
" ----------\n",
" ShortName: the ShortName of the dataset (can be identified from https://search.earthdata.nasa.gov/search?fpj=ECCO, selecting the \"i\" information button and the ShortName will appear in a gray box in the upper-left corner)\n",
" \n",
" StartDate: the start of the time range to be downloaded, expressed in the format \"YYYY-MM-DD\"\n",
" ShortName: str, the ShortName that identifies the dataset on PO.DAAC.\n",
" \n",
" EndDate: the end of the time range to be downloaded, expressed in the format \"YYYY-MM-DD\"\n",
" StartDate,EndDate: str, in 'YYYY', 'YYYY-MM', or 'YYYY-MM-DD' format, \n",
" define date range [StartDate,EndDate] for download.\n",
" EndDate is included in the time range (unlike typical Python ranges).\n",
" ECCOv4r4 date range is '1992-01-01' to '2017-12-31'.\n",
" For 'SNAPSHOT' datasets, an additional day is added to EndDate to enable closed budgets\n",
" within the specified date range.\n",
" \n",
" download_root_dir: path of the parent directory to download ECCO files\n",
" n_workers: int, number of workers to use in concurrent downloads.\n",
" \n",
" n_workers: number of workers to use in concurrent downloads\n",
" \n",
" force_redownload: if True, existing files will be redownloaded and replaced; if False, existing files will not be replaced\n",
" force_redownload: bool, if True, existing files will be redownloaded and replaced;\n",
" if False, existing files will not be replaced.\n",
"\n"
]
}
Expand All @@ -158,33 +163,14 @@
"output_type": "stream",
"text": [
"created download directory C:\\Users\\adelman\\Downloads\\ECCO_V4r4_PODAAC\\ECCO_L4_OCEAN_VEL_LLC0090GRID_MONTHLY_V4R4\n",
"{'ShortName': 'ECCO_L4_OCEAN_VEL_LLC0090GRID_MONTHLY_V4R4', 'temporal': '2000-01-02,2000-01-31'}\n",
"\n",
"Total number of matching granules: 1\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:08<00:00, 8.11s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Total number of matching granules: 1\n",
"DL Progress: 100%|###########################| 1/1 [00:06<00:00, 6.01s/it]\n",
"\n",
"=====================================\n",
"total downloaded: 30.6 Mb\n",
"avg download speed: 3.76 Mb/s\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n"
"avg download speed: 5.07 Mb/s\n",
"Time spent = 6.032305955886841 seconds\n"
]
}
],
Expand All @@ -193,7 +179,7 @@
"# to default path ~/Downloads/ECCO_V4r4_PODAAC/\n",
"vel_monthly_shortname = \"ECCO_L4_OCEAN_VEL_LLC0090GRID_MONTHLY_V4R4\"\n",
"ecco_podaac_download(ShortName=vel_monthly_shortname,\\\n",
" StartDate=\"2000-01-02\",EndDate=\"2000-01-31\",download_root_dir=None,\\\n",
" StartDate=\"2000-01-01\",EndDate=\"2000-01-31\",download_root_dir=None,\\\n",
" n_workers=6,force_redownload=False)"
]
},
Expand All @@ -207,33 +193,14 @@
"output_type": "stream",
"text": [
"created download directory C:\\Users\\adelman\\Downloads\\ECCO_V4r4_PODAAC\\ECCO_L4_DENS_STRAT_PRESS_LLC0090GRID_MONTHLY_V4R4\n",
"{'ShortName': 'ECCO_L4_DENS_STRAT_PRESS_LLC0090GRID_MONTHLY_V4R4', 'temporal': '2000-01-02,2000-01-31'}\n",
"\n",
"Total number of matching granules: 1\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:06<00:00, 6.12s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Total number of matching granules: 1\n",
"DL Progress: 100%|###########################| 1/1 [00:07<00:00, 7.11s/it]\n",
"\n",
"=====================================\n",
"total downloaded: 30.98 Mb\n",
"avg download speed: 5.05 Mb/s\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n"
"avg download speed: 4.36 Mb/s\n",
"Time spent = 7.113083362579346 seconds\n"
]
}
],
Expand All @@ -242,7 +209,7 @@
"# to default path ~/Downloads/ECCO_V4r4_PODAAC/\n",
"denspress_monthly_shortname = \"ECCO_L4_DENS_STRAT_PRESS_LLC0090GRID_MONTHLY_V4R4\"\n",
"ecco_podaac_download(ShortName=denspress_monthly_shortname,\\\n",
" StartDate=\"2000-01-02\",EndDate=\"2000-01-31\",download_root_dir=None,\\\n",
" StartDate=\"2000-01-01\",EndDate=\"2000-01-31\",download_root_dir=None,\\\n",
" n_workers=6,force_redownload=False)"
]
},
Expand All @@ -265,54 +232,23 @@
"output_type": "stream",
"text": [
"created download directory C:\\Users\\adelman\\Downloads\\ECCO_V4r4_PODAAC\\ECCO_L4_OCEAN_VEL_LLC0090GRID_DAILY_V4R4\n",
"{'ShortName': 'ECCO_L4_OCEAN_VEL_LLC0090GRID_DAILY_V4R4', 'temporal': '2000-01-01,2000-01-01'}\n",
"\n",
"Total number of matching granules: 2\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:07<00:00, 3.97s/it]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Total number of matching granules: 1\n",
"DL Progress: 100%|###########################| 1/1 [00:06<00:00, 6.58s/it]\n",
"\n",
"=====================================\n",
"total downloaded: 61.36 Mb\n",
"avg download speed: 7.72 Mb/s\n",
"total downloaded: 30.68 Mb\n",
"avg download speed: 4.65 Mb/s\n",
"Time spent = 6.5920562744140625 seconds\n",
"created download directory C:\\Users\\adelman\\Downloads\\ECCO_V4r4_PODAAC\\ECCO_L4_DENS_STRAT_PRESS_LLC0090GRID_DAILY_V4R4\n",
"{'ShortName': 'ECCO_L4_DENS_STRAT_PRESS_LLC0090GRID_DAILY_V4R4', 'temporal': '2000-01-01,2000-01-01'}\n",
"\n",
"Total number of matching granules: 2\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:07<00:00, 3.73s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Total number of matching granules: 1\n",
"DL Progress: 100%|###########################| 1/1 [00:04<00:00, 4.97s/it]\n",
"\n",
"=====================================\n",
"total downloaded: 62.39 Mb\n",
"avg download speed: 8.35 Mb/s\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n"
"total downloaded: 31.2 Mb\n",
"avg download speed: 6.27 Mb/s\n",
"Time spent = 4.976189613342285 seconds\n"
]
}
],
Expand Down Expand Up @@ -348,35 +284,14 @@
"output_type": "stream",
"text": [
"created download directory C:\\Users\\adelman\\Downloads\\ECCO_V4r4_PODAAC\\ECCO_L4_GEOMETRY_LLC0090GRID_V4R4\n",
"{'ShortName': 'ECCO_L4_GEOMETRY_LLC0090GRID_V4R4', 'temporal': '2000-01-01,2000-01-01'}\n",
"\n",
"Total number of matching granules: 1\n",
"\n",
"GRID_GEOMETRY_ECCO_V4r4_native_llc0090.nc already exists, and force=False, not re-downloading\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<?, ?it/s]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"DL Progress: 100%|###########################| 1/1 [00:03<00:00, 3.74s/it]\n",
"\n",
"=====================================\n",
"total downloaded: 0.0 Mb\n",
"avg download speed: 0.0 Mb/s\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n"
"total downloaded: 8.57 Mb\n",
"avg download speed: 2.29 Mb/s\n",
"Time spent = 3.7480597496032715 seconds\n"
]
}
],
Expand Down
29 changes: 5 additions & 24 deletions Tutorials_as_Jupyter_Notebooks/ECCO_v4_data_structure_basics.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,33 +27,14 @@
"output_type": "stream",
"text": [
"created download directory C:\\Users\\adelman\\Downloads\\ECCO_V4r4_PODAAC\\ECCO_L4_TEMP_SALINITY_LLC0090GRID_MONTHLY_V4R4\n",
"{'ShortName': 'ECCO_L4_TEMP_SALINITY_LLC0090GRID_MONTHLY_V4R4', 'temporal': '2010-01-02,2010-12-31'}\n",
"\n",
"Total number of matching granules: 12\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████████████████████████████████████████████████████████████████████████████| 12/12 [00:18<00:00, 1.55s/it]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Total number of matching granules: 12\n",
"DL Progress: 100%|#########################| 12/12 [00:15<00:00, 1.29s/it]\n",
"\n",
"=====================================\n",
"total downloaded: 208.75 Mb\n",
"avg download speed: 11.2 Mb/s\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n"
"avg download speed: 13.48 Mb/s\n",
"Time spent = 15.480218887329102 seconds\n"
]
}
],
Expand All @@ -68,7 +49,7 @@
"# download files (granules) containing 2010 monthly mean temperatures\n",
"curr_shortname = \"ECCO_L4_TEMP_SALINITY_LLC0090GRID_MONTHLY_V4R4\"\n",
"ecco_podaac_download(ShortName=curr_shortname,\\\n",
" StartDate=\"2010-01-02\",EndDate=\"2010-12-31\",\\\n",
" StartDate=\"2010-01-01\",EndDate=\"2010-12-31\",\\\n",
" download_root_dir=ECCO_dir,n_workers=6,force_redownload=False)"
]
},
Expand Down
1 change: 1 addition & 0 deletions doc/Downloading_Subsets_of_ECCO_Datasets.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
../ECCO-ACCESS/Downloading_ECCO_datasets_from_PODAAC/Tutorial_Python3_Downloading_ECCO_Subsets.ipynb
1 change: 1 addition & 0 deletions doc/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ The `ecco_v4_py`_ package used in this tutorial was inspired by the `xmitgcm`_ p
fields
Installing_Python_and_Python_Packages
Downloading_ECCO_Datasets_from_PODAAC_Python.ipynb
Downloading_Subsets_of_ECCO_Datasets.ipynb
Tutorial_wget_Command_Line_HTTPS_Downloading_ECCO_Datasets_from_PODAAC
Tutorial_Introduction

Expand Down