Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

extend -e option to handle regular expressions #115

Merged
merged 2 commits into from
Jan 30, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,12 @@ All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)

## 1.12.0
### Fixed
- Added EDL based token downloading, removing CMR tokens [98](https://github.com/podaac/data-subscriber/issues/98),
### Added
- Added ability to download by filename [109](https://github.com/podaac/data-subscriber/issues/109) and additional regression testing

## 1.11.0
### Fixed
- Fixed an issue where token-refresh was expecting a dictionary, not a list of tuples
Expand Down
18 changes: 17 additions & 1 deletion Downloader.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ For installation and dependency information, please see the [top-level README](R

```
$> podaac-data-downloader -h
usage: PO.DAAC bulk-data downloader [-h] -c COLLECTION -d OUTPUTDIRECTORY [--cycle SEARCH_CYCLES] [-sd STARTDATE] [-ed ENDDATE] [-f] [-b BBOX] [-dc] [-dydoy] [-dymd] [-dy] [--offset OFFSET] [-e EXTENSIONS] [--process PROCESS_CMD] [--version] [--verbose] [-p PROVIDER] [--limit LIMIT]
usage: PO.DAAC bulk-data downloader [-h] -c COLLECTION -d OUTPUTDIRECTORY [--cycle SEARCH_CYCLES] [-sd STARTDATE] [-ed ENDDATE] [-f] [-b BBOX] [-dc] [-dydoy] [-dymd] [-dy] [--offset OFFSET] [-e EXTENSIONS] [--granule-name GRANULE] [--process PROCESS_CMD] [--version] [--verbose] [-p PROVIDER] [--limit LIMIT]

optional arguments:
-h, --help show this help message and exit
Expand Down Expand Up @@ -34,6 +34,8 @@ optional arguments:
--offset OFFSET Flag used to shift timestamp. Units are in hours, e.g. 10 or -10.
-e EXTENSIONS, --extensions EXTENSIONS
The extensions of products to download. Default is [.nc, .h5, .zip, .tar.gz]
-gr GRANULE, --granule-name GRANULE
The name of the granule to download. Only one granule name can be specified. Script will download all files matching similar granule name sans extension.
--process PROCESS_CMD
Processing command to run on each downloaded file (e.g., compression). Can be specified multiple times.
--version Display script version information and exit.
Expand Down Expand Up @@ -121,6 +123,20 @@ machine urs.earthdata.nasa.gov

## Advanced Usage

### Download data by filename

If you're aware of a file you want to download, you can use the `-gr` option to download by a filename. The `-c` (COLLECTION) and `-d` (directory) options are still required.

The `-gr` option works by taking the file name, removing the suffix and searching for a CMR entry called the granuleUR. Some examples of this include:

| Collection | Filename | CMR GranuleUR |
| --- | ----------- | ----------- |
| MUR25-JPL-L4-GLOB-v04.2 | 20221206090000-JPL-L4_GHRSST-SSTfnd-MUR25-GLOB-v02.0-fv04.2.nc | 20221206090000-JPL-L4_GHRSST-SSTfnd-MUR25-GLOB-v02.0-fv04.2 |
| JASON_CS_S6A_L2_ALT_HR_STD_OST_NRT_F | S6A_P4_2__HR_STD__NR_077_039_20221212T181728_20221212T182728_F07.nc | S6A_P4_2__HR_STD__NR_077_039_20221212T181728_20221212T182728_F07 |

Because of this behavior, granules without data suffixes and granules where the the UR does not directly follow this convention may not work as anticipated. We will be adding the ability to download by granuleUR in a future enhancement.


### Download data by cycle

Some PO.DAAC datasets are better suited for cycles based search instead of start and end times. To enabled this, we've added 'cycle' based downloading to the data-downloader. The following example will download data from cycle 42:
Expand Down
113 changes: 50 additions & 63 deletions poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[tool.poetry]
name = "podaac-data-subscriber"
version = "1.11.0"
version = "1.12.0"
description = "PO.DAAC Data Subscriber Command Line Tool"
authors = ["PO.DAAC <podaac@podaac.jpl.nasa.gov>"]
readme = "README.md"
Expand Down
Loading