Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

i.sentinel.download: Enable downloading from Google Cloud #492

Merged
merged 10 commits into from
Apr 19, 2021
Merged

i.sentinel.download: Enable downloading from Google Cloud #492

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
60b2e7c
first working version of download via GCS
griembauer Mar 26, 2021
9a9a578
Merge branch 'master' of github.com:OSGeo/grass-addons into sentinel_…
griembauer Mar 30, 2021
5994580
correct folder handling for old inconsistent files
griembauer Mar 30, 2021
1342340
change logging to verbose; adaption to i.sentinel.parallel.download
griembauer Mar 31, 2021
65bb903
remove unfinished download; correct for L2A PHOEBUS manifest.safe inc…
griembauer Mar 31, 2021
377200b
split up functions; skip existing file download
griembauer Apr 1, 2021
c78655d
adapt description and html
griembauer Apr 1, 2021
cdadd3a
add status bar
griembauer Apr 1, 2021
fca9ec9
double quotes in all grass messages
griembauer Apr 6, 2021
593b367
add required QI_DATA/AUX_DATA folders even if empty
griembauer Apr 6, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
61 changes: 53 additions & 8 deletions grass7/imagery/i.sentinel/i.sentinel.download/i.sentinel.download.html
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
<h2>DESCRIPTION</h2>

The <em>i.sentinel.download</em> allows downloading Sentinel satellite products
from the <a href="https://scihub.copernicus.eu/">Copernicus Open Access Hub</a>.
Sentinel-2 Level-1C data can also be downloaded from the <a href="https://earthexplorer.usgs.gov/">USGS Earth Explorer</a>.
The <em>i.sentinel.download</em> addon allows downloading Sentinel satellite products
from the <a href="https://scihub.copernicus.eu/">Copernicus Open Access Hub</a>,
<a href="https://earthexplorer.usgs.gov/">USGS Earth Explorer</a>, or
<a href="https://cloud.google.com/storage/docs/public-datasets/sentinel-2?hl=en">Google Cloud Storage</a>.

<h3>Copernicus Open Access Hub</h3>
<p>
Expand Down Expand Up @@ -53,6 +54,30 @@ <h3>USGS Earth Explorer</h3>
see <a href="https://ers.cr.usgs.gov/register">EROS Registration System</a>
page for signing up.

<h3>Google Cloud Storage</h3>
<p>
The following product types (parameter <b>producttype</b>) are currently supported
for download from <b>Google Cloud Storage</b>:
<ul>
<li>Sentinel-2 (optical and infrared):
<ul>
<li>S2MSI2A: operational Bottom-Of-Atmosphere reflectances in cartographic geometry Level-2A)</li>
</ul>
<ul>
<li>S2MSI1C: Top-Of-Atmosphere reflectances in cartographic geometry (Level-1C)</li>
</ul>
</li>
</ul>
Querying and filtering data on Google Cloud Storage requires the fee-based usage of
<a href="https://cloud.google.com/bigquery/external-data-cloud-storage?hl=en">BigQuery</a>, which
is currently not supported by <em>i.sentinel.download</em>. Instead, when using the <b>datasource=GCS</b>
option, querying is performed using the <a href="https://scihub.copernicus.eu/">Copernicus Open Access Hub</a>
interface, while only the download step makes use of Google Cloud Storage (download links are
automatically generated from the respective Sentinel-2 identifier). As a consequence, a <b>settings</b> file
with valid credentials for the Copernicus Open Access Hub is required for downloading
from Google Cloud Storage in <em>i.sentinel.download</em>.


<h3>Credentials file</h3>
<p><em>i.sentinel.download</em> reads the user credentials
from the <b>settings</b> file. The file must contain at least two lines:
Expand Down Expand Up @@ -156,8 +181,8 @@ <h3>Long Term Archive (LTA)</h3>

<p>
As an alternative to triggering re-upload Sentinel-2 data from the LTA, scenes can be downloaded
from the <b>USGS Earth Explorer</b> where all data remain online and directly accessible.
However, this applies only to Sentinel-2 S2MSI1C (Level-1C) data.
from <b>USGS Earth Explorer</b> or <b>Google Cloud Storage</b> where all data remain online and directly accessible.
However, this applies only to Sentinel-2 data in levels 1C (USGS and GCS) and 2A (only GCS).

<h2>EXAMPLES</h2>

Expand Down Expand Up @@ -286,6 +311,11 @@ <h3>Download Sentinel products</h3>
i.sentinel.download settings=credentials.txt producttype=S2MSI2A start=2018-05-01 end=2018-05-31 limit=1 output=s2_L2A_may2018/
</pre></div>

Download first (<b>limit=1</b>) S2MSI2A product found from <b>Google Cloud Storage</b>:
<div class="code"><pre>
i.sentinel.download settings=credentials.txt producttype=S2MSI2A start=2018-05-01 end=2018-05-31 limit=1 output=s2_L2A_may2018/ datasource=GCS
</pre></div>

The downloaded Sentinel data can subsequently be easily imported into GRASS GIS
using <em><a href="i.sentinel.import.html">i.sentinel.import</a></em>
module.
Expand All @@ -298,6 +328,12 @@ <h3>Download Sentinel products by UUID</h3>
i.sentinel.download settings=credentials.txt uuid=6a10da0f-7777-4818-bc2c-4680e82297ac output=s2_data/
<pre></div>

<p>
Example of downloading a single Sentinel product by UUID from <b>Google Cloud Storage</b>
<div class="code"><pre>
i.sentinel.download datasource=GCS settings=credentials.txt uuid=6a10da0f-7777-4818-bc2c-4680e82297ac output=s2_data/
<pre></div>

<p>
Example of downloading a single Sentinel product by UUID from the <b>USGS Earth Explorer</b>:

Expand Down Expand Up @@ -343,9 +379,18 @@ <h3>Download Sentinel products by scene name</h3>
producttype=S2MSI2A start=2021-01-01 end=2021-02-15 output=s2_data/
<pre></div>

<p>
Example of downloading a single Sentinel product by scene name from
<b>Google Cloud Storage</b>:
<div class="code"><pre>
i.sentinel.download datasource=GCS start=2021-01-01 end=2021-02-15 producttype=S2MSI2A \
query="filename=S2B_MSIL2A_20210120T155559_N0214_R054_T17SPV_20210120T201821.SAFE" settings=credentials.txt \
output=s2_data/
<pre></div>

<p>
Example of downloading a single Sentinel product from <b>USGS Earth Explorer</b>
by the USGS-identifier.
by the USGS-identifier:
<div class="code"><pre>
i.sentinel.download datasource=USGS_EE start=2017-09-01 end=2017-12-01 producttype=S2MSI1C \
query="usgs_identifier=L1C_T32ULA_A003180_20171015T104525" settings=credentials.txt \
Expand All @@ -367,7 +412,7 @@ <h2>REQUIREMENTS</h2>
<ul>
<li><a href="https://pypi.python.org/pypi/sentinelsat">Sentinelsat library</a></li>
<li><a href="https://pypi.python.org/pypi/pandas">Pandas library</a></li>
<li><a href="https://pypi.python.org/pypi/landsatxplore">landsatxplore library</a></li>
<li><a href="https://pypi.python.org/pypi/landsatxplore">landsatxplore library</a></li> for downloading from USGS Earth Explorer
</ul>

<h2>SEE ALSO</h2>
Expand Down Expand Up @@ -402,7 +447,7 @@ <h2>AUTHOR</h2>
Lab</a>, CTU in Prague, Czech Republic with support
of <a href="https://opengeolabs.cz/en/home/">OpenGeoLabs</a> company
<br>
Guido Riembauer, <a href="https://www.mundialis.de/">mundialis</a> (USGS provider support)
Guido Riembauer, <a href="https://www.mundialis.de/">mundialis</a> (USGS and GCS provider support)
<!--
<p>
<i>Last changed: $Date$</i>
Expand Down