i.sentinel.download: Enable downloading from Google Cloud

[griembauer]

While Querying Sentinel-2 data from Google Cloud (https://cloud.google.com/storage/docs/public-datasets/sentinel-2) requires a google account and a GCP project (not free), the Sentinel-2 data themselves are freely available. They can be accessed by following the folder structure. Although it is not possible to download an entire .SAFE-folder, all individual files can be downloaded and put together to a complete S2-scene.

This PR enables i.sentinel.download to download from GCS. Searching and filtering scenes still requires an ESA-login.

[veroandreo]

This is great addition @griembauer! Maybe then we can do something similar for i.landsat ;-)

[griembauer]

Thanks @veroandreo! It is still a draft though and has many open things to test, revise, and add ;)
I can gladly help to add this functionality to i.landsat too - it should be actually more straightforward for landsat, because the data don't have this nested folder structure that S-2 has (with GRANULE etc.), but are all on one level (e.g. here).

[veroandreo]

One question: do you also download the metadata file or read it on the fly somehow? That info there in then needed for i.sentinel.mask

[griembauer]

One question: do you also download the metadata file or read it on the fly somehow? That info there in then needed for i.sentinel.mask

Yes, the current draft makes use of the manifest.safe file of each Sentinel-2 scene. This manifest.safe contains relative paths to all other individual files that belong to the respective S-2 scene, including the metadata file, e.g.:

[...]
  <dataObjectSection>
    <dataObject ID="INSPIRE_Metadata">
      <byteStream mimeType="text/xml" size="880">
        <fileLocation locatorType="URL" href="./INSPIRE.xml"/>
        <checksum checksumName="MD5">4eebb822481179ed0aa75f042b6aad04</checksum>
      </byteStream>
    </dataObject>
    <dataObject ID="S2_Level-2A_Product_Metadata">
      <byteStream mimeType="text/xml" size="892">
        <fileLocation locatorType="URL" href="./MTD_MSIL2A.xml"/>
        <checksum checksumName="MD5">22194e6718b7f1c8f5895315d106fbc1</checksum>
      </byteStream>
    </dataObject>
[...]
    <dataObject ID="IMG_DATA_10m_Band4_Tile1_Data">
      <byteStream mimeType="application/octet-stream" size="1192">
        <fileLocation locatorType="URL" href="GRANULE/L2A_T31UCR_A006312_20180522T111119/IMG_DATA/R10m/T31UCR_20180522T111049_B08_10m.jp2"/>
        <checksum checksumName="MD5">349e77e866fd03bba0853bb88d2070b5</checksum>
      </byteStream>
    </dataObject>
[...]

With this information all required files/folders for download are identified and downloaded. However sometimes there are some inconsistencies in themanifest.safe which I am currently trying to catch.

[neteler]

FYI, here some background into to this weird Sentinel-2 L2A "PHOEBUS" manifest.safe mess (which apparently affected L2A data between 22 May 2018 and 8 October 2018): https://forum.step.esa.int/t/when-did-l2a-manifest-safe-format-change/15637

[veroandreo]

It's really fast! We need it in i.landsat too :)

My only comment is about use of double quotes in messages, just run black and it will change it for you