Add draft sections on openEO, STAC, COG, and gdalcubes #784

appelmar · 2022-04-28T12:14:48Z

This is a first draft for two sections on openEO, STAC, COG, and gdalcubes (see #755). Let me know if you have any ideas for improvement, find this too long, or miss anything.

For now, I've moved both sections into a larger section on "Bridges to cloud technologies and services", which perhaps would also relate to the upcoming rgee section?!

The openEO section does not include a code example but happy to add one if you think this is helpful.

Robinlovelace · 2022-04-29T08:33:43Z

10-gis.Rmd

@@ -14,6 +14,8 @@ library(terra)
 library(qgisprocess)
 library(Rsagacmd)
 library(rgrass7)
+library(rstac)


👍 will need to add to the Suggests in geocompkg

Robinlovelace · 2022-04-29T08:33:54Z

10-gis.Rmd

@@ -93,7 +95,7 @@ This chapter focuses on 'bridges' to three mature open source GIS products (see
 One can also use R scripts from within QGIS\index{QGIS} (see https://docs.qgis.org/3.16/en/docs/training_manual/processing/r_intro.html).
 Finally, it is also possible to use R from the GRASS GIS\index{GRASS} command line (see https://grasswiki.osgeo.org/wiki/R_statistics/rgrass7).
 ]
-To complement the R-GIS bridges, the chapter ends with a very brief introduction to interfaces to spatial libraries (Section \@ref(gdal)) and spatial databases\index{spatial database} (Section \@ref(postgis)).
+To complement the R-GIS bridges, the chapter ends with a very brief introduction to interfaces to spatial libraries (Section \@ref(gdal)), spatial databases\index{spatial database} (Section \@ref(postgis)), and  cloud-based processing of Earth observation data (Section \@ref(cloud)).


Robinlovelace · 2022-04-29T08:34:21Z

10-gis.Rmd

+
+
+### STAC, COGs, and data cubes in the cloud
+Major cloud computing providers (Amazon Web Services, Microsoft Azure / Planetary Computer, Google Cloud Platform, and others) offer huge catalogs of open Earth observation data such as the complete Sentinel-2 archive on their platforms. We can use R and directly connect to and process data from these archives, ideally from a machine in the same cloud and region.


Great intro.

Robinlovelace · 2022-04-29T08:34:38Z

10-gis.Rmd

+### STAC, COGs, and data cubes in the cloud
+Major cloud computing providers (Amazon Web Services, Microsoft Azure / Planetary Computer, Google Cloud Platform, and others) offer huge catalogs of open Earth observation data such as the complete Sentinel-2 archive on their platforms. We can use R and directly connect to and process data from these archives, ideally from a machine in the same cloud and region.
+
+Three promising developments that make working with such image archives on cloud platforms _easier_ and _more efficient_ are the [SpatioTemporal Asset Catalog (STAC)](https://stacspec.org), the [cloud-optimized GeoTIFF (COG)](https://www.cogeo.org/) image file format, and the concept of data cubes. Below, we introduce these individual developments and briefly describe how they can be used from R. 


Robinlovelace · 2022-04-29T08:35:08Z

10-gis.Rmd

+# Connect to the STAC-API endpoint for Sentinel-2 data
+# and search for images intersecting our AOI
+s = stac("https://earth-search.aws.element84.com/v0")
+items = s |>


Looks good, note to self we need to switch to base pipe soon #616

Base pipe FTW. (https://r4ds.hadley.nz/workflow-pipes.html)

Robinlovelace · 2022-04-29T08:35:46Z

Many thanks for this @appelmar, looking great!

Nowosad · 2022-04-29T09:54:20Z

Thank you a lot, Marius. I will review it in the next few days.

appelmar · 2022-04-29T10:27:36Z

Thanks, looking forward to discuss and improve.

Nowosad

@appelmar great sections: dense, informative, but also easy to follow.

I only added some small comments. One additional thing -- can you split each sentence into a separate line? We are using the "one line per sentence" approach in the geocompr book.

Nowosad · 2022-05-02T07:25:31Z

10-gis.Rmd

+
+Three promising developments that make working with such image archives on cloud platforms _easier_ and _more efficient_ are the [SpatioTemporal Asset Catalog (STAC)](https://stacspec.org), the [cloud-optimized GeoTIFF (COG)](https://www.cogeo.org/) image file format, and the concept of data cubes. Below, we introduce these individual developments and briefly describe how they can be used from R. 
+
+The SpatioTemporal Asset Catalog (STAC) is a general description format for spatiotemporal data that is used to describe a variety of datasets on cloud platforms including imagery, SAR data, and point clouds. Besides simple static catalog descriptions, STAC-API presents a web service to query items (e.g. images) of catalogs by space, time, and other properties. In R, the **rstac** package [@simoes_rstac_2021] allows to connect to STAC-API endpoints and search for items. In the example below, we request all images from the [Sentinel-2 Cloud-Optimized GeoTIFF (COG) dataset on Amazon Web Services](https://registry.opendata.aws/sentinel-2-l2a-cogs) that intersect with a predefined area and time of interest. The result contains all found images and their metadata and URLs pointing to actual files on AWS. 


@appelmar consider explaining the SAR abbreviation (we are not using SAR data in the book)

Do you know of any list of the most popular STAC-APIs? We, for example, could mention them in chapter 8.

https://stacindex.org/ might be helpful. It is possible to search for public APIs only.

Nowosad · 2022-05-02T07:27:55Z

10-gis.Rmd

+
+
+### STAC, COGs, and data cubes in the cloud
+Major cloud computing providers (Amazon Web Services, Microsoft Azure / Planetary Computer, Google Cloud Platform, and others) offer huge catalogs of open Earth observation data such as the complete Sentinel-2 archive on their platforms. We can use R and directly connect to and process data from these archives, ideally from a machine in the same cloud and region.


@appelmar could you add some indexes to your new sections (\index{} -- see examples of its use in the previous chapters)

I've added some indexes for GDAL, STAC, COG, data cube, and cloud computing.

Nowosad · 2022-05-02T07:28:23Z

10-gis.Rmd

+
+For larger areas of interest, requested images are still relatively difficult to work with: they may use different map projections, may spatially overlap, and the spatial resolution often depends on the spectral band. The **gdalcubes** package [@appel_gdalcubes_2019] can be used to abstract from individual images and to create and process image collections as four-dimensional data cubes.
+
+The example below shows a minimal example to create a lower resolution (250m) maximum NDVI composite from the Sentinel-2 images returned by the previous STAC-API search.


"The example below shows a minimal example" -> "The code below shows a minimal example"?

Thanks, fixed!

Nowosad · 2022-05-02T07:28:47Z

10-gis.Rmd

+library(gdalcubes)
+
+# Create an image collection object from STAC items
+collection = stac_image_collection(items$features, property_filter = function(x) {x[["eo:cloud_cover"]] < 10})


Can you shortly explain the x[["eo:cloud_cover"]] < 10 code?

The comment above now states "Filter images from STAC response by cloud cover and create an image collection object". Is this understandable or do you think I should mention some details how the function is applied in the text?

This is the only line of code in your sections that I do not understand (I do not know where did "eo:cloud_cover" came from...). It would be great if you could add a short explanation in the text.

Thanks, I've added a short explanation (see here) after the code chunk. Does this help to understand the function?

Nowosad · 2022-05-02T07:29:01Z

10-gis.Rmd

+collection = stac_image_collection(items$features, property_filter = function(x) {x[["eo:cloud_cover"]] < 10})
+
+# Define spatiotemporal extent, resolution and CRS of the target data cube
+v = cube_view(srs = "EPSG:3857", extent = collection, dx = 250, dy = 250, dt = "P1D")


What is P1D?

It's an ISO 8601 duration string, I've added this as a comment.

Nowosad · 2022-05-02T07:31:28Z

10-gis.Rmd

+raster_cube(collection, v) |>
+  select_bands(c("B04", "B08")) |>
+  apply_pixel("(B08-B04)/(B08+B04)", "NDVI") |>
+  reduce_time("max(NDVI)") -> cube


We are not using forward pipes in the book. Can you replace it with =?

Nowosad · 2022-05-02T07:32:42Z

10-gis.Rmd

+
+### openEO
+
+Besides hosting large data archives, numerous cloud-based services to process Earth observation data have been launched during the last years. OpenEO [@schramm_openeo_2021] is an initiative to support interoperability among cloud services by defining a common language for processing the data. The initial idea has been described in an [r-spatial.org blog post](https://r-spatial.org/2016/11/29/openeo.html) and aims at making it possible for users to change between cloud services easily with as little code changes as possible. The [standardized processes](https://processes.openeo.org) use a multidimensional data cube model as an interface to the data. Implementations are available for eight different backends (see https://hub.openeo.org) to which users can connect with R, Python, JavaScript, QGIS, or a web editor and define (and chain) processes on collections. Since the functionality and data availability differs among the backends, the **openeo** R package [@lahn_openeo_2021] dynamically loads available processes and collections from the connected backend. Afterwards, users can load image collections, apply and chain processes, submit jobs, and explore and plot results. 


@appelmar can you think of a small code example for this section? I think it would be great to see openeo in action (if possible).

AFAIK, all available services require a user account. Do you think it would be useful to include an example without connecting to a specific backend?

Yes -- it could help people to do their first steps (after getting the user account).

I've added an example, hope this is useful and easy to understand?

…om/Robinlovelace/geocompr/pull/784)

appelmar · 2022-05-02T12:37:16Z

Thanks, Jakub and Robin for your comments and improvements. The latest commit should now follow the "one line per sentence" approach and address the other comments!

…ps://github.com/Robinlovelace/geocompr/pull/784)

Nowosad · 2022-05-03T06:59:31Z

Great work, @appelmar.
@Robinlovelace -- can we merge this PR?

Robinlovelace · 2022-05-07T18:33:07Z

@Robinlovelace -- can we merge this PR?

Was just about to ask the same question, apologies for missing the question! Yes for sure, many thanks @appelmar and Jakub for the detailed review. And feel free to share it widely when the new section builds 🤞

…om/Robinlovelace/geocompr/pull/784)

…xample (see https://github.com/Robinlovelace/geocompr/pull/784) ad980df

add draft sections on openEO, STAC, COG, and gdalcubes

43a10cb

Robinlovelace approved these changes Apr 29, 2022

View reviewed changes

Nowosad reviewed May 2, 2022

View reviewed changes

revise sections on openEO, STAC, COG, and gdalcubes (https://github.c…

6401690

…om/Robinlovelace/geocompr/pull/784)

add openEO example, improve descirption of gdalcubes example (see htt…

2a20310

…ps://github.com/Robinlovelace/geocompr/pull/784)

Nowosad approved these changes May 3, 2022

View reviewed changes

Robinlovelace merged commit ad980df into geocompx:main May 7, 2022

Robinlovelace referenced this pull request May 7, 2022

revise sections on openEO, STAC, COG, and gdalcubes (https://github.c…

26a8d60

…om/Robinlovelace/geocompr/pull/784)

github-actions bot referenced this pull request May 7, 2022

Deploy commit: add openEO example, improve descirption of gdalcubes e…

b0e4e47

…xample (see https://github.com/Robinlovelace/geocompr/pull/784) ad980df

github-actions bot referenced this pull request in defuneste/geocompr May 7, 2022

Deploy commit: add openEO example, improve descirption of gdalcubes e…

6ebd44f

…xample (see https://github.com/Robinlovelace/geocompr/pull/784) ad980df

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add draft sections on openEO, STAC, COG, and gdalcubes #784

Add draft sections on openEO, STAC, COG, and gdalcubes #784

appelmar commented Apr 28, 2022

Robinlovelace Apr 29, 2022

Robinlovelace Apr 29, 2022

Robinlovelace Apr 29, 2022

Robinlovelace Apr 29, 2022

Robinlovelace Apr 29, 2022

Nowosad Apr 29, 2022

Robinlovelace commented Apr 29, 2022

Nowosad commented Apr 29, 2022

appelmar commented Apr 29, 2022

Nowosad left a comment

Nowosad May 2, 2022

Nowosad May 2, 2022

appelmar May 2, 2022

Nowosad May 2, 2022

appelmar May 2, 2022

Nowosad May 2, 2022

appelmar May 2, 2022

Nowosad May 2, 2022

appelmar May 2, 2022

Nowosad May 2, 2022

appelmar May 2, 2022

Nowosad May 2, 2022

appelmar May 2, 2022

Nowosad May 2, 2022

appelmar May 2, 2022

Nowosad May 2, 2022

appelmar May 2, 2022

Nowosad May 2, 2022

appelmar May 2, 2022

appelmar commented May 2, 2022

Nowosad commented May 3, 2022

Robinlovelace commented May 7, 2022



		### STAC, COGs, and data cubes in the cloud
		Major cloud computing providers (Amazon Web Services, Microsoft Azure / Planetary Computer, Google Cloud Platform, and others) offer huge catalogs of open Earth observation data such as the complete Sentinel-2 archive on their platforms. We can use R and directly connect to and process data from these archives, ideally from a machine in the same cloud and region.


		Three promising developments that make working with such image archives on cloud platforms _easier_ and _more efficient_ are the [SpatioTemporal Asset Catalog (STAC)](https://stacspec.org), the [cloud-optimized GeoTIFF (COG)](https://www.cogeo.org/) image file format, and the concept of data cubes. Below, we introduce these individual developments and briefly describe how they can be used from R.

		The SpatioTemporal Asset Catalog (STAC) is a general description format for spatiotemporal data that is used to describe a variety of datasets on cloud platforms including imagery, SAR data, and point clouds. Besides simple static catalog descriptions, STAC-API presents a web service to query items (e.g. images) of catalogs by space, time, and other properties. In R, the rstac package [@simoes_rstac_2021] allows to connect to STAC-API endpoints and search for items. In the example below, we request all images from the [Sentinel-2 Cloud-Optimized GeoTIFF (COG) dataset on Amazon Web Services](https://registry.opendata.aws/sentinel-2-l2a-cogs) that intersect with a predefined area and time of interest. The result contains all found images and their metadata and URLs pointing to actual files on AWS.


		For larger areas of interest, requested images are still relatively difficult to work with: they may use different map projections, may spatially overlap, and the spatial resolution often depends on the spectral band. The gdalcubes package [@appel_gdalcubes_2019] can be used to abstract from individual images and to create and process image collections as four-dimensional data cubes.

		The example below shows a minimal example to create a lower resolution (250m) maximum NDVI composite from the Sentinel-2 images returned by the previous STAC-API search.


		### openEO

		Besides hosting large data archives, numerous cloud-based services to process Earth observation data have been launched during the last years. OpenEO [@schramm_openeo_2021] is an initiative to support interoperability among cloud services by defining a common language for processing the data. The initial idea has been described in an [r-spatial.org blog post](https://r-spatial.org/2016/11/29/openeo.html) and aims at making it possible for users to change between cloud services easily with as little code changes as possible. The [standardized processes](https://processes.openeo.org) use a multidimensional data cube model as an interface to the data. Implementations are available for eight different backends (see https://hub.openeo.org) to which users can connect with R, Python, JavaScript, QGIS, or a web editor and define (and chain) processes on collections. Since the functionality and data availability differs among the backends, the openeo R package [@lahn_openeo_2021] dynamically loads available processes and collections from the connected backend. Afterwards, users can load image collections, apply and chain processes, submit jobs, and explore and plot results.

Add draft sections on openEO, STAC, COG, and gdalcubes #784

Add draft sections on openEO, STAC, COG, and gdalcubes #784

Conversation

appelmar commented Apr 28, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Robinlovelace commented Apr 29, 2022

Nowosad commented Apr 29, 2022

appelmar commented Apr 29, 2022

Nowosad left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

appelmar commented May 2, 2022

Nowosad commented May 3, 2022

Robinlovelace commented May 7, 2022