Support reading/loading of raw cloud-optimized geotiff (COG) #9320

dsuren1 · 2023-08-02T13:17:22Z

Description

Requirement is support a new layer type to allow loading remote cloud-optimized geotiff (COG). The OL >v7 supports the new GeoTIFF source. So does MapStore as of #9293.

Acceptance criteria

MapStore updated to OL that supports GeoTIFF
Support only Openlayer as of this issue
Type of the layer is COG
The configuration should accept url source to load the COG. Support for Catalog tool must be also included as part of this issue
Projection data should be read from tiff source by MapStore to reproject raster data. (Check during implementation) (With latest OL this is already handled by OL. Sandbox Example
Use of COG layers as background layers
No further support requested as part of this issue. Therefore support Identify, Styling, Export etc are not mandatory for this implementation. To evaluate during the implementation if something can anyway included with small effort
Support for cog layer to be added from catalog
Exclude COG layers when printing

Configuration (example)

{
  "type": "cog",
  "title": "Cloud optimized geotiff",
  "name": "COG exmaple",
  "visibility": false,
  "sources": [{
    "url": "https://cogeo.craig.fr/opendata/ortho/ortho25_cc46_clerco_1996.cog.tif"
  }],
  "group": "background"
}

GeoTIFF example

import GeoTIFF from 'ol/source/GeoTIFF.js';
import TileLayer from 'ol/layer/WebGLTile.js';

const source = new GeoTIFF({
  sources: [
    {
      url: 'something.tif',
    },
  ],
});
const layer = new TileLayer({
      source: source,
})

Catalog:
Support for cog layer to be added from catalog. Add multiple urls via comma separated, which each url corresponds to individual layer i.e same structure above is to be maintained .

Layer settings
The default layer settings to be maintained.
Note: Check if visibility limits works.

Printing
Filter COG layers and background while printing. (Should be avoided in preview as well)

Other useful information

Ref: georchestra/mapstore2-georchestra#338

Source for geotiff tryouts: https://geotiffjs.github.io/cog-explorer

Questions

Should we support multiple sources as urls? Not mandatory for this issue
Should we provide configuration for color banding? Not mandatory for this issue
Authentication on protected COG url? Not mandatory for this issue
Thumbnail Not mandatory for this issue. But check if we can configure from background selector and it is simple enough to implement

Check

Save the map with cog layer
Permalink the map with cog layer

The text was updated successfully, but these errors were encountered:

landryb · 2023-08-02T13:45:10Z

for testing you can use:

those url return an error message if you click on it (to prevent full fetching) but they open fine as rasters urls in qgis. And they should work in https://geotiffjs.github.io/cog-explorer

EDIT: i have a working example in https://cogeo.craig.fr/viewer/ if you load the URLs in the top field (not the samples) they're displayed.

dsuren1 · 2023-08-10T12:05:38Z

@landryb Kindly confirm if I can use this for testing https://cogeo.craig.fr/opendata/ortho/2018_25cm_ain+isere.tif instead. As the above ones consume lots of bandwidth when loading the layer and takes longer time to load which makes it not suitable for testing and to work with when it needs to be loaded often during implementation.

If possible can you also provide some more urls that is considerably smaller is size with projection required for geOrchestra. Thanks!

landryb · 2023-08-10T12:47:45Z

you can use this one if you prefer yes (it's 'only' 65Gb), but ultimately the idea is to be able to use large ones.. and all the COGs i have are all using EPSG:2154 for the projection.

i've added https://cogeo.craig.fr/opendata/ortho/orthocraig3_vichy_2021.cog.tif which is a 3cm resolution image on the city of vichy, the file is less than 9Gb and loads straight away in QGIS

$gdalinfo /vsicurl/https://cogeo.craig.fr/opendata/ortho/orthocraig3_vichy_2021.cog.tif
Files: /vsicurl/https://cogeo.craig.fr/opendata/ortho/orthocraig3_vichy_2021.cog.tif
Size is 140000, 170000
Coordinate System is:
PROJCRS["RGF93 v1 / Lambert-93",
...
Origin = (731000.000000000000000,6562000.000000000000000)
Pixel Size = (0.030000000000000,-0.030000000000000)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  COMPRESSION=YCbCr JPEG
  INTERLEAVE=PIXEL
  JPEGTABLESMODE=1
  JPEG_QUALITY=90
  LAYOUT=COG
  SOURCE_COLOR_SPACE=YCbCr
Corner Coordinates:
Upper Left  (  731000.000, 6562000.000) (  3d24' 6.13"E, 46d 9'25.60"N)
Lower Left  (  731000.000, 6556900.000) (  3d24' 4.92"E, 46d 6'40.27"N)
Upper Right (  735200.000, 6562000.000) (  3d27'22.05"E, 46d 9'24.86"N)
Lower Right (  735200.000, 6556900.000) (  3d27'20.68"E, 46d 6'39.53"N)
Center      (  733100.000, 6559450.000) (  3d25'43.44"E, 46d 8' 2.58"N)
Band 1 Block=512x512 Type=Byte, ColorInterp=Red
  Overviews: 70000x85000, 35000x42500, 17500x21250, 8750x10625, 4375x5312, 2187x2656, 1093x1328, 546x664, 273x332
  Mask Flags: PER_DATASET 
  Overviews of mask band: 70000x85000, 35000x42500, 17500x21250, 8750x10625, 4375x5312, 2187x2656, 1093x1328, 546x664, 273x332
Band 2 Block=512x512 Type=Byte, ColorInterp=Green
  Overviews: 70000x85000, 35000x42500, 17500x21250, 8750x10625, 4375x5312, 2187x2656, 1093x1328, 546x664, 273x332
  Mask Flags: PER_DATASET 
  Overviews of mask band: 70000x85000, 35000x42500, 17500x21250, 8750x10625, 4375x5312, 2187x2656, 1093x1328, 546x664, 273x332
Band 3 Block=512x512 Type=Byte, ColorInterp=Blue
  Overviews: 70000x85000, 35000x42500, 17500x21250, 8750x10625, 4375x5312, 2187x2656, 1093x1328, 546x664, 273x332
  Mask Flags: PER_DATASET 
  Overviews of mask band: 70000x85000, 35000x42500, 17500x21250, 8750x10625, 4375x5312, 2187x2656, 1093x1328, 546x664, 273x332

dsuren1 · 2023-08-10T12:49:27Z

i've added https://cogeo.craig.fr/opendata/ortho/orthocraig3_vichy_2021.cog.tif which is a 3cm resolution image on the city of vichy, the file is less than 9Gb and loads straight away in QGIS

Thanks @landryb

landryb · 2023-08-10T12:52:59Z

and https://cogeo.craig.fr/opendata/ortho/ortho25_cc46_clerco_1996.cog.tif which is an EPSG:3946 1.2Gb file.

…tiff (COG)

offtherailz · 2023-09-20T12:48:12Z

Added backport needed label because we need to add it to 2023.02.xx that is already branched

* #9320 - Support reading/loading of cloud-optimized geotiff (COG) * Url validation modified * Update cog layer model * unit test

…tiff (COG) (geosolutions-it#9394) * geosolutions-it#9320 - Support reading/loading of cloud-optimized geotiff (COG) * Url validation modified * Update cog layer model * unit test (cherry picked from commit 920ff39)

…) (#9484) * #9320 - Support reading/loading of cloud-optimized geotiff (COG) * Url validation modified * Update cog layer model * unit test (cherry picked from commit 920ff39)

landryb · 2023-09-27T12:47:45Z

Fwiw i'm testing this on the tip of 2023.02.xx (on http://localhost:8081) and no image is displayed with the vichy COG. I've seen some hits on the server when i added the COG catalog entry, the layer is added to the toc but i have no image displayed, whatever the zoom level.

there's no zoom to layer but my understanding is that #9469 isnt backported yet so that's fine, but what more is needed ? testing with the default localconfig.json btw, but i dont see errors about CORS in the browser console.

dsuren1 · 2023-09-27T12:49:57Z

@landryb
Kindly make sure to add needed projectionDefs in localConfig.json and give it a try

landryb · 2023-09-27T13:02:02Z

@landryb Kindly make sure to add needed projectionDefs in localConfig.json and give it a try

thanks, that was the missing bit, with the definitions for EPSG:2154 & EPSG:3946 i can load the vichy and clerco cc46 COGs fine in the same viewer. I suppose that should be documented somewhere ? dunno if that should be in the user guide though (#9492 doesnt mention it)

anyway, great work on this feature, will do more testing (as backgrounds too) with the COGs i have here.

tdipisa · 2023-09-27T13:50:53Z

@landryb Kindly make sure to add needed projectionDefs in localConfig.json and give it a try

thanks, that was the missing bit, with the definitions for EPSG:2154 & EPSG:3946 i can load the vichy and clerco cc46 COGs fine in the same viewer. I suppose that should be documented somewhere ? dunno if that should be in the user guide though (#9492 doesnt mention it)

A reference will be included in the online documentation by @ElenaGallo

anyway, great work on this feature, will do more testing (as backgrounds too) with the COGs i have here.

Thank you so much

landryb · 2023-09-27T14:41:48Z

i havent been able to have a working default background using a cog layer, tried this the layers array in new.json per the first comment in #9394:

--- a/web/client/configs/new.json
+++ b/web/client/configs/new.json
@@ -118,6 +118,18 @@
                 "source": "osm",
                 "group": "background",
                 "visibility": false
+            }, {
+                "type": "cog",
+                "title": "Cloud optimized geotiff",
+                "name": "COG background example",
+                "url": [
+                  "https://cogeo.craig.fr/opendata/ortho/ortho25_cc46_clerco_1996.cog.tif"
+                ],
+                "sources" : [ {
+                  "url" : "https://cogeo.craig.fr/opendata/ortho/ortho25_cc46_clerco_1996.cog.tif"
+                } ],
+                "group": "background",
+                "visibility": false
             }, {
                 "source": "ol",
                 "group": "background",

what more is needed ? sources ?

dsuren1 · 2023-09-28T05:20:29Z

@landryb

{
  "type": "cog",
  "title": "Cloud optimized geotiff",
  "name": "COG exmaple",
  "visibility": false,
  "sources": [{
    "url": "https://cogeo.craig.fr/opendata/ortho/2018_25cm_ain+isere.tif"
  }],
  "group": "background"
}

This config is enough to visualize the COG layer as background. Please make note, that background will not zoom to extent. So I believe the background layer is added but the ones in the issues are small, when the map is zoomed to max extent, that it we may fail to notice it. Kindly check by zooming to extent that you are interested and then switching to COG background layer

landryb · 2023-09-28T07:52:24Z

i'll triple check as im probably doing something wrong, but with that section added to the 'layers' array in new.json:

visiting http://localhost:8081/#/viewer/new i see hits being done on the cog file in the network console, but zooming on clermont i have no data displayed, and there's no extra background in the list of backgrounds.

i must be doing smth wrong because i only have 6 backgrounds available when clicking on the background selector:

osm simple dark
osm simple light
osm bright
NE political
sentinel 2 cloudless
open street map

while i have 8 entries in the layers array - COG isnt available, nor is the empty one.

landryb · 2023-09-28T08:11:31Z

bah. i dont understand why, but with the EPSG:2154 ain+isere cog, testing on 2023.02.xx instead of my previous test with master, it is properly available as a background in the list, and displays fine.

dsuren1 added New Feature used for new functionalities C169-Rennes-Métropole-2023-GeOrchestra4 Phase 3 labels Aug 2, 2023

dsuren1 added this to the 2023.02.00 milestone Aug 2, 2023

dsuren1 self-assigned this Aug 2, 2023

dsuren1 mentioned this issue Aug 2, 2023

support reading/loading raw cloud-optimized geotiff georchestra/mapstore2-georchestra#338

Closed

dsuren1 added a commit to dsuren1/MapStore2 that referenced this issue Sep 6, 2023

geosolutions-it#9320 - Support reading/loading of cloud-optimized geo…

087ee4f

…tiff (COG)

dsuren1 mentioned this issue Sep 6, 2023

#9320 - Support reading/loading of cloud-optimized geotiff (COG) #9394

Merged

6 tasks

tdipisa linked a pull request Sep 7, 2023 that will close this issue

#9320 - Support reading/loading of cloud-optimized geotiff (COG) #9394

Merged

6 tasks

offtherailz added the BackportNeeded Commits provided for an issue need to be backported to the milestone's stable branch label Sep 20, 2023

tdipisa closed this as completed in #9394 Sep 25, 2023

tdipisa pushed a commit that referenced this issue Sep 25, 2023

#9320 - Support reading/loading of cloud-optimized geotiff (COG) (#9394)

920ff39

* #9320 - Support reading/loading of cloud-optimized geotiff (COG) * Url validation modified * Update cog layer model * unit test

ElenaGallo self-assigned this Sep 25, 2023

ElenaGallo mentioned this issue Sep 25, 2023

User Guide - Add COG catalog option #9482

Closed

2 tasks

dsuren1 mentioned this issue Sep 25, 2023

[Backport 2023.02.xx] #9320 - Support reading/loading of cloud-optimized geotiff (COG) (#9394) #9484

Merged

dsuren1 linked a pull request Sep 25, 2023 that will close this issue

[Backport 2023.02.xx] #9320 - Support reading/loading of cloud-optimized geotiff (COG) (#9394) #9484

Merged

tdipisa removed the BackportNeeded Commits provided for an issue need to be backported to the milestone's stable branch label Sep 25, 2023

ElenaGallo added the Accepted label Sep 26, 2023

ElenaGallo mentioned this issue Sep 27, 2023

Update User Guide - Add layer zoom to extent option #9502

Merged

landryb mentioned this issue Oct 2, 2023

support loading COG layers from query params #9527

Closed

landryb mentioned this issue Oct 20, 2023

add cog dialog refuses percent-encoded chars in URLs #9635

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support reading/loading of raw cloud-optimized geotiff (COG) #9320

Support reading/loading of raw cloud-optimized geotiff (COG) #9320

dsuren1 commented Aug 2, 2023 •

edited

landryb commented Aug 2, 2023 •

edited

dsuren1 commented Aug 10, 2023

landryb commented Aug 10, 2023

dsuren1 commented Aug 10, 2023

landryb commented Aug 10, 2023

offtherailz commented Sep 20, 2023 •

edited

landryb commented Sep 27, 2023

dsuren1 commented Sep 27, 2023 •

edited

landryb commented Sep 27, 2023

tdipisa commented Sep 27, 2023

landryb commented Sep 27, 2023

dsuren1 commented Sep 28, 2023

landryb commented Sep 28, 2023

landryb commented Sep 28, 2023

Support reading/loading of raw cloud-optimized geotiff (COG) #9320

Support reading/loading of raw cloud-optimized geotiff (COG) #9320

Comments

dsuren1 commented Aug 2, 2023 • edited

Description

Acceptance criteria

Configuration (example)

Other useful information

Questions

landryb commented Aug 2, 2023 • edited

dsuren1 commented Aug 10, 2023

landryb commented Aug 10, 2023

dsuren1 commented Aug 10, 2023

landryb commented Aug 10, 2023

offtherailz commented Sep 20, 2023 • edited

landryb commented Sep 27, 2023

dsuren1 commented Sep 27, 2023 • edited

landryb commented Sep 27, 2023

tdipisa commented Sep 27, 2023

landryb commented Sep 27, 2023

dsuren1 commented Sep 28, 2023

landryb commented Sep 28, 2023

landryb commented Sep 28, 2023

dsuren1 commented Aug 2, 2023 •

edited

landryb commented Aug 2, 2023 •

edited

offtherailz commented Sep 20, 2023 •

edited

dsuren1 commented Sep 27, 2023 •

edited