support reading/loading raw cloud-optimized geotiff

[landryb]

Should be more in geosolutions-it/Mapstore2 project but oh well..

nowadays, the new trend to serve background layers is cogeo (cf https://www.cogeo.org/) - using https://geotiffjs.github.io/ it should be doable in Mapstore2 to have a new layer type allowing to load a remote cloud-optimized geotiff directly in the browser (an example at https://geotiffjs.github.io/cog-explorer/), only giving an url (and eventual credentials ?) - no more WMS, no more caching.. a dream come true ?

[fvanderbiest]

Depends on openlayers/openlayers#11810 ... or openlayers/openlayers#12008 ;-)

[landryb]

hah, i thought more about cramming smth on top of the existing viewer modes, but right if that comes for free with an openlayers/leaflet upgrade..

[landryb]

cf mviewer/mviewer#698 too :)

[landryb]

as seen in the mviewer issue above and the progress done by @Gaetanbrl to support COG in mviewer, this requires OL >= 7.1.1.

[landryb]

@tdipisa: what is expected for the estimation of this issue is:

• update openlayers to a version that supports COG
• only for the openlayers map type
• a new data type (named COG or COGEO?), with two fields:
  • a way to load a COG from a given URL
  • a way to specify its native SRS (which might be different from the map projection, so support needs to handle reprojection of the raster via proj)
• nothing more for now (eg no identify/thumbnail/styling/export support)

we have plenty of EPSG:2154 COGs available online for testing

[tdipisa]

Dear @landryb the aim of the POC I've suggested for this is to well evaluate the integration of COG in MapStore. Since it is a completely new element, under a certain extent of course, it is not easy to foresee all the possibilities until we start the implementation. That's the same approach we have followed for the 3D Tiles implementation that allowed us to integrate it and better understand possibilities to evolve it in a good way: what we have done effectively in 2022.
Of course, the aim of the POC is to do a first implementation by collecting/evaluating also elements for further steps. The more days we have for this the better as you can imagine, but for what we have estimated we can effectively consider the following:

• update openlayers to a version that supports COG

yes

• only for the openlayers map type

yes

• a new data type (named COG or COGEO?), with two fields:

yes

• a way to load a COG from a given URL

yes

• a way to specify its native SRS (which might be different from the map projection, so support needs to handle reprojection of the raster via proj)

To evaluate, but I think it should be possible

• nothing more for now (eg no identify/thumbnail/styling/export support)

I cannot guarantee for further things for now but, given our estimates, we will evaluate what can be included within them and we will do that if possible

we have plenty of EPSG:2154 COGs available online for testing

That's really useful of course. Thank you for sharing when needed.

[tdipisa]

@catmorales @landryb after our yesterday's exchanges I can confirm that these two points are included in our estimates:

Use of COG layers as background layers is the main requirement for the implementation

A way to specify its native SRS (which might be different from the map projection, so support needs to handle reprojection of the raster via proj)

[landryb]

thanks for the confirmation - perfect !

[landryb]

i thought about the projection bit, and the necessity or not to add a field in the UI so that the user/data admin can specify the projection of the remote COG.

A proper COG file is expected to have a valid projection in the tiff header, which should be considered the ultimate source of truth for the projection.

I don't think there are many COG files without projection around (or they might be considered 'invalid COG files'?), for example QGIS only allows to load a COG from an URL without specifying its projection in the raster add from url dialog:

so Mapstore should read the projection from the COG, and take it into account to eventually reproject raster data on the fly if it's different from the viewer projection. Pretty sure that's what https://geotiffjs.github.io/cog-explorer/ does.

[landryb]

i guess made possible with the openlayers update in geosolutions-it/MapStore2#9293

[dsuren1]

MS issue ref:
geosolutions-it/MapStore2#9320
geosolutions-it/MapStore2#9469

[landryb]

for ms2-georchestra users: to test this feature, don't forget to add some entries to projectionDefs in localConfig.json for the COGs you plan to load,if they're in non-default projections (eg 2154, 394x...)

[dsuren1]

@landryb Of course and btw, these projectionDefs already in ms2-georchestra including ms2 defaults. Kindly let me know if something is missing that needs to be added.

EPSG:2154, EPSG:3942, EPSG:3943, EPSG:3944, EPSG:3945, EPSG:3946, EPSG:3947, EPSG:3948, EPSG:3949, EPSG:3950.

[landryb]

@landryb Of course and btw, these projectionDefs already in ms2-georchestra including ms2 defaults. Kindly let me know if something is missing that needs to be added.

EPSG:2154, EPSG:3942, EPSG:3943, EPSG:3944, EPSG:3945, EPSG:3946, EPSG:3947, EPSG:3948, EPSG:3949, EPSG:3950.

oh you're right, hadnt checked that those projections definitions were present in ms2-geor's localConfig.json (https://github.com/georchestra/mapstore2-georchestra/blob/master/configs/localConfig.json#L53), and since i was doing my testing in bare ms2 that's why i had missing projection defs. so it should be fine by default in ms2-geor :)

[landryb]

@tdipisa i'm not sure if feedback should be in this issue (since its closed) or a different one, but:

• loading https://cogeo.craig.fr/opendata/ortho/2019_20cm_rvb.tif (which is 200Go) fetches 200mo according to the browser console. that's a bit heavy, and i dunno if that's "expected".
• same thing for https://cogeo.craig.fr/opendata/ortho/orthocraig20_auvergne%2Bloire_2016.cog.tif which is 250Go, ~250Mb is fetched for the initial display.
• @jusabatier has been seeing the same but with much longer loading times, probably because it takes longer for him to fetch 200Mb

i've also noted that the catalog add COG dialog refuses an url that contains percent-encoded chars. the above url will fail, but works if i replace %2B by its original +. Will file an issue for that.

[jusabatier]

I think I solved my problem :

• My first COG used BLOCKSIZE=256, so I had very big IFD's list size in headers (543Mo)
• Trying with BLOCKSIZE=1024 and it load faster in MS2 as IFD's list size is drastically reduced (34Mo)
• I'm trying to generate another with BLOCKSIZE=4096 to compare if it improve again response time

[landryb]

i think i've used BLOCKSIZE=512 for all my COGs as iirc that's what is advised