Proof of concept for integrating a WMS-only STAC collection

[j08lue]

We are trying out different approaches of integrating ArcGIS-served resources into VEDA. Using their WMS service could be one option and we could create a STAC collection in our catalog without items that links to the WMS/WMTS/whatnot service, to make these resources discoverable in our STAC and the UI more or less self-configured based on these collections.

• Ingest ArcGIS collection into VEDA dev catalog veda-backend#359 (comment)

The goal for this ticket would be to set up a proof of concept for this integration pattern. We could probably work off of @slesaad's POC against the Arc2STAC proxy, which probably only needs a few small changes since it was also using WMS.

e.g. look for the wms link instead of relying on the first links, etc.

https://github.com/NASA-IMPACT/veda-ui/pull/887/files#diff-51d9ee79e77cc94b891a168ca648f90b458b5f81ef12e235705881bc5af03ec0R141

We still need to confirm that this (STAC collection w/o items but with pointers to the services) is the integration pattern we will follow. Discussions with @anayeaye, @smohiudd, and @slesaad will follow up on that question shortly.

Related epics

• https://github.com/NASA-IMPACT/veda-architecture/issues/405

Acceptance criteria

• A POC is available that makes an ArcGIS-served resource discoverable and browsable in VEDA UI

[smohiudd]

@hanbyul-here this the test ArcGIS collection in the dev catalog: https://dev.openveda.cloud/api/stac/collections/POWER_901_MONTHLY_RADIATION_UTC

This collection should have enough metadata to get you started but we will need to discuss some changes:

• the collection has a "rel": "items" link that must be removed for these "no item" collections
• we may need to distinguish these special ArcGIS collections in the STAC metadata so the backend knows to remove certain links that don't pertain to these types of collections (i.e. item links). That meta data may also help the UI.
• Is there other metadata that would otherwise be in veda-config that we can offload the arcgis collections?

[smohiudd]

The collection above was derived from an ImageServer dataset.

Here is a MapServer dataset also for reference: https://dev.openveda.cloud/api/stac/collections/ESI_4WK

[j08lue]

@hanbyul-here and I were today trying to make sense of the parameters we would need to pass to the WMS services in GetMap requests, like layers, styles, etc. (see ArcGIS docs on WMS).

It was impossible to retrieve that information - on the ESI_4WK server, GetCapabilities is incomplete and since we do not know the available layers, also GetStyles will not work...

The gist: The layers and styles information is crucial for defining a WMS service URL to add to the map. It would be ideal if this info was available with the STAC collection, probably in the render extension.

[smohiudd]

@j08lue, @hanbyul-here thanks for the feedback. Are you having the same issues with the ImageServer collection: https://dev.openveda.cloud/api/stac/collections/POWER_901_MONTHLY_RADIATION_UTC?

@amarouane-ABDELHAK I believe what Jonas mentioned was what we were talking about last week and the limitation of MapServer. Is it possible to dynamically retrieve metadata concerning layers from MapServer?

[hanbyul-here]

@smohiudd I could make the ArcGIS Layer work by manually putting the configuration in MDX file. @slesaad helped me figure out what values I should use, but I probably couldn't figure it out alone. It would be nice to be able to get this info through STAC endpoint

[smohiudd]

@hanbyul-here if you outline here what metadata you need we can try and get it in STAC! One of our goals if to offload as much as we can from the mdx file if possible

[hanbyul-here]

These are the parameters that I had to pass to get the successful visualization layer for esi_4wk dataset!

        format: 'image/png',
        service: 'WMS',
        version: '1.3.0',
        request: 'GetMap',
        crs: 'EPSG:3857',
        transparent: 'true',
        width: '256',
        height: '256',
        styles: '',
        layers: 1

[hanbyul-here]

From the conversation with @amarouane-ABDELHAK & @slesaad , I could get more insights related to ArcGIS data.

datasets

• We will narrow down the datasets to these three: ESI 4WK, amphibian_richness_habitat30m (both mapserver - @slesaad can you confirm if amphibian_richness_habitat30m is the dataset without any timestamp), power_901_annual_meterology_utc (image server)
• ESI 4WK : timestamps are there, but it is not 'periodic' in dashboard's term. This data will be re-ingested with all the available timestamps

analysis feature and /statistics

It will make the most sense to include the analysis functionality for the demo since the time-series capacity was already shown in the previous demo. Both of the datasets will eventually have /statistics endpoint, that the dashboard won't need to tell the difference between the,

timestamps

We could discuss a bit more about the dataset that will miss the timestamps. I should check LOE to disable any tim- related UI components for the demo. The long-term solution should be resolved through #999

We also discussed how to depend on MDX files less for layers (and more on stac) but both agreed that we do not have to start this with ArcGIS Layer 😌

I will close this ticket as I address the errors in the currently opened PR: #991 and will move to the new issue dedicated for analysis #996

[slesaad]

@hanbyul-here here are the updated datasets:

power_power_901_annual_meterology_utc (imageserver)

bcb_amphibian_richness_habitat30m (mapserver, timeless) - note that the dashboard:is_timeless field is added and is true, and temporal extent is null

global_esi_4wk (mapserver, with timestamps) - added timestamps in summaries based on the 7 days time difference

Note

These are new collections with new ids