GEUS 1:500k geological map

[trey-stafford]

• Joe M. has a link to some data. It’s apparently really large but we could point users to it.

• Someone at GEUS noted making a QGIS compatible version of the dataset from Joe M.

[trey-stafford]

Paper for dataset: https://geusbulletin.org/index.php/geusb/article/view/4727/10367

It notes:

The new seamless map has been integrated in such a Web-GIS application which can be found at http://data.geus.dk/map2/geogreen.

However, attempting to access the given URL gives a "service unavailable" page (HTTP 503)

It also notes:

The seamless map is furthermore an important GEUS contribution to the global OneGeology portal (http://por-tal.onegeology.org). This portal aims to put geological maps from all countries in the world together on a scale of 1:1 mil-lion or better, by using distributed web map services (WMS). The geological map of Greenland on this portal is currently a 1:2 500 000 scale map. The new 1:500 000 map will fulfil GEUS’ participation in this important international initia-tive.

However, the map does not appear to be there either. There is a "GEUS Geology of Greenland" WMS service available from onegeology but it notes:

Available in this OGC WMS service for personal, non-commercial use only

We're non-commercial but not for 'personal' use.

I tried adding the "GEUS Geology of Greenland" WMS layer in QGIS and found that there is a 1:500,000 layer available, but it does not seem to work. I get "map request failed" warning messages in the log. When attempting to hit the request URL directly in the browser, I get a 404 error w/ message "JSP file [/ows/.jsp] not found"

[trey-stafford]

A WMS version of the map is available via GEUS: https://data.geus.dk/arcgis/rest/services/Greenland/Geological_map_500k/MapServer/WMTS/1.0.0/WMTSCapabilities.xml

However, there does not appear to be a legend (the link to the legend is broken, 404).

Moreover, this page notes:

• We have limited hardware ressources. Use our services responsibly. If in doubt, contact us!
• If email is added to the service-URL, we can better monitor usage and give feedback. Anonymous users might be refused during peak hours.

[trey-stafford]

There are some vector data available via https://dataverse.geus.dk/dataset.xhtml?persistentId=doi:10.22008/FK2/FWX5ET. The data are stored in an mpkx format (Esri Map Package)

Downloading the dataset requires that the user acknowledge their "Dataset Terms":

This data set and documentation (hereafter 'Data') is covered by the following terms of use.

Rights

The 'Data' are subject to copyright by the Geological Survey of Denmark and Greenland (hereafter GEUS).
The user has a nonexclusive right to use the 'Data' for internal purposes only, including the right to process the 'Data', the right to data extraction from the 'Data', and the right to make one safety copy of the 'Data' for own use only.
The user is not entitled to produce or transfer to third parties products based on processed 'Data', unless it is for pure non-commercial information purposes within the user's field of business/field of competence. However, the user is not entitled to produce or transfer to third parties new maps based fully or partly on processed 'Data'.
The user's rights according to these terms of use may be utilised by individuals employed by the user or with any fully controlled subsidiaries of the user. Furthermore, the user is entitled to make the 'Data' available to contractors, consultants and the like in connection with work undertaken for the user. In this event, the 'Data' may only be made available to others to the extent dictated by the specific purpose.
If the user is a consultant or the like, who downloads the 'Data' for the purpose of performing work for a customer, the ”for internal purposes” may be extended to cover ”the customer's internal purposes”, which is conditioned upon the term that 'Data' (original as well as processed 'Data') is transferred to one customer only. If the User is a municipality, county or another governmental body within the Kingdom of Denmark, including Greenland and the Faroe Islands, ”for internal purposes” may include the user's right to use the 'Data' for administrative purposes within its jurisdiction, including handling of specific cases, and for planning purposes, including preparation of regional plans etc.
Furthermore, the user is entitled to make the 'Data' (original as well as processed 'Data') available to other governmental bodies or private individuals as part of the handling of specific cases for the user.

Redistribution

The use is not entitled to copy, publish, resell, lend or rent samples of the 'Data' or part hereof or otherwise redistribute the 'Data' (original as well as processed 'Data') in any other way than described in the section above without the prior written consent of one of the copyright holders.
If the 'Data' are redistributed or otherwise made available to third parties in accordance with the section above or in accordance with a specific written agreement, the user is obliged to inform the third party of and impose on him the obligation to respect GEUS’ and MMR’s rights according to the present terms of use.

This bit seems to suggest that we would be OK redistributing the data:

The user is not entitled to produce or transfer to third parties products based on processed 'Data', unless it is for pure non-commercial information purposes within the user's field of business/field of competence.

But does providing the data w/ symbology in a QGIS project file consitute a "new map"?

However, the user is not entitled to produce or transfer to third parties new maps based fully or partly on processed 'Data'.

I'm guessing they are trying to prevent someone from taking the data and publishing it as a "new" geological map without proper attribution. Maybe wouldn't hurt to reach out to GEUS and confirm they're ok with us including the data in QGreenland though.

[trey-stafford]

Blocked until we contact GEUS

[MattF-NSIDC]

When we do contact GEUS, we might mention that the license terms are a bit vague.

[trey-stafford]

I was about to send an email to Niels when I realized that we probably had already asked this question of GEUS before. Indeed, we have! Back in Dev. 2020, we asked for permission to include data from the PROMICE portal and also the 1:500 000 geological map!

We never got a definitive answer back, as far as I can tell. I'm going to wait to get in touch with Niels or follow up on the existing email thread ("RE: Permissions follow up to seek GEUS permission for including data within QGreenland") until we chat with Twila since she was the primary point of contact.

Here's the email text I was about to send:

Hey Niels,

I hope you are doing well! The QGreenland team has been hard at work preparing v3 of our core data package, which we expect to release later this summer (probably late August).

One of the datasets we are currently considering for v3 of QGreenland is the "Seamless digital 1:500 000 scale geological map of Greenland, version 2.0" dataset found on the GEUS dataverse (https://dataverse.geus.dk/dataset.xhtml?persistentId=doi:10.22008/FK2/FWX5ET). Our only hesitation about this dataset right now is related to its License/Data Use Agreement. It has "custom dataset terms" which we have some uncertainty over.

It states:

The user is not entitled to produce or transfer to third parties products based on processed 'Data', unless it is for pure non-commercial information purposes within the user's field of business/field of competence.

Since we are a non-commercial group who would be redistributing the data for educational purposes, I think we meet this requirement. However, the license also states:

However, the user is not entitled to produce or transfer to third parties new maps based fully or partly on processed 'Data'.

We do not think that redistributing the data via QGreenland would constitute a 'new map' necessarily, but given this language we wanted to double-check with you and/or others at GEUS who may have thoughts on this. Do we have permission to include this data in QGreenland? We would, of course, include full attribution/citation information and provenance for the data.

Please let us know what you think! We appreciate your time and assistance!

Trey Stafford, on behalf of the QGreenland team

[trey-stafford]

Lets continue the thread in the QGreenland email. I'll follow up.

[trey-stafford]

I followed-up with GEUS about this today via QGreenland's email. Hopefully we hear back soon that we can use the data in QGreenland v3!

[twilamoon-science]

If we do not receive permission for direct distribution, this should be a dataset that we highlight as an add-on (like with the ArcticDEM custom Greenland dataset and the ADC QGreenland Data Portal). Want to make sure @alysethurber has this in mind for website update - we do want these few specific add-on/data sources top and center on the "Data" page.

[MattF-NSIDC]

This won't make it in v3, but we may want to link to the data on our website.

[MattF-NSIDC]

Actually, we got an email from GEUS ~6 hours ago that clears us to use this dataset!

[trey-stafford]

The dataset provided via the GEUS dataverse is a zip package containing an Esri Map package (.mpkx).

This map package is a 7zip compressed directory that includes esri-specific metadata and datasets.

After unzipping the data, I see two directories (p20 and p30), each of which have a g500_2023_pro.gdb directory. These gdb directories are geodatabases.

The geodatabases contain vector layers that appear to be the same for both the p20 and p30 directories. They are given by gdalinfo:

$ ogrinfo p20/g500_2023_pro.gdb
INFO: Open of `p20/g500_2023_pro.gdb'
      using driver `OpenFileGDB' successful.
Layer: g500_geopoint_line_anno (Multi Polygon)
Layer: g500_rivers_arc (3D Multi Line String)
Layer: g500_geopoint_line (3D Multi Line String)
Layer: g500_geology_trends (3D Multi Line String)
Layer: g500_contours_arc (3D Multi Line String)
Layer: g500_geology_arcs (3D Multi Line String)
Layer: g500_geology_polygon (3D Multi Polygon)

These layers are easy enough to extract and can be opened in QGIS. The main problem is related to symbology.

The symbology for these layers is important, and the configuration for it appears to be specific to Esri products. I am not sure yet if we can convert these styles into something with QGIS.

Each layer on its own has limited value, and it seems like having them all visualized together is important to interpreting the map For example, features in the g500_geopoint_line layer appears as various straight lines when added to QGIS, but the map as it's supposed to be symbolized represents these lines as arrows of different types depending on the attributes of the feature.

The image below shows one such feature highlighted. Other features from this layer can be seen as red lines.

I plan to do a little more digging into if we can convert the styles from Esri to something compatible with QGIS, but my instinct right now is that this might be more effort than it's worth. Instead, it may be more reasonable to just add this layer as an online-only resource (https://data.geus.dk/arcgis/rest/services/Greenland/Geological_map_500k/MapServer/).

[MattF-NSIDC]

Nice sleuthing. I think adding it as an online layer is a good temporary compromise. Maybe in the meantime, this dataset becomes more FAIR and easier to add.

[twilamoon-science]

It's kind of blowing my mind that they build the archived dataset this way. I agree that it's reasonable to do a bit more digging (also just interesting learning) but that online-only may be the final conclusion. When I linked to the site I didn't see an option for viewing with QGIS. Is that something we do via one of the other viewing methods they list prominently (all proprietary :( )?

[trey-stafford]

I think adding it as an online layer is a good temporary compromise.

Agreed. If we go this route I would probably follow up with GEUS again and explain our plan/reasoning. They request a whoami parameter be set with an email address. I think using the qgreenland info email would probably work.

It's kind of blowing my mind that they build the archived dataset this way.

My guess is that they developed the map & all of it's symbology in Esri products (ArcGIS), which they host a view of via the web map service I mentioned above. When it came to publishing the data, saving it as a map package was probably just a convenient choice. Unfortunately that means hose of us without an ArcGIS license have to do some extra legwork.

I wonder if they could easily publish a raster (e.g., GeoTiff)...maybe it's something we could suggest.

When I linked to the site I didn't see an option for viewing with QGIS. Is that something we do via one of the other viewing methods they list prominently (all proprietary :( )?

I'm not sure QGIS can read the .mpkx file directly. I manually unzipped it and then accessed the gpd databases from QGIS (add as vector).

To access the map service, use the "WMS/WMTS" option in the Data Source Manager and create a new connection using this URL: https://data.geus.dk/arcgis/rest/services/Greenland/Geological_map_500k/MapServer/WMTS/1.0.0/WMTSCapabilities.xml . More detailed instructions on adding a WMS layer from our documentation here.

[trey-stafford]

At least one tool exists for convering ArcGIS .mapx packages into something compatible with QGIS. It's a tool called slyr, produced by a company called North Road. It has a pricey license at 1200 euro. I strongly prefer to look elsewhere before giving this any serious consideration, but I thought I would take note of it.

Funnily, one of the examples they have on their website is a collection of geology related layers! We're not the first ones with this problem.

[trey-stafford]

I took another look at the data this morning, thinking that we could perhaps just add the polygon layer and symbolize by label. This sort of works, but some polygons do not have all of their attributes set, so the legend is still necessary! Moreover, some polygons have a URL associated with them, but all of the ones I tried do not work (I get a message indicating that the geological unit isn't defined in the GEUS data base).

[trey-stafford]

PR for adding the layer as online-only: https://github.com/nsidc/qgreenland/pull/725/files

I plan to follow-up with GEUS to let them know that we plan to utilize their WMS service.

[twilamoon-science]

Thanks @trey-stafford. In your contact with them, a brief note on what was attempted and the problem we encountered would be good to include. And an inquiry on whether they would expect to update/add other formats at some point.

[trey-stafford]

After unzipping the data, I see two directories (p20 and p30), each of which have a g500_2023_pro.gdb directory

This might explain why there are two versions of the geodatabase in the .mpkx map package: https://community.esri.com/t5/arcgis-pro-questions/why-is-there-a-p20-and-p30-folder-when-i-open/td-p/1271259

[trey-stafford]

Should we keep the existing geological map from the Canadian geological survey?

[twilamoon-science]

I think since the GEUS map will be internet-only, it makes sense to keep the current download geological map layers too.

[trey-stafford]

One of our contacts at GEUS provided us with .lyrx style files which should be usable in QGIS with a plugin called qlyrx. Re-opening while I investigate.

[trey-stafford]

I tried using the qlyrx plugin in QGIS 3.28 (our v3 supported LTS) but encountered errors.

An error has occurred while executing Python code: 

TypeError: QgsRendererCategory(): arguments did not match any overloaded call:   overload 1: too many arguments   overload 2: argument 2 has unexpected type 'QgsFillSymbol'   overload 3: argument 1 has unexpected type 'str' 
Traceback (most recent call last):
  File "/home/trst2284/.local/share/QGIS/QGIS3/profiles/default/python/plugins/qlyrx/qlyrx.py", line 1613, in run
    self.apply_lyrx_symbols(layer, self.j_data, geometry_general_type_str)
  File "/home/trst2284/.local/share/QGIS/QGIS3/profiles/default/python/plugins/qlyrx/qlyrx.py", line 1523, in apply_lyrx_symbols
    category = QgsRendererCategory(symbol_val_prep, new_symbol, symbols_labels[idx])
TypeError: QgsRendererCategory(): arguments did not match any overloaded call:
  overload 1: too many arguments
  overload 2: argument 2 has unexpected type 'QgsFillSymbol'
  overload 3: argument 1 has unexpected type 'str'


Python version: 3.10.11 | packaged by conda-forge | (main, May 10 2023, 19:14:33) [GCC 11.3.0] 
QGIS version: 3.28.7-Firenze Firenze, exported 

Python Path:
/home/trst2284/miniconda3/envs/qgreenland/share/qgis/python
/home/trst2284/.local/share/QGIS/QGIS3/profiles/default/python
/home/trst2284/.local/share/QGIS/QGIS3/profiles/default/python/plugins
/home/trst2284/miniconda3/envs/qgreenland/share/qgis/python/plugins
/home/trst2284/miniconda3/envs/qgreenland/share/qgis/python/plugins
/home/trst2284/miniconda3/envs/qgreenland/share/qgis/python
/tmp/geo/lyrx_styles
/home/trst2284/miniconda3/envs/qgreenland/lib/python310.zip
/home/trst2284/miniconda3/envs/qgreenland/lib/python3.10
/home/trst2284/miniconda3/envs/qgreenland/lib/python3.10/lib-dynload
/home/trst2284/miniconda3/envs/qgreenland/lib/python3.10/site-packages
/home/trst2284/.local/share/QGIS/QGIS3/profiles/default/python

I upgraded to QGIS 3.32 and the plugin didn't work at all (got a "core dumped" message).

The plugin says it works with QGIS >=3.4, so I downgraded to 3.4. The result is mixed.

I can apply the style of the g500_geology_polygon and g500_geopoint_line (I think the style isn't right, but no error encountered).

The g500_contours_arc, g500_geology_arc, g500_geology_trends, g500_rivers_arc, gets this error:

An error has occurred while executing Python code: 

AttributeError: 'QgsSimpleLineSymbolLayer' object has no attribute 'setStrokeWidth' 
Traceback (most recent call last):
  File "/home/trst2284/.local/share/QGIS/QGIS3/profiles/default/python/plugins/qlyrx/qlyrx.py", line 1613, in run
    self.apply_lyrx_symbols(layer, self.j_data, geometry_general_type_str)
  File "/home/trst2284/.local/share/QGIS/QGIS3/profiles/default/python/plugins/qlyrx/qlyrx.py", line 1411, in apply_lyrx_symbols
    ret_val = self.parseStroke(symbol_def, ret, layer)
  File "/home/trst2284/.local/share/QGIS/QGIS3/profiles/default/python/plugins/qlyrx/qlyrx.py", line 575, in parseStroke
    symb.symbolLayer(0).setStrokeWidth(stroke_width)
AttributeError: 'QgsSimpleLineSymbolLayer' object has no attribute 'setStrokeWidth'


Python version: 3.7.8 | packaged by conda-forge | (default, Nov 17 2020, 23:49:52) [GCC 7.5.0] 
QGIS version: 3.4.14-Madeira Madeira, exported 

Python Path:
/home/trst2284/miniconda3/envs/qgis-3_4/share/qgis/python
/home/trst2284/.local/share/QGIS/QGIS3/profiles/default/python
/home/trst2284/.local/share/QGIS/QGIS3/profiles/default/python/plugins
/home/trst2284/miniconda3/envs/qgis-3_4/share/qgis/python/plugins
/home/trst2284/miniconda3/envs/qgis-3_4/lib/python37.zip
/home/trst2284/miniconda3/envs/qgis-3_4/lib/python3.7
/home/trst2284/miniconda3/envs/qgis-3_4/lib/python3.7/lib-dynload
/home/trst2284/miniconda3/envs/qgis-3_4/lib/python3.7/site-packages
/home/trst2284/.local/share/QGIS/QGIS3/profiles/default/python

[trey-stafford]

Here's a screenshot of the styles applied to the polygon and geopoint line layers:

Close-up of the geopoint lines:

[trey-stafford]

Given these issues, sticking with the WMST online-only layer still seems like the right choice. We can re-open if we learn more or come up with another solution.

[twilamoon-science]

Yes, good call. In responding to GEUS, please let them know of this decision and that we would like notification if/when they make further updates. Of course, do tell them you tried this and relate the issues.

[trey-stafford]

GEUS provided us with .sld style files for some of the layers, which QGIS can natively read:

• g500_rivers_arc
• g500_geology_trends
• g500_contours_arc
• g500_geology_arc
• g500_geology_polygon

Note that SLD styles were not provided for the geopoint or annotation layers. Our contact at GEUS noted that the geopoint style in particular was giving them trouble, but that they will let us know once it is resolved.

Two of the provided .sld styles did not work at first - I had to manually edit the rules for applying different styles based on an attribute, but it wasn't too difficult.

Screenshot of the result below:

Compare that to the WMS layer:

Assuming we want to add these layers, I'll need to do some more research on what each of these actually represent so that I can put something useful in the 'description'. Maybe GEUS can help us with that!

[MattF-NSIDC]

Sounds like we're in closer contact now, so that's good!

[trey-stafford]

Draft PR: #739

[trey-stafford]

We plan to stick with the WMS version.

The styles as defined work, but causes visualization to be extremely slow. Moreover, we agreed upon review that the layer order and symbolization provided by GEUS are still key to interpretation (e.g., many polygons have NULL descriptions but their symbology still indicates the geological unit type), so a rendered view (e.g., from WMS) makes the most sense. The main drawback is the need for a separate legend that has to be downloaded (pdf) and used alongside the layer.

Note that I did just try re-configuring the symbology for the polygon layer to use the "categorization" type using the gm_label as the value and the layer now performs just fine. Maybe there's a way to convert a rule-based style into a categorization style (we could likely write some code to do this too), but given our limited time the best solution right now is to communicate our findings to GEUS.

[twilamoon-science]

Plus, mentioning any elements that we were successful with (e.g., your experiment with "categorization" type) can help GEUS to provide guidance to other users and continue to improve the data on their end.

[MattF-NSIDC]

Note that I did just try re-configuring the symbology for the polygon layer to use the "categorization" type using the gm_label as the value and the layer now performs just fine

Great find!

[trey-stafford]

Just sent a reply to our email thread with GEUS. Closing this issue for now!