Meeting minutes on interfacing GRASS from QGIS

[petrasovaa]

Meeting minutes on interfacing GRASS from QGIS

2023-06-05 10:00 CEST
Conversation as part of the GRASS Community Meeting in Prague 2023
Participated: Nyall Dawson (online), Markus Neteler, Vaclav Petras, Anna Petrasova, Micha Silver, Loïc Bartoletti (online), Floris Vanderhaeghe

Overview

Nyall presented an overview of GRASS-QGIS integration components and their strengths and weaknesses. (Nyall Dawson has emailed us his slides.)

• QGIS Processing plugin
  • https://github.com/qgis/QGIS/tree/master/python/plugins/grassprovider
  • nice that users can mix & match tools (QGIS, GDAL, PDAL, ...), also in automated workflows
  • GRASS tools accessible to 'less expert' audience
  • maintained, up-to-date set of GRASS algorithms
  • slow, because data is always imported and exported: does not run on GRASS database directly
• GRASS Data provider
  • https://github.com/qgis/QGIS/tree/master/src/providers/grass
  • natively render GRASS maps, editing works too
  • i.e. the QGIS Layers panel can load the data from a GRASS database directly as layers
• GRASS C++ plugin
  • https://github.com/qgis/QGIS/tree/master/src/plugins/grass
  • runs directly on GRASS database, hence fast
  • understands GRASS concepts
  • BUT not maintained at all (algorithms outdated, its terminal has no Qt6 support, outdated UI); has no real QGIS integration (has its own menu entries & window)

To reduce maintenance burden and make it less confusing for users, the C++ plugin needs to be removed.

Removing GRASS C++ plugin

The goal is to remove GRASS C++ plugin. To do that, some functionality should be moved to Processing or QGIS Settings. Mapset handling unclear where to put it.

Nyall's goals

Short-term goals

• Processing should be able to work on GRASS data directly. User could specify that input or output layer is a GRASS map.
  • This will allow to efficiently chain multiple GRASS tools. The default would be to still export to tiff and gpkg, so we need to think how to inform users. There is already some mechanism for chaining.
  • Also, this would allow non-GRASS algorithms to directly read or write from/to the GRASS database.
• Remove terminal widget from the C++ plugin (due to Qt); remove its tools support; move remaining bits elsewhere.

Long-term goals

• QGIS native understanding of topological data sources (topoJSON, GRASS, pgtopo, ESRI)
• Start a GDAL for topological data in the future? TDAL?

GDAL-GRASS driver

Markus points out the existence of GDAL-GRASS driver, which is maintained, used by GeoServer GRASS raster datastore and planned to use in rgrass (again). Packaging could be done, but perhaps it is not worth adding another dependency when QGIS GRASS provider works fine. Moreover, the latter operates directly, doesn't depend on GDAL, which Nyall regards as an advantage.

GRASS Tools in Processing

GRASS C++ plugin often split the GRASS tools into multiple (e.g., r.slope), which in the past caused issues. That is not the case with GRASS tools available from the Processing plugin.

Micha sees many users complaining, when the tool fails, but the error message is confusing for both users and GRASS developers.

In general, the interface is being kept up-to-date, unlike e.g. SAGA, which had to become a 3rd party provider.

GRASS GIS Addons

GRASS GIS Addons are often requested, but are not available in Processing, see recent related QGIS issues/PRs:

• Add support for GRASS addons qgis/QGIS#53048
• Add algorithm to manage GRASS addons qgis/QGIS#53049
• support grass algorithms in profile folder qgis/QGIS#53246

There needs to be more discussion how to support addons, some ideas/questions/concerns:

• having GRASS Addons as a separate provider
• the current approach (no separate provider), would just need to add all addon algorithms in QGIS' native GRASS processing provider even though the GRASS Addons are not necessarily installed. In such case QGIS would provide an error message when an addon is missing. g.extension can be used to install addons.

Algorithm definition and description files

This applies to all GRASS algorithms (as for other providers). Each GRASS tool in the Processing plugin has a .txt file for converting options to whatever QGIS needs. Significant number of tools have also an associated Python file for some special cases. This complicates updating tools when they change in GRASS GIS and adding Addons.

Some ideas/questions/concerns:

• auto-generate description files, there is no tool for that yet
• unclear how much they can be auto-generated
• where to store the files, would there be an advantage to storing them on GRASS side so that they can be updated when the tool changes?
• Nyall wants to change the approach for the definition files. These many files are read at startup, taking long especially on Windows. Another format seems a better solution.
  • this issue will only grow if we add all addons.

Threading

Currently, GRASS calls block the QGIS GUI since grass --exec g.something always locks the mapset. There is no reason for that, since Processing calls GRASS with grass --tmp-location. GRASS command can be safely run in separate mapsets. Calling in the same mapsets is routinely done in GRASS, but users need to know what they are doing. Issues would arise with different computational regions, or when writing to the same raster layer or to the same sqlite database (different vector layers share same sqlite db). This could be solved, if needed. Nyall will remove the restriction so that GRASS tools can be called in the background.

Closing notes

Nyall is eager to continue supporting GRASS natively in QGIS.

A follow-up would be nice in e.g. 2 months. Nyall, please share your slides.

Other developers to contact: Alexander Bruy (alexbruy), Jürgen Fischer (jef-n), Loïc Bartoletti (lbartoletti)

[florisvdh]

Regarding 'GRASS Tools in Processing', IIRC some discussion also took place about the current "grass7:xxxx" ID for each GRASS algorithm in QGIS. "grass7" should be replaced by "grass" since it works with both versions 7 and 8 of GRASS GIS.

[florisvdh]

Adding an update about:

• Nyall wants to change the approach for the definition files. These many files are read at startup, taking long especially on Windows. Another format seems a better solution.
  - this issue will only grow if we add all addons.

=> I saw that @nyalldawson has implemented an integration of these txt files (which remain in place) into 1 JSON file (not in the repo), which is read instead to speed up QGIS loading. See qgis/QGIS#55513.

[florisvdh]

In relation to above comment: starting with the upcoming QGIS release (3.36) the GRASS GIS algorithms will be referred to as grass:xxxxx instead of grass7:xxxxx; see qgis/QGIS#56196.