Expose OGR feature styles via a new "Embedded styling" renderer

[nyalldawson]

QGIS Enhancement: Expose OGR feature styles via a new "Embedded styling" renderer

Date 2021/02/16

Author Nyall Dawson (@nyalldawson)

Contact nyall dot dawson at gmail dot com

maintainer @nyalldawson

Version QGIS 3.20

Sponsor QGIS Denmark User Group

Summary

Some vector data formats have native support for embedding feature styling information, such as MapInfo TAB files. These formats allow embedded symbology to be set on a feature-by-feature basis. This proposal concerns exposing this per-feature styling information via the QGIS API and creation of a new "embedded styling" renderer for compatible vector layers, which allows users to view features using the closest possible match to their original symbology.

Proposed Solution

This proposal relies on the underlying GDAL "feature styling" support. You can read more about this here: https://gdal.org/user/ogr_feature_style.html

Notably: "The following GDAL vector drivers have varying levels of support for feature styles: DWG (libopencad), DWG (Teigha), DXF, KML (libkml), MapInfo, MicroStation DGN v7 and DGN v8, OpenJUMP JML and PDF."

The main driver behind this work is to expose the existing support in GDAL for MapInfo feature styling, and accordingly no work will be done to enhance the feature styling support for GDAL drivers. It is our hope that the increased exposure of this functionality from GDAL will see more attention (and sponsorship) given to extending the GDAL feature styling functionality and adding support for other suitable formats, to the benefit of all GDAL clients.

API Changes

QgsOgrUtils

A new method will be added to QgsOgrUtils to convert OGR style strings to their equivalent QgsSymbol:

/**
 * Creates a new QgsSymbol matching an OGR style \a string.
 *
 * Caller takes ownership of the returned object.
 */
QgsSymbol* symbolFromStyleString( const QString& string ) SIP_FACTORY;

There is 100% overlap between the symbology supported in OGR style strings and QGIS symbology, so the returned symbols should be a very close match to their OGR representation. (Note that the GDAL reading of symbols from the original data source may be lossy in order to represent them as OGR style strings, so there is no guarantee that the QGIS representation will be an exact match for their original appearance, only that the conversion from GDAL -> QGIS will be lossless.)

QgsFeature

The QgsFeature class will gain a new getter/setter for the feature's embedded symbol:

/**
 * Returns the feature's embedded symbology, or NULLPTR if the feature has no embedded symbol.
 */
const QgsSymbol* embeddedSymbol() const;

/**
 * Sets the feature's embedded \a symbol.
 * Ownership of \a symbol is transferred to the feature.
 */
void setEmbeddedSymbol( QgsSymbol* symbol SIP_TRANSFER );

If present, the symbol will be stored as a unique_ptr in QgsFeaturePrivate. Accordingly the impact on the size of storing QgsFeatures will be minimal, as symbol-less features will only grow by the size of the new pointer member. Furthermore, QgsFeature objects are implicitly shared, so the cost of shallow copies of features with attached symbols will not be impacted.

QgsFeatureRequest

A new QgsFeatureRequest::Flag will be added: EmbeddedSymbols. By default feature requests will NOT retrieve any embedded symbols in order to keep the feature requests as fast as possible. Callers must explicitly opt-in to retrieval and conversion of embedded symbols by setting the new flag on their feature requests.

If the flag is NOT present, then QgsFeatures::embeddedSymbol() will always be nullptr, regardless of the presence or absence of feature level symbology in the underlying dataset.

QgsOgrFeatureIterator

The QgsOgrFeatureIterator class will check for the presence of the QgsFeatureRequest::EmbeddedSymbols flag and if present, the underlying OGR feature styling will be read for each feature and converted to a QgsSymbol set for the returned features.

While this proposal only covers support for OGR feature level symbology, it is entirely possible that feature level symbology is supported by other vector data providers and future work could see their feature iterators gain support for this functionality too. All API will be kept generalised in order to support this potential future work.

QgsVectorDataProvider

QgsVectorDataProvider will gain a new Capability flag FeatureSymbology. Providers which support feature level symbology will return this flag. Initially, only QgsOgrProvider objects reading from supported OGR data sources (i.e. MapInfo TAB files) will return this flag.

Additionally, QgsOgrProvider will also have the QgsVectorDataProvider::createRenderer method implemented so that newly added layers with compatible datasources will use the embedded symbology by default.

QgsFeatureRenderer

QgsFeatureRenderer will gain a new virtual method for determining whether the feature renderer requires embedded symbols:

/**
 * Returns TRUE if the renderer uses embedded symbols for features.
 * The default implementation returns FALSE.
 */
virtual bool usesEmbeddedSymbols() const;

The QgsVectorLayerRenderer class will test the usesEmbeddedSymbols return value for the layer's renderer in order to determine whether the feature requests created by QgsVectorLayerRenderer should include the new QgsFeatureRequest::EmbeddedSymbols flag. By default renderers will indicate that they do not use embedded symbols, so there will be no extra cost to the existing use cases and rendering of layers.

New "Embedded Renderer" vector layer renderer

A new "Embedded Symbol" vector renderer will be created. Whenever this renderer is used, the original embedded symbol for each feature will be rendered. The renderer will also support setting a default fallback symbol to use for features which do not have any embedded symbol available.

Accordingly, users will be able to set their layer to use the Embedded Symbols renderer in order to see the original stored symbology for features!

[jgrocha]

Is this just one way? Are you considering writing the embedded symbol back to the provider?

[nyalldawson]

Is this just one way? Are you considering writing the embedded symbol back to the provider?

It's one way. There is already support for saving symbols when writing to files in the vector layer "save as" dialog, but support is dependant on whether gdal supports saving features for the output file type.

[bvthomsen]

How will the new renderer handle missing TrueType fonts on the QGIS computer?

Use Case: A MapInfo user copies a tab - file to a QGIS user. However, the QGIS user doesn't have the standard MapInfo specific fonts installed on her computer.

Ex: A OGR feature style definition:
SYMBOL(c:#00FF00,s:12pt,id:"font-sym-75,ogr-sym-9",f:"MapInfo_Cartographic")

"MapInfo_Cartographic" is a MapInfo specific TrueType font.

[bvthomsen]

Which data format will be supported ? You mention TAB, but what about other OGR vector formats with support for embedded styling.

I'm especially interested in DGN support because this data format is used by a lot of Danish architectural and cadastral service companies

[jgrocha]

Which data format will be supported ? You mention TAB, but what about other OGR vector formats with support for embedded styling.

I'm especially interested in DGN support because this data format is used by a lot of Danish architectural and cadastral service companies

@bvthomsen I did some recent improvements to GDAL DGNv8 driver to read User Data Linkage. I've completely ignored style properties, as I was only focused on importing the classification defined in user data linkage. If this is import to you/Danish community I'm available to further improve DGN support in GDAL. But I think, as @nyalldawson pointed out, this is out of scope of this QEP. This QEQ will use what is provided by GDAL.

[bvthomsen]

Hi Jorge ( @jgrocha ) -

I assume, that the GDAL-OGR driver for DGN 7/8 exposes some or all symbology using the same "OGR style string" mechanism as TAB files.

From the GDAL webpage about DGN ver 7:

Styling Information
Some drawing information about features can be extracted from the ColorIndex, Weight and Style generic attributes; however, for all features an OGR style string has been prepared with the values encoded in ready-to-use form for applications supporting OGR style strings.

So my question is more: Which specific data formats in QGIS - besides MapInfo - will support the new "embedded styling" feature ?
According to the GDAL webpage there is several candidates:

Notably: "The following GDAL vector drivers have varying levels of support for feature styles: DWG (libopencad), DWG (Teigha), DXF, KML (libkml), MapInfo, MicroStation DGN v7 and DGN v8, OpenJUMP JML and PDF."

[Houska1]

+1 from me. Being able to faithfully render -- data and styling -- output from other GIS apps is good. Piggybacking on GDAL/OGR even better!

Your primary motivation seems to be MapInfo, others mention DGN. It would also be useful for importing KMLs from Google Earth with styling.

As a small point, it would be super helpful to (optionally?) expose OGR's OGR_STYLE raw as well, as a feature attribute. This will enable the option that users can start by using the new embedded styling renderer, but if/when desired build their own styles in QGIS' current renderers which selectively parse elements of the embedded style string as data-defined overrides. I've done this in the past, extracting the colors of researchers' marked waypoints from KML. However, at this time you have to jump through hoops to access this (see this issue).

More broadly, while perhaps out of scope of this specific proposal, it would be helpful if key elements of the parsing done by symbolFromStyleString were eventually also accessible via QGIS expressions, for use in data-defined override expressions. Perhaps something like embedded_style(string,tool,[parameter]) so that embedded_style("OGR_STYLE",'BRUSH','fc') would for instance return the embedded polygon fill colour, or embedded_style("OGR_STYLE",'@') would return a predefined style name if specified for that feature in the datasource. Would have to think through what to do about invalid/missing specifications, how to deal with units (px vs m vs pts etc.), and whether to add a shortcut $embedded_style(t,p)=embedded_style("OGR_STYLE",t,p).

Even more fundamentally, I guess this new functionality could open the door to other forms of markup for feature-by-feature embedded styling, for instance based on QGIS' own <renderer-v2> XML schema, being implemented in the future?

[nyalldawson]

@bvthomsen

How will the new renderer handle missing TrueType fonts on the QGIS computer?

Use Case: A MapInfo user copies a tab - file to a QGIS user. However, the QGIS user doesn't have the standard MapInfo specific fonts installed on her computer.

Ex: A OGR feature style definition:
SYMBOL(c:#00FF00,s:12pt,id:"font-sym-75,ogr-sym-9",f:"MapInfo_Cartographic")

"MapInfo_Cartographic" is a MapInfo specific TrueType font.

Great question! I'd suggest we could handle this in a two-part way:

1. For simple font shapes, such as the boxes, circles and stars from MapInfo Cartographic we could have logic in place to auto-convert these to their equivalent QGIS Simple Marker shape. I'd apply this to the red shaded symbols below:
   
   SLYR uses this approach and it works well!

2. For all others, add a mechanism to warn users when a referenced font from a layer's symbology is not available at render time. We would do this once per font name per session only. (This would also help with purely QGIS styles/projects shared between users with potentially different installed fonts!)

[nyalldawson]

@Houska1

Even more fundamentally, I guess this new functionality could open the door to other forms of markup for feature-by-feature
embedded styling, for instance based on QGIS' own XML schema, being implemented in the future?

I'd also love to see this at some stage -- but there's actually little overlap in the development required between that in this proposal, so it would need to be done as a separate piece of work.

[elpaso]

@nyalldawson I've gone through your QEP and the API looks good to me. Nice to see some progress in this area.

[KVO-cartographer]

@nyalldawson @elpaso It will be nice and very useful to add ability to read text objects from MapInfo .MIF and .TAB formats. I can't say it about .TAB format, but the .MIF format is human understandable and intuitive. I have already created a feature request about it Ability to import text objects from .TAB and .MIF formats