Add the ability to import / export sub sections of layer definition properties (AKA QML sub-part export)

[haubourg]

QGIS Enhancement: Add the ability to import / export sub sections of layer definition properties (AKA QML sub-part export)

Date 2018/05/24

Author Régis Haubourg (@haubourg)

Contact regis dot haubourg at oslandia dot com

maintainer TBD

Version QGIS 3.2

Summary

This proposal intends to improve current situation by clarifying namings and adding the ability to export / import / copy-paste sub-parts of layer definition sections.
Currently, layer definition is composed of sections (symbology, forms, metadata, etc..). The UI allows to save several "styles" for a layer, which is used in map theme presets.

The "style" naming is a source of confusion because it is not clear which parts of a layer definition belong to the "style". It is hard to know which settings can be variants when switching from a style to another one. Moreover, most users get puzzled when they lose their forms settings just when changing a map theme preset.

It reveals also hard to transfer a layer property section to another style or another layer definition, like duplicating the form or the labels to another style. Currently, the only way is to play with the python API.

This QEP proposes to improve style definition, rework the UI so that users understand these concepts more naturally, and then add ability to import / export sub sections.

Proposed Solution

Clarify "Style" and "Layer definition" concepts

• Documentation : Add a section in https://docs.qgis.org/testing/en/docs/user_manual/working_with_vector/vector_properties.html to clarify QGIS concepts.

  Layer definition as in QLR is a full layer definition, including datasource link. A layer definition contains one or more layer styles.
  Layer style A layer style file saves only one combination of properties, and it does not contains the datasource reference.

• reorder layer properties UI :

We propose to visually materialize which part of the layer settings are unique or potentially multiple in layer properties dialog.

Add import/export and copy/paste ability for layer definition sub sections

The requirements are:

• Copy a subset of layer properties sections

• Paste a subset of layer properties sections

• Import a subset of layer properties sections from a saved file or database (should be SLD / QML / DB)

• Export a subset of layer properties sections to a saved file or database

• Quick access to those shortcuts from layer properties List Widget (right click on the section header)

• Quick access to copy to clipboard of one current section from the layer contextual menu. This is limited to only copy one section quickly.

• Paste of a layer definition from clipboard will replace only the target property section. Ie Copying only the label section to the clipboard and paste it to a layer will replace only the label section, and not reset the others

• Layer contextual menu (right click)

The idea is to have here a limited feature limited to copying the one sub section only. Pasting will replace the current subsection. Nota: pasting a xml definition with several sub section should replace all target sections.

• Dynamic styling dock

Current dynamic styling dock shows only a subset of the layer definition (labels, symoblogy, and styles). The proposal is to only rename the "Style manager toolip" to "Layer style manager"

• Classic Layer properties dialog

This is where we need the power features and cleanups.

Side note : The situation of the current "Style" Button is a bit weird. It is accessible from every section, except from the new metadata one, where it is replaced by a "metada" button.

Any other proposal is welcome here to improve the situation.

Style widget current situation

The main idea is to wire two dedicated dialogs for saving and loading layer style, unioning the current features and the section chooser.
Those dialogs will allow to manage properties sections and allow to select the sub sections for import and export. This will allow to finely control the section sets to import or export.

• API

The API and some UI files uses the "Style" naming convention, and we have a QgsMapLayerStyleManager to handle all that. Renaming it will be considered as an API break.
If we choose to rename it to something else, we should add the new classes, decorate the old one and declare them obsolete. This part is to be discussed, I have no strong opinion here.

Affected Files

TBD

Performance Implications

None but it should improve a lot the GIS project administrators productivity !

Further Considerations/Improvements

That work raises unaddressed issues:

** There are dependencies between sections ** currently. For instance, field aliases can now be used by expressions anywhere, so pasting a symbology section using those aliases would require to import those aliases too.
The same stands for joins and relations. They are stored at a project file level, but form definition and join definition will require them. It is currently not handled specifically. We have users requesting that feature https://issues.qgis.org/issues/9968 Layer relations saved in .qml file

The form definition section is currently stored in the style, so each time one changes a style, the form also changes
In some use cases, this can be useful, but most of the time this is a caveat for most users. Allowing an easy copy - paste will help to duplicate a form in all possible styles, but this is a poor workaround.
The ideal fix would be to have a set of possible forms saved in layer definition, and each style could be associated with one explicit default form ID.

This is not planned in the current QEP but deserves discussion as this is far from being optimal. Any proposal is welcome to improve the situation.

Backwards Compatibility

To be determined. If we don't change the naming and style concepts, there should be no change.

Issue Tracking ID(s)

https://issues.qgis.org/issues/15733 "improve qml API to allow filtering different layer definition sections"

https://issues.qgis.org/issues/5115 "Save more than one style in a .qml file and give user a choice when loading the layer"

Votes

(required)

[haubourg]

@3nids @m-kuhn @pblottiere @mhugo any opinion here, do you think it's strong enough to start implementation process?

[m-kuhn]

Thanks for writing this up, @haubourg

Looks like a good approach 👍

Some comments and questions follow

Copy sections

Is the requirement to be able to copy/paste multiple sections at once (I think so reading the first two requirements with the checkboxes). If I understand it correctly, on the layer properties dialog, only single sections can be copied with a right click. On the layer tree context menu, you propose to add a submenu, will that allow to export multiple sections at once? Will this not require an extra dialog like the ones mocked up in the export/import section?

Import/Export dialog

Layer style vs. definition

In your mockups for the export, I see qml / sld (classic layer style files) whereas in the same dialog layer definition is written several times. Is this going to be a single dialog to also allow exporting qlr files? As far as I can tell, qlr files are currently exported via a dedicated dialog.

Visualize available sections on import

It would be nice to already visualize in the import dialog what parts are present in a .qml file. I'd propose when selecting a qml file to gray out those entries which are not present in the qml file and pre-select all others, so a user can then unselect those he is not interested in. What do you think about that?

Map themes

Are Map Themes also a focus here? I see them mentioned on the top but I cannot find them anywhere in the requirements themselves.

Terminology

• Copy a subset of layer properties sections

  Should be layer style sections I guess

• Paste of a layer definition from clipboard

  layer style?

• Is it sub sections or sections?

API

The ui files and classes in app are not affected by the API freeze. QgsMapLayerStyleManager is, but we could potentially add an alias if there's a with to rename this.

File extension

We currently use the file extension qml, which is very unfortunate because Qt QML is becoming more and more important. We had a proposal in the past to rename this to rename qml to qlp. It would possibly be a good point in time to revive this idea, allow importing qml but new exports will be done to qlp (with added support for subsections).

Further considerations

Dependencies between sections, I'd also ignore this in a first step, this can be added later on easily if we see that there are too many mistakes.

Joins and relations also raise a couple of questions, comments on the issue on redmine. I agree to ignore this for now.

The form definition section is currently stored in the style, so each time one changes a style, the form also changes

Not sure I can follow here, I think this is exactly what is being fixed with this proposal by allowing to export/import styles without forms. Is this addressing map themes rather than styles?

Documentation update "Style" and "Layer definition" concepts

I think one of the hardest parts is to realize that a style is not the symbology, but the symbology plus extra parts. I'd put a focus on this distinction in the doc update.

[haubourg]

Hi Matthias, thanks for your feeback.

If I understand it correctly, on the layer properties dialog, only single sections can be copied with a right click. On the layer tree context menu, you propose to add a submenu, will that allow to export multiple sections at once? Will this not require an extra dialog like the ones mocked up in the export/import section?

I meant right the opposite, ie have the full features in the layer properties sub dialog, and allow to copy sub sections one by one only in the layer true contextual menu. Sorry if this was unclear

In your mockups for the export, I see qml / sld (classic layer style files) whereas in the same dialog layer definition is written several times. Is this going to be a single dialog to also allow exporting qlr files? As far as I can tell, qlr files are currently exported via a dedicated dialog.

I made the mockups when I was unsure about the right naming of the concepts when handle. "style/definition" in the labels should have been written [% theNewNameOfStyle %] . Reusing "layer definition" is confusing with QLR indeed, I'll remove it.

Are Map Themes also a focus here? I see them mentioned on the top but I cannot find them anywhere in the requirements themselves.

No, it was a context reminder, I just mentioned that map themes is one of the feature taking benefit of the style switcher.

Should be layer style sections I guess

Sold! I like it !

Is it sub sections or sections?

Right again, section is already a sub style :)

It would possibly be a good point in time to revive this idea, allow importing qml but new exports will be done to qlp (with added support for subsections).

Yes again, that will be a lot easier to get.

The form definition section is currently stored in the style, so each time one changes a style, the form also changes
Not sure I can follow here, I think this is exactly what is being fixed with this proposal by allowing to export/import styles without forms. Is this addressing map themes rather than styles?

Well, the export of section will ease the work for keeping all styles of a single layer with the same form template. However, each time a form is changed, users will have to replicate again the form to all the styles. This is an open question here, we can ignore this, but I think most use cases I know of, users suppose there is only one form for a layer. I never saw anyone taking benefit of having different forms using styles, so I'm tending to think we could have a "main" form by default, whatever the style displayed.
I used to duplicate layers and name them differently if I needed different forms for the same datasource.
Again, I wanted to raise the discussion but I don't have a strong opinion here.

I think one of the hardest parts is to realize that a style is not the symbology, but the symbology plus extra parts. I'd put a focus on this distinction in the doc update.

Good, agreed. If a more natural naming existed I think we probably already had found it.

[m-kuhn]

I meant right the opposite, ie have the full features in the layer properties sub dialog, and allow to copy sub sections one by one only in the layer true contextual menu. Sorry if this was unclear

I have only found a dialog to select which sections to export. I couldn't find a GUI to select which sections to copy (and paste). Is this planned, if yes how?

Well, the export of section will ease the work for keeping all styles of a single layer with the same form template. However, each time a form is changed, users will have to replicate again the form to all the styles.

A user can export one style with the form to one file and several styles with different symbologies to other files, no?

[haubourg]

I have only found a dialog to select which sections to export. I couldn't find a GUI to select which sections to copy (and paste). Is this planned, if yes how?

There was this unprecise mockup here :
https://user-images.githubusercontent.com/1629853/40008651-1ea9176c-57a0-11e8-9e82-cff3ee0bb094.png

Would this one be better ?

A user can export one style with the form to one file and several styles with different symbologies to other files, no?

Mmm yes, not sure I get your point. My main concern is for GIS administrator, that keep maintaining forms, and have additional overhead to replicate this form definition in each style of the same layer again and again. A simple python routine can adress that for sure. But the real question is Do we really need many different forms for one layer?. Any opinion on that?

[haubourg]

Ouch, wrong screenshot above, I fixed it just now.

[m-kuhn]

Screenshot: sorry, I was a bit unclear with my question. Is it possible to copy multiple sections at once?

Mmm yes, not sure I get your point. My main concern is for GIS administrator, that keep maintaining forms, and have additional overhead to replicate this form definition in each style of the same layer again and again.

I assume "each style" refers to "each style file" here. If "each style file" only contains symbology and no form definitions, he doesn't have to do anything, no?

I would not disable the possibility to share forms via style files, but turning it off by default could be an option.

[haubourg]

Screenshot: sorry, I was a bit unclear with my question. Is it possible to copy multiple sections at once?

Nope, Let's keep it simple in the context menu. I can't find an UI authorizing this without creating a new dialog, unless you have a better idea.

I assume "each style" refers to "each style file" here. If "each style file" only contains symbology and no form definitions, he doesn't have to do anything, no?

My use case is the following:

1. add a layer and customize a form
2. Save a style name for this (no file export, only in QGIS project)
3. Save a new style and change anything
4. Switch back to the first style, amend some form settings
5. This is the step where users need to do a manual copy of the form section, then switch again to style number 2, and paste the form definition so that style 1 and 2 forms are inline again.

Maybe I miss something in that workflow, show me the light :)

I would not disable the possibility to share forms via style files, but turning it off by default could be an option.

Me neither, we need to export forms for sure in style files

[m-kuhn]

3. Save a new style and change anything

Save a new style file without forms here

[haubourg]

I think we still don't speak of the same scenario. Here is a quick footage where I never use any file saving in that workflow:
https://youtu.be/nhzMj_3SEXA

What would be your advice?

[m-kuhn]

I would say at some point a dialog will need to be integrated that also allows defining what is saved into these styles, just like for export to file.
Is there a reason to treat this differently? (and disabling forms by default here - and potentially even making the defaults configurable on application level, given that actions are a similar topic than forms)

[haubourg]

OK I get it, you suggest that the user chooses what sections are relevant to keep from current style when creating a new one right ?
I'm 50/50 on that. I like the current way of creating a new style from the current state. It's simple and intuitive. I think that adding dialog there would clutter the UX. But this is a personal opinion and not a very strong one. I'm totally opened to suggestions.

[m-kuhn]

Yup, always being forced to use a dialog would be horrible. But a dialog via keyboard modifier (Alt + click) or a separate menu entry for project managers that prepare a project and/or a global application setting to define which sections are saved within styles should be ok also from a UX perspective?

[haubourg]

ok, let's keep that in the implementation / PR freechoices then.

[dmarteau]

Just to mention the layer 'Variables' section should be an option for saving them in the qml or not:
Actually these variables are used in expressions and are overwritten/removed each time the qml is loaded.
Choosing to store them or not in the qml could be interesting to define them explicitely in layers and use them for some style parametrization with global layer properties (I have processing outputs in mind and their associated qml files).

[3nids]

After digging a bit into the code, I would have a few questions/remarks:

A. General

1. We here speak only about vector layers. Anything implemented will be easily ported for raster layers, but this is not in the scope here.

B. Code / API

Writing XML is called via QgsMapLayer::writeLayerXml which does the common writing and then calls the virtual writeXml, being subclassed in vector layer and friends.

It goes like this:
QgsVecvotLayer::writeXml first writes vector joins, relations, metadata, auxiliary storage
then calls writeSymbology:

1. calls writeStyle:
   a. renderer
   b. labeling
   c. simplify method (actually written in the node layer, how come???)
   d. custom properties (which is already done in QgsMapLayer, but apparently info is not duplicated, still have to investigate)
   e. blend mode field, feature blend mode field
   f. layer opacity
   g. diagram rendere
2. geometry options
3. fields (incl. widgets)
4. attribute aliases
5. excluded fields for WMS/WFS
6. default expressions, constraints, constraints expressions
7. actions
8. attribute table config
9. edit form config
10. conditional styles
11. expression field buffer
12. layer read only
13. layer searchable
14. display expression
15. map tip

This is actually the result of writeSymbology which is saved in the layer styles (as XML) when calling QgsMapLayer::exportNamedStyle.

So here come questions and ideas:

C. Categories for export (for vector layers)

A list of categories could be:

1. Symbology
2. Labels
3. Fields (Aliases, WMS/WFS, expressions, constraints)
4. Forms
5. Actions
6. Tooltips
7. Diagrams
8. AttributeTable
9. Rendering (scale visibility, simplify method)
10. Display Expression (worth a dedicated category? merge with attribute table? fields?)

Now the question is, should everything lie in a category?
If yes, I don't know what to do with:

• custom properties
• expression field buffer (although I don't understand yet what it is)
• layer read-only, searchable
  ???

If not, we would have to choose in the export dialog between the complete export or the partial export (selection of categories). But that means that complete export != sum of all categories.

D. Naming mismatches

Here are some naming mismatches between the code and the GUI:

• writeSymbology in the code corresponds to the style in the GUI (the style button to import/export)
• the symbology in the GUI (symbology tab in layer properties) actually corresponds to the renderer in the code/XML
  I think we would gain to harmonize a bit the vocabulary.

I think that the methods in the API should be inverted: writeSymbology becomes writeStyle and writeStyle becomes writeSymbology. Technically, they have the same signature so it's not an API break.... But I guess this has to wait for QGIS 4.
Anyone sees another approach?

E. There are some settings saved in the main node layer:

<maplayer simplifyDrawingTol="1" geometry="Polygon" hasScaleBasedVisibilityFlag="0" simplifyMaxScale="1" maxScale="0" refreshOnNotifyMessage="" minScale="1e+8" labelsEnabled="0" refreshOnNotifyEnabled="0" simplifyAlgorithm="0" autoRefreshEnabled="0" simplifyDrawingHints="0" readOnly="0" autoRefreshTime="0" type="vector" simplifyLocal="1">

similarly, in the style:

<qgis simplifyDrawingTol="1" version="3.3.0-Master" hasScaleBasedVisibilityFlag="0" simplifyMaxScale="1" maxScale="0" minScale="1e+8" labelsEnabled="0" simplifyAlgorithm="0" simplifyDrawingHints="0" readOnly="0" simplifyLocal="1">

I think these should be moved away from the top level node.
One issue for instance: exportNamedStyle integrates the scale visibility (usually done in QgsMapLayer::writeLayerXml) by duplicating the code while the simplify is performed in writeStyle(hence no need to duplicate)

F. Layers capabilities

In project properties, we can define for each layer:

• identifiable (saved in project as non-identifiable list)
• read-only (saved in layer)
• searchable (saved in layer)
• non-removable (saved in project)

Being saved in the project means it is not written in the style definition (XML).
While it makes sense for non-removable tag, I am not really sure about the 3 others.
We should have coherent way of saving these 3 values.
Shall they be seen as layer properties (as style) or as project properties?

[3nids]

Feature implemented in qgis/QGIS#7863
Documentation PR qgis/QGIS-Documentation#3041
The tabs of the vector layer properties have not been moved around as I am not really convinced by the proposition (joins far away from the fields, sources and forms not close to each other, symbology further down, etc.).

[lochkalyan]

Can we save sld included style of diagram?

[m-kuhn]

Can we save sld included style of diagram?

Hi, it's not exactly visible for me, what you want/need, I am sorry.

If you are unsure how to do things, please use one of the support channels https://www.qgis.org/en/site/forusers/support.html

If you think a functionality is missing, please open an issue, explaining in detail, what you propose https://qgis.org/en/site/getinvolved/development/bugreporting.html

[lochkalyan]

When I save sld style i get only the style of symbology.

[haubourg]

Let's close this PR, implemented since long