Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Overhaul Metadata Management In QGIS #91

timlinux opened this issue Feb 25, 2017 · 18 comments

Overhaul Metadata Management In QGIS #91

timlinux opened this issue Feb 25, 2017 · 18 comments


Copy link

@timlinux timlinux commented Feb 25, 2017

QGIS Enhancement: Overhaul Metadata Management In QGIS

Date 2017/02/26

Author Tim Sutton (@timlinux)


maintainer @timlinux

Version QGIS 3.0


  • Provide the Infrastructure in QGIS to author, consume and share Dublin Core, ISO 19115 etc. GIS Metadata
  • Support the mandatory fields and those fields that are needed to support service descriptions for QGIS Server
  • Make the creation of metadata seamless where it does not exist
  • Use templates and heuristics so that we need the minimum number of clicks in order to define basic functionally metadata
  • Focus on minimal compliance first (Dublin Core)
  • Support the use of presets so when a new document is being created, the minimum amount of work is needed.
  • Start with something simple, plan for incremental improvements over time

Preceding work

QGIS Metadata Strategy:

Keep in mind the existing QEPs:

And keep in mind existing metadata editors:

Our intent is to build on these previous works and ideas by building a number of components that provide a comprehensive metadata strategy for QGIS.

Proposed Solution

The big picture of what we plan to produce is here:


We propose to first implement these components (WP = Work Package):

  • WP1: A QGIS Specific Metadata Schema (under the guidance of Angleos Tzotsos and Tom Kralidis)
  • WP 2: general purpose class for representing QGIS Metadata internally (under the guidance of Nyall Dawson)
  • A replacement for the current layer metadata tab that will have:
  • WP 4: A viewing mode where metadata is nicely laid out in an html document
  • WP 8: An editing mode where metadata can be edited.

We will make a separate QEP for the other work packages once these are the above ones are taken care of. More details on these Work packages can be found below.

Work package 1 - Schema Definition/Selection:

Input schema selection: In this phase we will identify input schemas to be used for validation. We propose initially to support Dublin Core and validate against the following CSW Record schemas:

Although to keep it simple, we start by supporting Dublin Core, we expect that the future evolution will be towards supporting ISO.

Internal Schema: In this phase we will specify a schema for internal representation of metadata within QGIS (‘the QGIS Metadata Schema’). This schema would be independent of any existing standards and would be the basic structure in which all incoming metadata would be stored. When we add support for additional formats in the future, the expectation would be that these formats are also transitioned to the QGIS internal format on import so that we can deal with a single common metadata structure internally.

Since the QGIS internal schema most likely won’t be a superset of all existing schemas, conversions between this and any other schemas may result in a loss information, which mean we won’t support metadata “round trips”. One proposed solution to loss of round tripping is to keep the original metadata document (if provided) and then interpolate new values into it if it is updated.

We will also identify which fields should be mandatory within the QGIS Metadata Schema. These should include mostly information which we can extract automatically from the dataset, without requiring any intervention from the user. Only in this way, we can guarantee the automatic generation of internal metadata for every dataset.

Other things to mention:

  • The QGIS metadata format will be used for when writing metadata into project files (and validator used to parse that back).

  • Service descriptions will be part of the schema

  • A document can describe either a service or a dataset, not both and not more than one dataset.

  • We plan to initially support layer level metadata and then project (service) level metadata in the future.

  • A small mockup by Tom to show how a QGIS Schema may look (incomplete) :
    We can take some inspiration from the GeoNode data model too - possibly providing a similar data model.
    Although we start with Dublin Core, keep ISO in mind, as a future target.

Status: A proposed schema has been written here qgis/qgis/#4330

Work package 2 - QGIS Metadata API:

In this work package we will build the basic C++ framework for parsing metadata from a schema - initially Dublin Core and QGIS Metadata Schema. This includes implementing an internal model for representing metadata, based on the metadata schema created on WP1.

Additional deliverables:

  • Regression tests to ensure that we do not have regressions in the metadata subsystem.
  • Clear API documentation, explaining how to use the API.
  • Example code, to be included in the QGIS python cookbook, in order to make it easy to “get started” manipulating metadata programmatically.
  • Python bindings, in order to expose the metadata classes to the plugin authors.

Work package 3 - Implement QGIS Metadata Storage support

In this WP we will introduce an external physical format for storing metadata internally, the “metadata store”. The goal is to support portability, enabling users to share their metadata, even in offline scenarios. This WP will build directly on the outputs of WP1, which will define an "internal metadata schema" and WP2, “QGIS metadata API”, which will encode/decode from the internal schema to the supported schemas (right now, only Dublin core).

screen shot 2017-04-04 at 11 57 36 am

QGIS will support two types of metadata stores: stores and local. In this WP we will focus on local stores, only. In the diagram below we depict the inheritance model for metadata stores, where an abstract metadata store will have a polymorphic behavior, according to the particular data format. For instance in the case of a PostgreSQL DB, the method “save” will create a table on the database, whether in the case of a Shapefile, it would create an XML file.

screen shot 2017-04-04 at 11 57 46 am

Some formats, such as text files, can be more limited than others. For that reason, we will create a “prime” format, the “QGIS metadata store”, which can accompany more restrictive formats.The prime format will be an SQLite database, because of its lightweight, and because it is well-known within the QGIS community.

As the goal is to support all these different formats in the future, we will design an infrastructure to accommodate that, but in this first iteration we will focus on the simple use case of creating an xml file, and an SQLite data store. The metadata contents will be passed by the metadata API. In this WP we will implement format translation, but not schema translation.

We will implement a user interface to allow the user to configure serialization/deserialization behavior, e.g.: in which format we should write metadata, and where. In WP5, we will add metadata detection (which perhaps we can turn on and off in the project settings). For instance, if there is an xml file with the same name and path as a Shapefile, QGIS would attempt to automatically import metadata.

The QGIS metadata store will be synced with any changes that we apply to the metadata. In the moment that we export metadata into XML, it will write those changes to the XML file.

Metadata search will also be polymorphic, according to the data format. In this iteration we will implement some text search for SQLite, and will use that rather than searching in text files which tends to be slower.


  • Design infrastructure for metadata storage in QGIS
  • Implement use case for “QGIS Metadata store” + XML
  • Sync “QGIS metadata store” with user edits.
  • Design and implement UI for serializing/deserializing behaviour.
  • Implement metadata search for this use case.


  • Infrastructure to accommodate the external storage of metadata in QGIS, fully implemented for the use case of XML files.
  • Support for searching the metadata store.
  • UI for saving/loading metadata.

WP1, WP2

Work package 4 - Implement QGIS metadata viewer:

Metadata is only useful if it is visible to the users of the dataset that the metadata is associated with. For this reason we should have provision for presenting the metadata in an eye-pleasing and informative manner and with minimal work required on behalf of the user. We also aim to implement this, earlier on in the project workflow, so that we can start outputting the data stored in QgsMetadata.

Some thoughts:

  • Implementation will be C++ with python bindings. For this reason we will not use Jinja2 or similar templating languages.
  • We should probably include derived data (even if it is not stored in the internal model) such as layer extents, min / max / mean etc. values and so on as there is also a utilitarian value to the metadata in QGIS. So this WP will display the information from the metadata storage + generate some metadata from the data.

The ideas is to replace this:


With something like this (taken from GeoNode):


Work package 8 - Implement QGIS metadata editor for layers

  • The idea would be to closely emulate the GeoNode editor under developement by GeoSolutions - prototype here:

In wizard mode:


In form mode:


  • For editing additional fields not covered by the guided interface, we could in future consider to use a tree editor as seen in the metatools plugin. See.



Affected Files

(required if applicable)

Performance Implications

(required if known at design time)

Further Considerations/Improvements

We have some funding to make these work packages happen (for around 80%) - if anyone is interested in co funding the shortfall, please let us know.

There is a discussion group at: for those who wish to collaborate in making QGIS metadata better.

The following people have already joined the effort and will be doing implementation work, planning, offering advice etc.

  • Tom Kralidis (advisor, metadata)
  • Angelos Tzotsos (metadata)
  • Joana Simoes (metadata)
  • Nyall Dawson (Core QGIS integration)
  • Tim Sutton / Ismail Sunni / Etienne Trimaille (QGIS Layer Properties Metadata Panel)

Backwards Compatibility

This will be new code and will replace any existing metadata implementation work (including what is currently in layer properties dialog). We will try to make sure that server and other parts that rely on metadata do not break - we would welcome support and input from those working on QGIS Server.

Issue Tracking ID(s)




@timlinux timlinux added this to the 3.0 milestone Mar 1, 2017
@timlinux timlinux added the Draft label Mar 1, 2017
@timlinux timlinux changed the title Overhaul Metadata Management In QGIS (in draft) Overhaul Metadata Management In QGIS Mar 1, 2017
Copy link

@samperd samperd commented Mar 20, 2017

@timlinux I was just notified of this thread. Recently I have been thinking of MD management within QGIS as well. What follows is a brain dump I shared via e-mail. So I release these ideas and thoughts to the wild as is.

Problem statement: although there are great tools for working with geospatial data, techs still spend a huge amount of time searching, collecting, storing and managing data.

Solution: A toolkit to facilitate a common work flow or tool set for GIS techs. Thinking of a Git Flow model for GIS.

User story: As a qgis user
I want a tool or catalogue to keep track of all the data I download or create throughout all my projects.
So that I can better manage my local data inventory, keep track of data sources, their age and any MD I create or import.

Audience: the solo data wrangler managing data and resources on the desktop. Not a large team environment but maybe a small team accessing the same data?

Develop a "catalogue for everyone". This is not an enterprise catalogue but a personal one.

It will use PYCSW as the back end. A simple qgis interface to create and manage MD. You can add local data to the catalogue or import one or more MD records from any CSW resource added to projects. Metasearch will be the search interface.

The Metadata editing interface will be created dynamically from the schema. Or perhaps a generic model with XSLT to transform into other schemas?

Records can be exported or pushed to other catalogues through CSW.

It will use OGR and GDAL tools to populate the known data .

ISO 19115 as the profile (HNAP?).

Manage both geo and non geo data. Tabular data and images.

Did I just describe metatools by nextGIS ( inactive for a year or more) or meta edit (dead since 2011)? They both seem to have parts. Next GIS was the company involved in metasearch at one point I think.

I also wonder if it should come with a template for directory structures to store and access local data. Similar to how GRASS works but not using that data format.

A tool that can manage downloaded data. For instance monitoring an FTP or HTTP directory. When data is updated it can either notify the user or download and update local dataset.

A tool to create data management plans.

A tool to create data quality reports and data dictionaries

just some random thoughts.

Copy link
Member Author

@timlinux timlinux commented Mar 20, 2017

Hi @samperd. Thanks very much for your brain dump. You will be pleased to know that a lot of your ideas are already incorporated into our thinking / planning. Take a look at our google doc for more details. I only included the 4 work packages above in this QEP because I don't want to muddy the waters by tabling too many features and sub proposals all at the same time. If metadata is something you care about, I encourage you to join our little subgroup in the gitter channel mentioned above.

Copy link
Member Author

@timlinux timlinux commented Apr 4, 2017

Update: We have updated the QEP to include Work Package 3 since on reflection we believe it needs to be implemented in the first round of development.

CC: @doublebyte1 @tomkralidis @nyalldawson @Gustry @kalxas

@kalxas kalxas mentioned this issue Apr 4, 2017
0 of 8 tasks complete
Copy link

@archaeogeek archaeogeek commented Apr 5, 2017

Sorry for the late response to this but I have only just been made aware of this document. Is this the best way to make comments or would you prefer I did that somewhere else?

One thing that immediately springs to mind is that the definition of a service is a bit fuzzy. Here you define it as a qgis project, but for INSPIRE etc service means a view or download service, eg a WMS/WFS server. I can see a scenario where a user might want to store WMS/WFS service metadata, and metadata about the layers those services provide, and metadata about the project. There's also the concept of how these things link together- eg WMS/WFS services have related child datasets, but then the relation between the datasets and some parent qgis project would also need to be considered.

Copy link
Member Author

@timlinux timlinux commented Apr 5, 2017

Note: Updated work package 1 to include a reference to the pull request for the proposed schema:

Status: A proposed schema has been written here qgis/qgis/#4330

Copy link
Member Author

@timlinux timlinux commented Apr 5, 2017

@archaeogeek comments here are welcome, we also have a live chat channel on gitter at

Currently we have only two concepts:

  • layer level metadata (which maps to a QgsMapLayer)
  • service level metadata (which maps to a QgsProject - with the expectation that it provides the service capability response when publishing that project via QGIS Server)

It would be great to get your inputs about what design changes you would envisage. At the schema level this comments are probably best directed to the PR at qgis/qgis/#4330. At the higher level, it would be great to hear your ideas on practically when where and how the concepts you propose would be captured and displayed within the QGIS desktop application.

Bear in mind we have further work packages planned - this QEP covers only the first pass implementation addressing layer level metadata. Perhaps take a look at our scratch document to see what other things we plan for the short term.

Copy link

@mhugo mhugo commented Apr 5, 2017

Hi, about the "QGIS Metadata storage", could it be implemented with our proposed "auxiliary storage" (#27 - which we will hopefully start soon) ? Or do you need something more / different ?

Copy link

@doublebyte1 doublebyte1 commented Apr 7, 2017

Hi @mhugo, I think the objective here is slightly different, since AFAIU you want to store auxiliary data for a layer (row based) and we want to store the schema described in WP1 (layer based). We will use the same backend (SpatiaLite) for now, but the idea is to support a polymorphic storage. You can read more details in this blog post:
Beyond the scope of this enhancement, I really like your idea of changing the project storage format. In this plugin, we though of actually packaging the whole project as a spatialite database.

Copy link

@mj10777 mj10777 commented Apr 8, 2017

What thoughts are being made to add a 'Copyright' field

  • that Acknowledge the copyright and the source of the data

to the Layers Panel Description area?

This is often a requirement for the use of OpenData sources, such as:
Acknowledge the copyright and the source of the data by including the following attribution statement: ‘Contains Ordnance Survey data © Crown copyright and database right 2013’.

When geo-referencing an image from such a source, I add a

  • TIFFTAG_COPYRIGHT to the original image

which is then taken over when geo-referencing the image with gdal, being one of the tags supported by Geo-Tiffs:

  • look for 'Metadata'

For the RasterLite2 project, Alessandro Furieri (Sandro) and I discussed this matter and the final decision was to add a copyright and licence fields, together with the existing title and abstract fields for both

  • raster and vector data

to the metadata tables.

The next spatialite version will create a data_licenses table and fill it with a list of common licenses for all new Databases.

The idea being, in this way, to avoid any legal hassle by offering a means to store and for views to display the information as needed or required.

So when a Geo-Tiff is being imported, when the following TIFFTAGs are found:

  • TIFFTAG_COPYRIGHT as copyright

they will be taken over.

After reading this and the other concepts, I was surprised to see that this aspect was not included.
It seems that even the 'Dublin Core' and 'ISO 19115' also do not deal with this

In the case of 'Dublin Core', it seems to even have been removed:
As a result of the restricted focus of the workshop, certain issues required for a complete description of DLOs, such as cost, archival status and copyright information, were eliminated from the scope of the discussion.

But for QGIS, I would say, it would be wise to also to avoid any legal hassle by adding a copyright field to to Layers Panel Description area and the metadata concept.

Copy link
Member Author

@timlinux timlinux commented Apr 11, 2017

@tomkralidis @kalxas @archaeogeek What do you guys think of @mj10777's proposal? I'm OK to add it, or at least something that indicates ownership and usage terms of the data.

Copy link

@archaeogeek archaeogeek commented Apr 11, 2017

I'm in full agreement. Dublin Core is pretty limited, so adding something for ownership/access constraints would be helpful. I wouldn't describe it as copyright though, as that's just one form of access constraint.

Copy link

@nyalldawson nyalldawson commented Apr 12, 2017

@tomkralidis @kalxas @archaeogeek What do you guys think of @mj10777's proposal? I'm OK to add it, or at least something that indicates ownership and usage terms of the data.

I'm +1, but would prefer "attribution" over "copyright". Many geospatial datasets are now under copyleft, so naming the field "copyright" just seems wrong to me.

Copy link

@tomkralidis tomkralidis commented Apr 12, 2017

Can the rm:rights element cover attribution as well?

Copy link

@nyalldawson nyalldawson commented Apr 12, 2017

Can the rm:rights element cover attribution as well?

Yes - ignore my comment - I think it's fine without a dedicated attribution/copyright tag

Copy link

@mj10777 mj10777 commented Apr 12, 2017

Important is only that a User does not get into trouble by any infringement of copyright laws.
It should be clear where to place this information

  • i.e. where something in the form of TIFFTAG_COPYRIGHT should be placed

Can the rm:rights element cover attribution as well?

Sounds good for me together with some sort of licence tag.

Copy link

@ninsbl ninsbl commented Nov 3, 2017

Seen qgis/QGIS#5467 which includes storage of alias and comments for fields / attributes. That important aspects of metadata from my point of view. Just to make sure it is considered here as well...

Copy link

@mj10777 mj10777 commented Nov 6, 2017

Question to the use of QgsLayerMetadata

My assumption is that it a Provider would be a major source to gather the needed information an set QgsLayerMetadata.
I am doing this now for the new QgsSpatiaLiteProvider.

QgsDataProvider does not contain (at present) a mMetadata; member as QgsMapLayer does.

So when setDataProvider runs in QgsVectorLayer and QgsRasterLayer, any Metadata gathered by a provider (in the form of QgsLayerMetadata) cannot be set.

  • setMetadata(mDataProvider->metadata());

So adding a metadata() and setMetadata(..) in QgsDataProvider would be needed to make QgsLayerMetadata truly usefull.

Copy link

@georgecorea georgecorea commented Jul 26, 2019

Is it possible to also include a list of all the fields in the file and the value and number of unique values in some key fields that the user can select into the metadata?
Annotation 2019-07-26 111753

Additionally is anyone creating a stylesheet that can be used to format the qmd into html.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
10 participants
You can’t perform that action at this time.