Client For Document Management System in QGIS

[troopa81]

Client For Document Management System in QGIS

Date 2020/09/15

Author Julien Cabieces (@troopa81)

Contact julien dot cabieces at oslandia dot com

maintainer @troopa81

Version QGIS 3.18

Summary

QGIS can view and edit a link to an external resource (local file or distant URL) using the Attachment widget.

Usually, users store their data on an external system (shared directory or file hosting service).

For now, they need to push their data on their system then copy resulting URL and paste it in the Attachment widget (or select it with the file widget selector if the file is on a local directory).

This QEP proposes to have a more integrated behavior, so if the attachment widget has been properly configured, user just needs to drag'n drop a document to upload it on the external system and update the attribute URL once it's done.

This proposal is not about implementing a DMS in QGIS, but to have QGIS become a nice client for DMS protocols.

Here is what it could look like

Different type of external system

All users do not store their files the same way, so the new feature should be able to support different type of storage system. A non exhaustive list could be:

• Network shared directory: Samba mount, NFS. In this case, a simple file copy is needed.
• WebDAV protocol: It's an HTTP protocol to create and change document on a server. It's widely supported by file hosting system (Nextcloud, Pydio, Alfresco). It can also be set using a classic HTTP server (with Nginx for instance)
• Object Storage : All Cloud providers proposes an Object Storage solution (Google, Amazon, or any one based on OpenStack). It uses HTTP GET/POST requests to store and retrieve files.
• FTP

This proposal consists in implementing the first 2 items (Simple copy and WebDAV) and making a generic enough API so other systems could be added later.

Proposed Solution

UI

From Layer properties > Attributes form > Attachment Widget type, user is able to configure the different ways of storing the selected file.

Local file selection is the current attachment behavior, no copy or upload is triggered, the attribute is simply updated with the selected file path.

Each external system method provides its own configuration widget

Remove file

In configuration dialog, user can choose if he wants to delete the file when it's removed or updated in the form, or if he wants to let it on server side.

Drag 'n drop

Drag'n drop of local file on attachment file widget is to be supported to enhance the user experience.

Metadata

From the configuration widget, user can set up a list of metadata attributes (or regular expression) to be displayed in the Attachment widget.

Project Storage

New external storage backend are exposed as project storage backend so it's possible to store/retrieve project from this external system.

Marker symbol renderer

User can select an external storage from which he wants to retrieve images to apply them as marker symbol. For now, it's already possible to set URL, but there is no way to give credentials or to manage non-http external storage.

This will be taken care of in the second step of the implementation.

API

New classes:

• QgsExternalStorage (abstract) defines several functions to be implemented to

  • Store asynchronously a resource and return the given URI
  • Retrieve asynchronously a resource from a URI (and given credentials). It may potentially reuse code in QgsAbstractContentCache
  • Get file metadatas
  • Define a function that provides a configuration widget

• QgsExternalStorageRegistry allows to register different methods for dealing with external resources (WebDAV, FTP, simple copy, ObjectStorage...)

• QgsWebDAVExternalStorage WebDAV implementation for QgsExternalStorage

• QgsSimpleCopyExternalStorage simple copy implementation for QgsExternalStorage

• QgsFileWidgetUploader extends existing QgsFileWidget to upload file when file is selected and update URI when it has finished. EDIT: Actually, I'm thinking more and more to no inherit but simply add the external storage capability to existing QgsFileWidget without changing the default (no external storage) behavior.

• QgsProjectExternalStorage wraps a QgsExternalStorage so we can use it to store project. When we register a new external storage, it register a wrapped QgsProjectExternalStorage in QgsProjectStorageRegistry

Implementing the new API class allows to add any external file hosting system either in QGIS Core or in a QGIS plugin.

Authentication

External system connection often requires authentication.
The different implementations of QgsExternalStorage rely on QgsAuthManager to manage the authentication credentials and QgsNetworkManager to deals with HTTP requests when necessary.

Performance Implications

Uploading file will be done asynchronously in a separated thread so it doesn't block the application (using potentially QgsTaskManager). This task could be canceled by the user directly from the UI.

Further Considerations/Improvements

Another external system which could be considered as a possible good way for storing files is PostGreSQL Large object. It could implement the same API described above.

Backwards Compatibility

All existing attachment widget configuration will lack the storage mode configuration parameter. When missing this parameter will default to Local file selection storage mode which is the current attachment widget behavior.

Votes

(required)

[pcav]

Very interesting, +1 for me.

[3nids]

Nice!
One question: have you considered how you will block the form validation while the file is being uploading and waiting for the URL in return? Or do you expect the URL to be generated directly?

[troopa81]

One question: have you considered how you will block the form validation while the file is being uploading and waiting for the URL in return? Or do you expect the URL to be generated directly?

I don't plan to block the form validation. I think about updating the field attached to attachment widget only when the upload has finished, so while the file is being uploaded, the field remains empty, meaning invalid if user selected a "not null" constraints for instance.

[wonder-sk]

Sounds interesting!

Some thoughts:

• how would the URLs of external resources look like?
• it would be good to have support for cancellation of async uploads in the interface
• if I decide to remove the linked file, is it going to get removed from the storage as well? Ideally there should be an option to do so
• shouldn't QgsExternalResourceMethod be able to also retrieve previously stored object? For example, if I have a picture assigned to each feature, and want to access it outside of forms - e.g. when rendering map to show thumbnails
• it would be interesting to think about bigger picture - currently we support saving/loading QGIS project files in postgresql or geopackages - and then refer to them with specific URLs. It would be nice to design this "external resource method" mechanism in a way that the implementation of project load/save mechanism could use it too, and we would get more backends for QGIS projects "for free". And of course this wouldn't be just for projects, but anything really that can be stored as a blob. (maybe worth looking at QgsProjectStorage and QgsProjectStorageRegistry)

[troopa81]

@wonder-sk

how would the URLs of external resources look like?

Something like https://myserv.com/webdav with the ability to set up an expression so you can generate the URL based on attribute values (If you want to upload a document in a specific folder according to its ID for instance). We could also have an option to store relative path with a base url.

it would be good to have support for cancellation of async uploads in the interface

Sure. I'll add it the QEP

if I decide to remove the linked file, is it going to get removed from the storage as well? Ideally there should be an option to do so

Agree, If you want to remove the file on the feature, or replace it with one with a different name, we should propose an option to remove it from the storage.

However, if you remove the feature whom one attribute contains the uploaded URL, the file wouldn't be removed. This case is more a backend/database concern, and should be treated with a trigger or a cron procedure to remove orphan files.

shouldn't QgsExternalResourceMethod be able to also retrieve previously stored object? For example, if I have a picture assigned to each feature, and want to access it outside of forms - e.g. when rendering map to show thumbnails

It could be nice indeed, but that means that the definition and configuration of an external resource method cannot be done only at form/widget level. It should be done at layer level so the rendering can access it. Maybe, we could add a specific tab External Resources to configure external resource behavior on fields.

it would be interesting to think about bigger picture - currently we support saving/loading QGIS project files in postgresql or geopackages - and then refer to them with specific URLs. It would be nice to design this "external resource method" mechanism in a way that the implementation of project load/save mechanism could use it too, and we would get more backends for QGIS projects "for free". And of course this wouldn't be just for projects, but anything really that can be stored as a blob. (maybe worth looking at QgsProjectStorage and QgsProjectStorageRegistry)

Good idea. I'll take a look on this and update the QEP accordingly.

[troopa81]

I finally found the time to update the QEP according to last @wonder-sk comments.

As nobody seems to be against such a feature, I will start developing in a few days.

Thanks everybody for your relevant comments.

[haubourg]

@troopa81 thanks for the update, I think this is now of great interest to me

[Jean-Roc]

@troopa81 could Oslandia make a blog post about this work to explain it to a larger public ? it would be great to spark interest about the wide range of possibilities it offers (sharepoint, alfresco, etc.).

[troopa81]

@troopa81 could Oslandia make a blog post about this work to explain it to a larger public ? it would be great to spark interest about the wide range of possibilities it offers (sharepoint, alfresco, etc.).

Sure, it will be done once I get it merged!

[troopa81]

Now fully available in QGIS, with also S3 support