INTERLIS import/export interface for VSA-KEK

[kandre]

General information

• Applicant: Alexandre Bosshard and Stefan Burckhardt, coordinators and project leaders of the QWAT/QGEP group
• Developer: OPENGIS.ch GmbH
• User group referent: Stefan Burckhardt
• Developments referent: Olivier Dalong, Opengis

Users and administrators needs

Graphical identity / Logo

Illustrations if applicable.

Description of users and administrators feature requirements

This feature should allow any GIS administrator or trained QGIS user to
a) Export network data (full or part of the network) to SIA405 Abwasser or VSA-DSS (2015)
b) Select language de, fr
c) Import VSA-KEK damage code data
d) Check whether it fits to existing network data. Get hints if there is missing or new network data
e) Import additional network data in SIA405 Abwasser or VSA-DSS (2015)

General requirements

See general requirements of QGEP #600 .

Related issues:

• qwat 307
• QGEP #603 .

Specific requirements (QGEP and VSA-KEK #600)

Necessary features:
Extension of the current WincCan2QGEP Plugin

Import:

• Add VSA-KEK 2019 model version as additional import format for the import of damage code data
• Integrate the existing INTERLIS Import for SIA405 Abwasser / VSA-DSS in WinCan2QGEP workflow or integrate it in QGEP separatly
• Create and display log file

Export:

• Integrate the existing INTERLIS Export for SIA405 Abwasser / VSA-DSS in WinCan2QGEP workflow or integrate it in QGEP separatly
• Create and display log file

This would be the basis to make a correct INTERLIS export e.g. for a TV inspection company or to another engineering company or the community or the canton. Some consideration and discussion about this has already taken place in #218

Illustrations of the features and the requirements

Schemes, screen captures, or any other illustrations.

Further details in here:
Workflow with existing data in VSA-DSS: Overview

Further details in here:
VSA-DSS_Transferdatensatz_INTERLIS2_Erlaeuterungen_Kap6.pdf

Graphical User Interface (GUI)

Technical analysis and specifications

Feasibility study

Feasibility, options, alternatives and technical specificities

• Check option to use the same INTERLIS import approach also for VSA-KEK datamodel

Risks and constraints

Mutualisation

Mutualisation and possible impacts with other projects (QGIS, QWAT, QGEP)

Selecting a subset of data in QGEP and having the option to

• store it some way for temporary use
• do any operation with it such as QGIS operations or export or setting a comon attribut or whatever action
  could be interesting for different other daily workflows too.

Cahier des charges et offres ( Developer)

Not shared on Github.

Specifications

Tenders

Not shared on Github.

Timeline / deadlines

• 01.09.2020 : Contract signature

• 11.09.2020: Preliminary meeting for user needs

• 30.09.2020: Draft for user needs and specifications

• 30.10.2020: User needs and specifications delivered to the developer

• 31.12.2020: 1st intermediate report to NGDI (Stefan)

• 28.02.2021: VSA-KEK INTERLIS import/export interface release (and demo - during meeting 19.3.)

• 30.01.202131.03.2021: SIA 405 INTERLIS import/export interface release (and demo ?)

• 31.03.2020 - 30.05.2021: Quality control and bug fixing

• 31.04.2021: 2nd intermediate report to NGDI (Stefan)

• 31.05.2020: Users tests in customers' spatial data infrastructures

• 30.06.2021: Corrections and bug fixing

• 30.07.2021: Documentation release: 30.07.2021

• 31.08.2021: Interfaces SIA 405 for QWAT and VSA-KEK for QGEP are operationnal and documented. Final report to NGDI (Stefan).

• Warranty period : to be adressed

Versions of related softwares

• QGIS version: 3.16 or higher
• Database model version: QGEP and QWAT (1.5.6/1.3.6) or newer
• INTERLIS Model version: 2015

Public issues links :

• …

Helpdesk issues links :

• …

Dates de réalisation

• Tender reception date :
• Tender validation date :

Documentation

Documentation links

• User manual : https://qgep.github.io/docs/admin-guide/interlis-io/index.html
• Administrator manual : https://qgep.github.io/docs/admin-guide/interlis-io/index.html and https://github.com/QGEP/qgepqwat2ili/

Development and intégration

Link and comments from the developer about the product

• Deliverable :
• Recommendations and comments :
• Future prospects :

Conclusion

Conclusions from applicant, users or admins group

• Conclusion:

• Future prospects: Integrate options to select different datamodels (full VSA-DSS, SIA405 Abwasser (network data only)

• Invoice date: 30.9.2021

• Payment date:

[kandre]

This issue is our shared document to define user needs, specifications, timeline and to follow up the execution.
As this feature should be thinked and developed in a participative way, feel free to react by using the comments below.
Issue #307 in QWAT is not a duplicate, but very similar as we aim at mutualising as much as possible this development between those projects.

@sjib @ponceta feel free to complete of make corrections.

[urskaufmann]

General Requirements / Export Log:
I think there should be a log file. I know the Logfiles of the infogrips-tools:

• which configuration / which model is used for the export
• number of records per class to be exported
• number of records in the xml-file per class
• errors

[urskaufmann]

General Requirements / Import:
When is a record a duplicate? Same obj_id, same identifier, same geometry?
If the TV-inspection company starts with a SIA405-export, then we will have duplicate records, but we have no problems - I do not want to see duplicates. I think in this case the obj-id will be the same and the datamodel is vsa-kek.
In other cases of importing data, I will have to look very good, if there are duplicate identifiers or geometries.
General: if there are new (wastewater structure or reach) records, I do always want to have this records in quarantine, then see this new records together with the existing records and then decide one by one, if I want to keep just new or just old or old with new attributes or both/all records. Quarantine is necessary.

[m-kuhn]

For the logs, some feedback about technical feasibility.
As a first step we will show the logs and errors from ili2pg, which does a lot of interlis checks out of the box (see ilivalidator).
E.g. duplicate object ids will according to my current understanding be reported. Depending on parameters as errors or warnings.
The format of this can be tested through qgis model baker.

Changing ili2pg is out of scope for this project. I would propose to see what we get already from ili2pg and only after discuss what needs further inspection of the imported data and where follow ups need to be started to improve ili2pg.

[ponceta]

@kandre as discussed today, @sjib provides as a table codes and values_fr and values_de for the VSA KEK extension

https://github.com/QGEP/datamodel/blob/master/04_vsa_kek_extension.sql#L264

About error logs :

Provide a simple error log : translation has run correctly or not, display errors, display entities count (especially if translation was partially efficient)

[olivierdalang]

@sjib @urskaufmann @ponceta
I'm starting to work on this, but need a little bit of information.

• Is https://www.sia.ch/de/dienstleistungen/sia-norm/geodaten/ the correct/official place from which to retrieve the current model ?
• From the files retrieved on the page above, which ili models exactly do we want to use ? SIA405_Abwasser_3D_2015_2_d-20180417.ili or SIA405_Abwasser_2015_2_d-20180417.ili ? (or even something else ?)
• With both of these files, I get failures with ili2pg (see more information below).
  • Any idea of what reason could make ili2pg fail with these newer releases, or even better, how to fix this ?
  • Does it work for you ?
  • If no to the two above, can we work on older releases of the ili model, or are they considered obsolete ? If we can, where can we get older release from ?
• Do you have some sample data in interlis format (to test importing into QGEP) ? (without that, we can only start working on import once export is functional, and it would be better to also try with actual data other than files we produced ourselves)
• All the above is about SIA 405 only. Is there a model definition for VSA-KEK as well ? Where can I get it from ?
• Do you have any existing scripts/procedures that you could share that could serve as example/starting point ?

---

Errors with ili2db and latest SIA405_Abwasser_2015 release

I've been able to create the PG model by adapting some scripts from another project (planteam) with an older version of the release (VERSION "14.09.2015"), but get errors when trying with newer version of the ili model (see logs below).

Error with SIA405_Abwasser_2015_2_d-20180417.ili:

[...]
Info: create table structure, if not existing...
java.lang.NullPointerException

Error with SIA405_Abwasser_3D_2015_2_d-20180417.ili, :

[...]
Info: ilifile <sia405_interlis_files\SIA405_Abwasser_3D_2015_2_d-20180417.ili>
Error: sia405_interlis_files\SIA405_Abwasser_3D_2015_2_d-20180417.ili:31:SIA405_ABWASSER_3D_2015.SIA405_Abwasser_3D must be declared as ABSTRACT. One of its elements (Abwasserbauwerk) is abstract.
Error: sia405_interlis_files\SIA405_Abwasser_3D_2015_2_d-20180417.ili:31:SIA405_ABWASSER_3D_2015.SIA405_Abwasser_3D must be declared as ABSTRACT. One of its elements (Abwassernetzelement) is abstract.
Error: sia405_interlis_files\SIA405_Abwasser_3D_2015_2_d-20180417.ili:31:SIA405_ABWASSER_3D_2015.SIA405_Abwasser_3D must be declared as ABSTRACT. One of its elements (BauwerksTeil) is abstract.
Error: sia405_interlis_files\SIA405_Abwasser_3D_2015_2_d-20180417.ili:276:SIA405_ABWASSER_3D_2015_LV95.SIA405_Abwasser_3D must be declared as ABSTRACT. One of its elements (Abwasserbauwerk) is abstract.
Error: sia405_interlis_files\SIA405_Abwasser_3D_2015_2_d-20180417.ili:276:SIA405_ABWASSER_3D_2015_LV95.SIA405_Abwasser_3D must be declared as ABSTRACT. One of its elements (Abwassernetzelement) is abstract.
Error: sia405_interlis_files\SIA405_Abwasser_3D_2015_2_d-20180417.ili:276:SIA405_ABWASSER_3D_2015_LV95.SIA405_Abwasser_3D must be declared as ABSTRACT. One of its elements (BauwerksTeil) is abstract.
compiler failed

I also tried with a slightly older ili2pg version (4.1.0), and then get the same error with the two files, but also slightly more information:

mapping of ili-classes to sql-tables failed
  mapping of SIA405_ABWASSER_3D_2015_LV95.SIA405_Abwasser_3D_LK.Abwasserbauwerk failed
    SIA405_ABWASSER_3D_2015_LV95.SIA405_Abwasser_3D_LK.Abwasserbauwerk.Detailgeometrie
      java.lang.NullPointerException

[sjib]

@olivierdalang

• I would start with the 2D version. In the new release 2020 the 3D extension will be modelled a bit differently and I think easier (just another EXTENDS).

• Can you send me the logs for SIA405_Abwasser_2015_2_d-20180417.ili then I can look at the failures and check, whether it is because of the model or ili2pg

• I developed the whole INTERLIS Import / export with ili2pg 3.12.2

• I can share those with you

• VSA-KEK should be available end of next week (see www.vsa.ch/models/2019 -> VSA-KEK)

[olivierdalang]

Thanks !

@sjib
Here's the full log with SIA405_Abwasser_2015_2_d-20180417.ili (added the --trace argument) : fail.log

[sjib]

VSA-KEK Testdata
Kanalschaden (channel_damage): both reference systems / reference by identifier / reference by oid

• testdata_vsa_kek_2019_channel_damage_8486.xtf
• testdata_vsa_kek_2019_channel_damage_8486_identifierref.xtf: <vonPunktBezeichnung>8486</vonPunktBezeichnung> and <bisPunktBezeichnung>8486.0010</bisPunktBezeichnung>
• testdata_vsa_kek_2019_channel_damage_8486_oidref: ` and

Schachtschaden (manhole_damage): both reference systems / reference by identifier / reference by oid

• testdata_vsa_kek_2019_manhole_damage_8486.xtf
• testdata_vsa_kek_2019_manhole_damage_8486_identifierref.xtf: <vonPunktBezeichnung>8486</vonPunktBezeichnung>
• testdata_vsa_kek_2019_manhole_damage_8486_oidref.xtf: <HaltungspunktRef /> <!-- not necessary for manhole damages --> <AbwasserbauwerkRef REF="ch13p7mzMA007381" /> <!-- identifier 8486 -->

If both reference systems are in a file then the oid reference system should be used.

`

[olivierdalang]

(attaching the data here so it's easier to keep track) inspectiondata20210126 (1).zip

[olivierdalang]

Current progress report (as presented during code sprint) : https://docs.google.com/document/d/14vW7cjpbyqXFGuwAvThzFhNPDjIiOYTy8csHz9h8bt4/edit?ts=6053791e#

[olivierdalang]

Implementation for this is tracked here : https://github.com/QGEP/qgepqwat2ili
(included as submodule in qgepplugin)