Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Missing classes INTERLIS VSA-DSS 2020 Import / Export #98

Closed
sjib opened this issue Jan 22, 2024 · 9 comments · Fixed by #111
Closed

Missing classes INTERLIS VSA-DSS 2020 Import / Export #98

sjib opened this issue Jan 22, 2024 · 9 comments · Fixed by #111
Labels
bug Something isn't working INTERLIS About INTERLIS exchange format (import / export) plugin Concerns the wastewater plugin

Comments

@sjib
Copy link
Contributor

sjib commented Jan 22, 2024

@domi4484 For VSA-DSS 2020 INTERLIS Import / Export I find only two additional classes wastewater_treatment_plant and wwtp_structure (compared to SIA405 Abwasser) in the matching configuration:

INTERLIS Import:

wastewater/plugin/teksi_wastewater/interlis/interlis_model_mapping/interlis_importer_to_intermediate_schema.py

Line 160 in 4d4b863

def _import_dss(self):

INTERLIS Export:

wastewater/plugin/teksi_wastewater/interlis/interlis_model_mapping/interlis_exporter_to_intermediate_schema.py

Line 155 in 4d4b863

def _export_dss(self):

Am I checking on the wrong branch or where are all others?
@sjib sjib added bug Something isn't working INTERLIS About INTERLIS exchange format (import / export) plugin Concerns the wastewater plugin labels Jan 22, 2024
domi4484 added a commit that referenced this issue Jan 23, 2024
@domi4484
Add missing DSS classes. Fix #98
a5f7751
domi4484 added a commit that referenced this issue Jan 23, 2024
@domi4484
Add missing DSS classes. Fix #98
e69d63f
@3nids
Copy link
Contributor

3nids commented Jan 23, 2024

should be fixed in last plugin demo, please re-open if needed

@3nids 3nids closed this as completed Jan 23, 2024
@sjib sjib reopened this Jan 23, 2024
@sjib
Copy link
Contributor Author

sjib commented Jan 23, 2024

@domi4484 I still don't see e.g. classes like catchment_area, log_file etc. etc. in the configuration scripts (correct me if I checked the wrong branch)

Based on the ilivalidator check stats you should get the following classes and number of objects imported:

Info: C:\Daten\QGEP\testing_dss2020\test-dataset-DSS.xtf: DSS_2020_1_LV95.Siedlungsentwaesserung BID=xBASKET1
Info:      10 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.ARABauwerk
Info:       1 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.ARAEnergienutzung
Info:       1 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Abflusslose_Toilette
Info:       2 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Absperr_Drosselorgan
Info:       1 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Abwasserbauwerk_Symbol
Info:       1 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Abwasserbauwerk_Text
Info:       4 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Abwasserbehandlung
Info:      88 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Abwasserknoten
Info:       1 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Abwasserreinigungsanlage
Info:       4 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Bankett
Info:       1 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Beckenentleerung
Info:       1 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Beckenreinigung
Info:       1 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Biol_oekol_Gesamtbeurteilung
Info:       1 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Brunnen
Info:      60 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Deckel
Info:       1 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.EZG_PARAMETER_ALLG
Info:       7 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Einleitstelle
Info:       6 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Einstiegshilfe
Info:       3 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Einzelflaeche
Info:       4 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Einzugsgebiet
Info:       1 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Einzugsgebiet_Text
Info:       4 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.ElektrischeEinrichtung
Info:       1 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.ElektromechanischeAusruestung
Info:       4 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Entsorgung
Info:       2 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Entwaesserungssystem
Info:       7 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Erhaltungsereignis_AbwasserbauwerkAssoc
Info:       2 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Feststoffrueckhalt
Info:       3 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.FoerderAggregat
Info:       6 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Gebaeude
Info:       5 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Gebaeudegruppe
Info:       1 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Gebaeudegruppe_BAUGWR
Info:       5 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Gebaeudegruppe_EntsorgungAssoc
Info:       1 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Gesamteinzugsgebiet
Info:       6 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.HQ_Relation
Info:     102 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Haltung
Info:       1 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Haltung_Alternativverlauf
Info:       2 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Haltung_Text
Info:     204 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Haltungspunkt
Info:      18 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Hydr_GeomRelation
Info:       7 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Hydr_Geometrie
Info:       2 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Hydr_Kennwerte
Info:       1 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.KLARA
Info:     102 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Kanal
Info:       1 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Landwirtschaftsbetrieb
Info:       1 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Leapingwehr
Info:       3 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Massnahme
Info:       2 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.MechanischeVorreinigung
Info:       2 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Messgeraet
Info:       3 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Messreihe
Info:    4363 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Messresultat
Info:       3 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Messstelle
Info:      39 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Mutation
Info:      49 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Normschacht
Info:       1 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Reservoir
Info:       1 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Retentionskoerper
Info:       5 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Rohrprofil
Info:      10 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Rohrprofil_Geometrie
Info:       1 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Rueckstausicherung
Info:       1 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Schlammbehandlung
Info:      19 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Spezialbauwerk
Info:       1 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Spuelstutzen
Info:       6 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Stammkarte
Info:       1 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Steuerungszentrale
Info:       3 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Streichwehr
Info:       1 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Trockenwetterfallrohr
Info:       2 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Trockenwetterrinne
Info:       2 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Ueberlaufcharakteristik
Info:       4 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Unterhalt
Info:       2 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Versickerungsanlage
Info:       1 objects in CLASS DSS_2020_1_LV95.Siedlungsentwaesserung.Versickerungsbereich
Info: ...validation done
Info: End date 2024-01-23 13:11 validation took 00h:00m:21s.0974ms

They should all show up here:
20240122_dss_import_new_plugin_version_wwtp_etc

3nids pushed a commit that referenced this issue Jan 23, 2024
@domi4484 @3nids
Add missing DSS classes. Fix #98
c7c833b
@domi4484
Copy link
Contributor

domi4484 commented Jan 24, 2024 •
edited
Loading

@sjib sorry i see I added only a small subset, no there isn't any other branch.

Now I am adding (hopefully) all the missing classes and I am missing some matches.

What is the TEKSI corresponding of:

  • FoerderAggregat

Waht is the Interlis corresponding of:

  • import_manhole_quarantine
  • param_ca_mouse1 -> EZG_PARAMETER_MOUSE1 found that one

@sjib
Copy link
Contributor Author

sjib commented Jan 24, 2024 •
edited
Loading

@domi4484
You can find all translations in the tww_sys tables dictionary_od_table, dictionary_od_field and dictionary_od_values.

So you don't have to invent any translation youself of the datamodel. For any other terms I can help you.

Förderaggregat is pump - INTERLS does not know Umlaute like ü, ä, ö but uses ue, ae, oe

@sjib
Copy link
Contributor Author

sjib commented Jan 24, 2024

import_manhole_quarantine is a special table used when importing field data with qfield - so it is not part of the vsadss 2020 datamodel.
In qgep it was in schema qgep_import.manhole_quarantine

@sjib
Copy link
Contributor Author

sjib commented Jan 26, 2024 •
edited
Loading

Datamodel DEREFERABLE:

Currently the following attributes have DEREFERABLE in the teksi wastewater datamodel (based on VSA-DSS 2020.1) :

  • wastewater_structure.fk_owner and fk_operator (linking on organisation)
  • maintenance_event.fk_operating_company
  • wastewater_networkelement.fk_wastewater_structure
  • reach_point.fk_wastewater_networkelement
  • structure_part.fk_wastewater_structure
  • all classes with fk_dataowner, fk_provider (linking on organisation)

@domi4484 As we have to import organisation first anyway, why did you set fk_owner, fk_operator, fk_dataowner and fk_provider as DEREFERABLE ?
I understand that we should set these attributes in qgep as DEREFERABLE where organisation data is included in the transfer xtf - I think this would fix errors like QGEP/qgepqwat2ili#125 QGEP/qgepqwat2ili#126 and QGEP/qgepqwat2ili#118, but with VSA-DSS 2020 this has changed.

@domi4484 When finalizing the configuration for VSA-DSS 2020 I would suggest to check, if it would make sense to also set the following fk attributes as DEREFERABLE :

and if you think links on organisation still need to be DEREFERABLE also:

  • measure.fk_responsible_entity (linking on organisation)
  • measure fk_responsible_start (linking on organisation)
  • log_card.fk_agency (linking on organisation)
  • log_card.fk_location_municipality (linking on organisation)
  • measuring_point.fk_operator (linking on organisation)

I do not know how much resources or time it costs if there are additional attributes set as DEREFERABLE and if there could be conflicts if there are too many.

@sjib sjib mentioned this issue Jan 26, 2024
Closed
@domi4484
Copy link
Contributor

As we have to import organisation first anyway, why did you set fk_owner, fk_operator, fk_dataowner and fk_provider as DEREFERABLE ?
I understand that we should set these attributes in qgep as DEREFERABLE where organisation data is included in the transfer xtf - I think this would fix errors like QGEP/qgepqwat2ili#125 QGEP/qgepqwat2ili#126 and QGEP/qgepqwat2ili#118, but with VSA-DSS 2020 this has changed.

@sjib probably you are right and it could be removed again for those.

@domi4484 domi4484 mentioned this issue Jan 28, 2024
Merged
domi4484 added a commit that referenced this issue Jan 28, 2024
@domi4484
Add plugin as it is now from https://github.com/teksi/wastewater_old
ee81c66 
qgep qwat to ili is added as not submodule

Remove timeout option. Fix #76

Show logs button visible only if log_path is set

Add missing DSS classes. Fix #98

remove profile action

Update plugin/teksi_wastewater/interlis/interlis_importer_exporter.py

remove libs from source

[pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

fix

sudo

plugin-ci

no sudo
@sjib
Copy link
Contributor Author

sjib commented Jan 30, 2024 •
edited
Loading

@domi4484 There is still the class reach_progression_alternative missing and should be added to INTERLIS Import and Export:

Haltung_AlternativVerlauf (reach_progression_alternative)

CLASS Haltung_Alternativverlauf EXTENDS Base_LV95.BaseClass =
!!@comment = "Anfangs-, Knick- und Endpunkte des Alternativverlaufs der Leitung im gewählten Plantyp (z.B. Uebersichtsplan)"
   Verlauf: Base_LV95.Polyline;
!!@comment = "Default: Uebersichtsplan"
   Plantyp: SIA405_Base_Abwasser_1_LV95.Plantyp;
END Haltung_Alternativverlauf;

If I read the code right it should be like this and I would add it after the #TEXTS block here:

wastewater/plugin/teksi_wastewater/interlis/interlis_model_mapping/model_interlis_sia405_abwasser.py

Line 165 in fd78330

ModelInterlisSia405Abwasser.abwasserbauwerk_text = abwasserbauwerk_text 

# REACH_PROGRESSION_ALTERNATIVE

class reach_progression_alternative(baseclass):
            __tablename__ = "reach_progression_alternative"
            __table_args__ = {"schema": config.ABWASSER_SCHEMA}

        ModelInterlisSia405Abwasser.reach_progression_alternative = reach_progression_alternative

Plus adding it to the other files of import / export configuration.

Corrected datamodel for reach_progression_alternative see pull request #110

3nids pushed a commit that referenced this issue Jan 31, 2024
@domi4484
Missing classes (#111)
3ee2088 
Import/export missing classes
Fixes #98

Note: commit of the sql session is very slow. Expect some minutes waiting after closing the import selection dialog
@domi4484
Copy link
Contributor

@sjib I just uploaded a new version to the discussion. reach_progression_alternative will come in a separate pr when #110 is merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working INTERLIS About INTERLIS exchange format (import / export) plugin Concerns the wastewater plugin
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Missing classes teksi/wastewater
3 participants
@3nids @sjib @domi4484