3DCityDB Import Fails: A fatal error occurred during import.

[WaffleAycen]

I am currently trying to import Lod2 Data from the German state of Brandenburg into 3DCityDB.

For a lot of Files the import gets aborted with the message:

The specific file is available here: https://data.geobasis-bb.de/geobasis/daten/3d_gebaeude/lod2_gml/
Filename: lod2_3314-5865_geb.gml

I get this error for a lot of files regarding the state of Brandenburg. Any idea why that is and how to fix it?

[clausnagel]

This is a data issue. Also see #301 for the same discussion.

The data must be fixed before importing it.

[WaffleAycen]

didnt see that it was already discussed. Thanks for the quick answer. :)

[msochan]

Hey guys, lately I also encountered similar problem. It's happening for me during import of LoD2 for Hessen. I'm curious how actually "--no-fail-fast" flag is working? Because even with the flag passed to the config file, the error stops the import.

Is that intended behaviour?

[clausnagel]

Fatal errors from the backend will let the import fail even when using the --no-fail-fast flag. Yes, that's intended behaviour.

If you really face the same situation in Hessen like with Brandenburg, then you try and import invalid geometries which PostGIS simply refuses to store. We could adapt the importer to skip such invalid geometries and not send them to the backend, which would mean losing information. I'd prefer that the data gets fixed before importing.

[msochan]

Thank you for your quick answer @clausnagel! 🙏 I agree to some extent that the data should be corrected before importing, but sometimes it can be a time-consuming process to fix the data or contact the relevant authorities, so with the functionality of skipping faulty objects that can make things easier. Even if this involves losing some information. I don't have much knowledge of the Java language and haven't paid much attention to the source code, but do you think this functionality might be difficult to implement?

[Krassnig]

I would also appreciate such a feature, the person importing the data is usually not the one able to fix it 🙏
The same issue occurs with Luxembourg.

z.B. --import-without-validation

However, Hessen worked perfectly for me @msochan 👍

[msochan]

@Krassnig That's interesting! Did you get data from here?

https://gds.hessen.de/INTERSHOP/web/WFS/HLBG-Geodaten-Site/de_DE/-/EUR/ViewDownloadcenter-Start?path=3D-Daten/3D-Geb%C3%A4udemodelle/3D-Geb%C3%A4udemodelle%20LoD2/Landkreis%20Kassel

[clausnagel]

@Krassnig, @msochan, ok, we will implement a test to optionally skip invalid curves during import. I can reproduce the error with datasets from Brandenburg and Hessen (I tested with "Ahnatal-LoD2" from the link provided by @msochan). @Krassnig, can you share a sample from Luxembourg so that we can make sure that the fix also works for that dataset?

Will let you know here when a fix is available for testing.

[clausnagel]

I have added a too-few-points check when importing curves with c85e3df. I haven't changed the default behaviour of the importer. So, by default, an invalid curve will cause the import to fail. But you can now let the importer ignore and skip invalid curves. When using the GUI, go to "Preferences -> Import -> General" and uncheck the option "Cancel import immediately in case of errors". For the CLI, use the --no-fail-fast flag.

To test the change, clone the master branch and build the Importer/Exporter from source following this guide. Alternatively, you can use the egde Docker image from here.

[msochan]

@clausnagel Thank you so much for quick resolvement! I just ran it for Hessen sample, which previously was causing the problem, and now it is working like a charm! 😍

[Krassnig]

@msochan Yes, I got it from gds-hessen. I redownloaded Ahnatal from gds-hessen and it seems that they recently updated their datasets. My version is much older (around 2021 ~ 2022). I get the same errors for the new dataset.

@clausnagel Here is a sample from Luxembourg Bettembourg that fails direct link. entire dataset. The error message from the CLI:

[14:18:23 ERROR] Caused by: org.citydb.core.operation.importer.CityGMLImportException: Failed to import bldg:Building '00910_00690-000008'.
[14:18:23 ERROR] Caused by: org.citydb.core.operation.importer.CityGMLImportException: gml:POLYGON 'ID_668df754-ceeb-4a52-b9ae-75c13db50c09': Ring has too few points.

After building the master I could import the entirety of Luxembourg. I'm also gonna try Brandenburg tomorrow.
The only issue I encountered is that I couldn't build the project with Java 21 and had to downgrade to version 17 🤷‍♂️

A big thank you from me as well 👍

[clausnagel]

The only issue I encountered is that I couldn't build the project with Java 21 and had to downgrade to version 17 🤷‍♂️

Most likely because Gradle was still at version 7.4. I updated Gradle to 8.10. Should work now with Java 21.

[Krassnig]

Very nice 👌
Brandenburg also worked.

[clausnagel]

Just released version 5.5.0 which includes the fix. Closing.