Unit tests are failing

[dstenger]

When project is built via mvn clean install following error occurs:

Results :

Failed tests: 
  VerifyTestNGController.cleanTestRun:56->runTests:91 Unexpected number of fail verdicts for file file:/home/stenger/git/ets-gpkg12-nsg/target/test-classes/gpkg/gdal_sample_v1.2_spi_nonlinear_webp_elevation.gpkg.
See file:/home/stenger/testng/d2ce7e0d-450e-40f0-bd85-0ff0d1d23aa1/testng-results.xml for details. expected:<0> but was:<5>

Tests run: 5, Failures: 1, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  6.560 s
[INFO] Finished at: 2023-01-31T22:44:37+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.19.1:test (default-test) on project ets-gpkg12-nsg: There are test failures.

When test with file gdal_sample_v1.2_spi_nonlinear_webp_elevation.gpkg is commented out, other files fail as well.

Seems to be related to #26.

[ghobona]

The CITE engineering team will look into this issue in April 2023.

[dstenger]

Following updates must be done to pass all tests again:

-        runTests(ClassLoader.getSystemResource("gpkg/gdal_sample_v1.2_spi_nonlinear_webp_elevation.gpkg"), 0);
-        runTests(ClassLoader.getSystemResource("gpkg/gdal_sample_v1.2_spatial_index_extension.gpkg"), 0);
-        runTests(ClassLoader.getSystemResource("gpkg/gdal_sample_v1.2_no_extensions.gpkg"), 0);
-        runTests(ClassLoader.getSystemResource("gpkg/gdal_sample_v1.2_no_extensions_with_gpkg_ogr_contents.gpkg"), 0);
-        runTests(ClassLoader.getSystemResource("gpkg/empty.gpkg"), 0);
-        runTests(ClassLoader.getSystemResource("gpkg/states10.gpkg"), 0);
-        runTests(ClassLoader.getSystemResource("gpkg/bluemarble.gpkg"), 0);
+        runTests(ClassLoader.getSystemResource("gpkg/gdal_sample_v1.2_spi_nonlinear_webp_elevation.gpkg"), 5);
+        runTests(ClassLoader.getSystemResource("gpkg/gdal_sample_v1.2_spatial_index_extension.gpkg"), 4);
+        runTests(ClassLoader.getSystemResource("gpkg/gdal_sample_v1.2_no_extensions.gpkg"), 4);
+        runTests(ClassLoader.getSystemResource("gpkg/gdal_sample_v1.2_no_extensions_with_gpkg_ogr_contents.gpkg"), 4);
+        runTests(ClassLoader.getSystemResource("gpkg/empty.gpkg"), 2);
+        runTests(ClassLoader.getSystemResource("gpkg/states10.gpkg"), 2);
+        runTests(ClassLoader.getSystemResource("gpkg/bluemarble.gpkg"), 4);
-        runTests(ClassLoader.getSystemResource("gpkg/elevation.gpkg"), 0); // These two are the id notnull thing
-        runTests(ClassLoader.getSystemResource("gpkg/coastline-polyline-hydro-115mil-and-smaller.gpkg"), 0);
-        runTests(ClassLoader.getSystemResource("gpkg/v12_bad_attributes.gpkg"), 1); // R119
-        runTests(ClassLoader.getSystemResource("gpkg/sample1_0.gpkg"), 1); // R77
-        runTests(ClassLoader.getSystemResource("gpkg/simple_sewer_features.gpkg"), 1); // This is an invalid 1.0 or 1.1 GPKG - it has an invalid metadata table (md_standard_URI instead of md_standard_uri) 
-        runTests(ClassLoader.getSystemResource("gpkg/sample1_1.gpkg"), 1); // R77
-        runTests(ClassLoader.getSystemResource("gpkg/sample1_2.gpkg"), 1); // R77
-        runTests(ClassLoader.getSystemResource("gpkg/sample1_2F10.gpkg"), 1); // Default "undefined", R77
-        runTests(ClassLoader.getSystemResource("gpkg/geonames_belgium.gpkg"), 3); // lower case data types, R77
-        runTests(ClassLoader.getSystemResource("gpkg/haiti-vectors-split.gpkg"), 2); // lower case data types, R77
-        runTests(ClassLoader.getSystemResource("gpkg/bentiu_southsudan-osm-20170213.gpkg"), 2); //R5, R29
+        runTests(ClassLoader.getSystemResource("gpkg/elevation.gpkg"), 5); // These two are the id notnull thing
+        runTests(ClassLoader.getSystemResource("gpkg/coastline-polyline-hydro-115mil-and-smaller.gpkg"), 3);
+        runTests(ClassLoader.getSystemResource("gpkg/v12_bad_attributes.gpkg"), 4); // R119
+        runTests(ClassLoader.getSystemResource("gpkg/sample1_0.gpkg"), 5); // R77
+        runTests(ClassLoader.getSystemResource("gpkg/simple_sewer_features.gpkg"), 5); // This is an invalid 1.0 or 1.1 GPKG - it has an invalid metadata table (md_standard_URI instead of md_standard_uri)
+        runTests(ClassLoader.getSystemResource("gpkg/sample1_1.gpkg"), 5); // R77
+        runTests(ClassLoader.getSystemResource("gpkg/sample1_2.gpkg"), 5); // R77
+        runTests(ClassLoader.getSystemResource("gpkg/sample1_2F10.gpkg"), 5); // Default "undefined", R77
+        runTests(ClassLoader.getSystemResource("gpkg/geonames_belgium.gpkg"), 6); // lower case data types, R77
+        runTests(ClassLoader.getSystemResource("gpkg/haiti-vectors-split.gpkg"), 5); // lower case data types, R77
+        runTests(ClassLoader.getSystemResource("gpkg/bentiu_southsudan-osm-20170213.gpkg"), 5); //R5, R29

All test files located here should be checked if the updated behavior regarding reported errors is correct.
E.g.: gdal_sample_v1.2_spi_nonlinear_webp_elevation.gpkg had 0 failures with v0.6 and 5 failures with current master branch.

[dstenger]

@cmorriscmps Could you please update VerifyTestNGController.java so that all unit tests are passing again? For details, please see my previous comment.

[cmorriscmps]

The better way to fix the tests would be to update the geopackage resource files. However, I'm not equipped to do that. My suggestion is to just comment out the tests that are failing, recognizing they still need work.

[jyutzler]

The better way to fix the tests would be to update the geopackage resource files. However, I'm not equipped to do that. My suggestion is to just comment out the tests that are failing, recognizing they still need work.

No, these files exist because they test specific situations that one might encounter in the field. It is important that the test cases in the test suite are as representative as possible. Do not change them and do not comment out the runTests calls that exercise them. Do consider creating new files based on existing ones to create exemplars for how to correctly exercise a particular part of the standard.

We need to understand why we are getting new failures and whether the failures are in alignment with the standard, including the abstract test suite. If these files are experiencing test failures that were missed by previous versions of the test suite, then that is a good thing. If the number of test failures (the second argument to runTests) changes, then it would be helpful to add a comment indicating what changed.

[cmorriscmps]

I have no objection if the community would prefer updating the tests as per dstenger's comment.

I haven't checked, but my guess is that the changes are due to WKT definitions not matching. The WKT files were changed to match the spec exactly in this version of the tests. The WKT comparison code is still too strict, as it says in the code:
// This is still an extremely incomplete test and the CRSs may still be the same. In part,
// this is due to different variations of WKT content.

[ghobona]

@cmorriscmps Ok, thanks for the response. By "changed to match the spec exactly", I presume you mean the NSG GeoPackage spec.

Since the changes to the unit tests would only be applied to the NSG GeoPackage executable test suite, we will proceed as per @dstenger's comment above.

[dstenger]

A pull request was created fixing the unit tests as described in #58 (comment): #59

[ghobona]

I have just merged Pull Request #59 .