DDL is incomplete

[jyutzler]

I found that GeoPackages in the wild were failing /opt/features/vector_features/data/feature_table_integer_primary_key because the primary key did not have a NOT NULL of 1 as per http://www.geopackage.org/spec/#example_feature_table_cols. (opengeospatial/ets-gpkg10#31) Upon further review it is because they are using our DDL from http://www.geopackage.org/spec/#_sample_feature_table_informative
The DDL needs to be updated to have a NOT NULL in it.

I suspect this problem exists in multiple tables.

[jyutzler]

One way of addressing this problem is by creating repair scripts as described here. Basically you make a new table with the right schema, copy the data from the original table to the new table, drop the old table, and rename the new one.

[jratike80]

By reading http://www.sqlite.org/lang_createtable.html#primkeyconst in SQLite the column is automatically NOT NULL if it is primary key AND integer.

According to the SQL standard, PRIMARY KEY should always imply NOT NULL. Unfortunately, due to a bug in some early versions, this is not the case in SQLite. Unless the column is an INTEGER PRIMARY KEY or the table is a WITHOUT ROWID table or the column is declared NOT NULL, SQLite allows NULL values in a PRIMARY KEY column.

I wonder if this change is necessary of if it would be better to edit the test. Execute "PRAGMA table_info(table)", and if id column has type=INTEGER and pk=1 then it is OK.

[rouault]

You're right @jratike80 . I read too fast/wrongly the SQLite documentation. What is surprising, and I'd say a sqlite bug, is that if you do "PRAGMA table_info(table_name)", then a column of type INTEGER PRIMARY KEY returns notnull = 0, whereas a column of type INTEGER PRIMARY KEY NOT NULL returns notnull = 1. In both cases, I'd expect notnull = 1 to be returned given SQLite doc.

[jyutzler]

The test will definitely be weakened for GPKG 1.0/1.1. It isn't pragmatic to make the test fail on nearly every single GPKG in existence.

The SWG will discuss what to do about 1.2 on Monday.

[jratike80]

More important than to check if INTEGER PRIMARY KEY AUTOINCREMENT column is NOT NULL (according to documentation it is always NOT NULL), would be to check if column is really AUTOINCREMENT. If it is not the ids of deleted rows may be re-used and that can lead to troubles if tables have foreing key relations.

Unfortunately it does not seem to be straight forward to check if column is AUTOINCREMENT. The "SELECT COUNT(*) FROM sqlite_sequence WHERE name='yourtable';" test that is explained in http://stackoverflow.com/questions/20979239/how-to-tell-if-a-sqlite-column-is-autoincrement works but I confirmed with SQLite version 3.9.2 that the sequence really appears only after inserting data into the table. If table is empty the only place to find this information is propably in the "sql" field of sqlite_master table:

SELECT sql FROM sqlite_master WHERE type = 'table' AND name = 'example';

Empty tables which have never contained any data are marginal cases but I suppose that such can exist in GeoPackage.

[jyutzler]

Yes, I think we have to be prepared for the possibility that someone has generated a GeoPackage template that will be empty until it is populated in the field.

[jyutzler]

In the SWG today we determined that we will add a separate test to ensure that there are no NULLs in the data. We will also add a test to check to see if the IDs are in fact unique.

[jyutzler]

Test added in jyutzler/ets-gpkg12@be9b089

[jyutzler]

The PR #283 was approved by the SWG today. Summary:

• the test is nerfed for GPKG 1.0/1.1
• the DDL was updated for GPKG 1.2
• there is an additional test to confirm that the IDs are unique
• Appropriate information will be added to the release notes