Temporal

[kzangeli]

ToDo for the implementation of injection of temporal representation of entities

• Tenants:

  • List databases
    • select datname from pg_database
  • Retrieve tables:
    • Connect to the DB
    • SELECT * FROM pg_tables WHERE schemaname='public'
  • Make sure the temp DB survives a restart of the broker
  • FuncTest for all possible combinations on tenants already there and not, and make sure the db data is never lost

• Misc

  • Merge the feature branch to develop and continue the work in "develop"
  • GeoProperty - both as attribute and as sub-attribute
  • Make Mintaka run locally
  • Connection Pool
  • How to deal with SQL injection?
    • there is a function for this, part of the psql driver but it uses malloc ... :(
      We need to urlencode every string value of all attrs+subattrs, and also their names and probably more stuff.
      So, we'd potentially end up with thousands of calls to malloc for the URL-encoding ...
      We DON'T WANT THAT - we'll have to use something else - preferably a home made routine
  • Check the null value of attributes for deletion - some requests support this feature
  • Valgrind to make sure we have no leaks nor errors
  • It should be enough with a single timestamp in the DB tables (createdAt, modifiedAt, deletedAt)
    • the opMode field tells us which timestamp it is.

• Do we need unitCode, observedAt inside the 'attributes' table? I'd like to see them in the sub-attributes table - to avoid copies!

• Performance:

  • Avoid cloning the request tree - put back "id" and "type" instead (for batch requests)
  • Make service routines expand entity type and attribute names - on expansion from inside TRoE routines

Done

• Prepare the functest framework for temporal
  • New CLI -tmp for the functest framework - to run all functests for temporal (and nothing more)
  • FT help function postgresCmd to query postgres db tables to check that the operations do what they should
• Untested implementation for creation of DB tables on startup
• DB Change: naming of tables and fields:
  • Tables:
    • entities
    • attributes
    • sub-attributes
• pgDrop and pgCreate in functest framework - to start the functest from an empty state
• Merge the feature/temporal branch with develop
• Change of the DB table layout to be able to support temporal representation of entities
• Initial definition of the architecture (issue Temporal Design Spec #607):
  • Use only one value_string field + AttributeType field to interpret the string_value field
  • Change almost all the names of the table fields
  • Change the name of the db tables to: { entities, attributes, subAttributes }
  • Use the exact same database names (tenants with prefix) in postgres as we use in mongo
  • deletedAt (to mark an entity/attribute/sub-attribute to be deleted in a point in time)
  • Prepared for TRANSACTIONS
  • Prepared for CONNECTION POOL
  • Short functions, with descriptive names
  • New DB tables
  • Get to the same degree of "supported features" that we had in the PoC
  • it was simply faster and easier to rewrite the whole thing than to try to fix the PoC
• Initial implementation of the new architecture
• DB Change: default tenant must be named "orion", not "orion_ld" - aligned with orion (and this prefix configurable via CLI)
• Support for Attributes of type "Relationship"
• Support for Attributes of type "String Property"
• Support for Attributes of type "Integer Property"
• Support for Attributes of type "Float Property"
• CLI options for postgres db params (IP, port, user, password) - must test IP and port !
• Meeting with Andres + preparation + report
• travis (github actions) - with much appreciated help from @wistefan , to support postgres for the functests for TRoE (Temporal Representation of Entities)
• Merge all the work (while waiting for github actions) to feature/temporal
• Transactions - group all the SQL commands over the three DB tables into one single transaction (so that all fail if one of them fails)
  ----------------- Dec 10 - reported in CTO meeting minutes -------------------------
• Allow for postgres to run in another host
• Make temporal functests work in github actions - temporal/postgres functests not working in GH-Actions #624 (with help from Stefan)
• Support for Attributes of type "Bool Property"
• Support for Attributes of type "Array Property"
• Support for Attributes of type "Object Property"
• Support special attribute unitCode only for Number Properties
• pgAttributePush() - new function taking care of the pushing to the "attributes" table
• Style guide, add the check of the temporal directory as part of the build - is run by travis on PR, and fix all style errors
  • First remove the remaining files from the PoC
• Support for Sub-attributes in functional test framework
• Support for all types of Sub-attributes except Geo
• BATCH Create Entity (POST /ngsi-ld/v1/entityOperations/create)
• CREATE Entity (POST /ngsi-ld/v1/entities)
• DELETE Entity
• DELETE Attribute
• DELETE Batch
• Modification of the database tables - entityRef nor createdAt cannot be supplied when updating an attribute
• Added a table item for attributes and entities: opMode
  • opMode is necessary especially to inform about REPLACE
• POST Entity
• POST Entity ?noOverwrite - existing attrs are ignored
• Make a list of all operations that trigger a change in the TRoE and decide exactly what to do for each operation
• PATCH Attribute
• PATCH Entity
• Tenants: new postgres databases added in run-time
• Batch Upsert/Replace (including multiple instances of one and the same entity)
• Batch Upsert ?options=update - attrs are replaced - mark as REPLACED in TRoE DB
• Batch Update
• Batch Update ?options=noOverwrite (already existing attrs are ignored) - SR gives this info
• Multiple instances of one and the same entity implemented for all four batch operations
• datasetId for the two requests that support it:
  • POST /entities
  • POST /entities/{EID}/attrs
• Support for Geo Properties
  • Point
  • Polygon
  • LineString

[kzangeli]

This issue is quite out of date.
TRoE is pretty much implemented and this issue is of no real use anymore - closing it