Planning

[kzangeli]

Orion-LD v1.6

• error counting entities #1557
• MQTT subscriptions with virtual host. #1576
• Give errors for Registration requests if "-forwarding" is not ON
• Turn off tmp traces if logLevel is ERROR/WARN
• VocabProperty:
  • reduce one-item-arrays into a string
  • retrieve entity with a VocabProp in "simplified" format returned the vocab as "normalized"
  • crash - see info in GChat with Jason
  • VocabularyProperty not working as expected #1542
• JsonProperty - just make it work
• Independency from Orion:
  • Reimplement the src/lib/rest library (and move functions to orionld/mhd). Started in "spider" branch)
    • restReply,
    • connectionTreat,
    • requestCompleted
  • Replace logMsg with ktrace (entire broker)
  • Reimplement sub-cache (and make it part of OrionldTenant)
    • only update subs with modifIed > current modifiedAt in cache
    • One cache per tenant - don't mix tenants!!!
  • Replace parseArgs with kargs in Orion-LD?
• Finish "task/entityMapsInEntityMap" in orionld-1
• Warning in DocumentDB:
  bson_append_array(): invalid array detected. first element of array parameter is not "0".
• MQTT timeout - MQTT subscriptions disconnected from mqtt broker #1427
• New connection socket descriptor (1024) is not less than FD_SETSIZE  #1441
• Investigate BOOST library dependence. Perhaps I can remove old deprecated (NGSIv1) and get rid of BOOST?
• Prefix compaction with default (@vocab)  #1449

Backlog

• VocabularyProperty/expandValues for POST Query
• VocabularyProperty/expandValues for Subscriptions
• JsonProperty
• ListRelationship
• ListProperty
• datasetId
• multiple entity types
• Websockets (just subs/notifs)
• OPC UA support
• PATCH of pernot subscription
• pick/omit to replace "attrs" + answer How to obtain id without additional attrs #232
• Double slashes in URLs #1402
• Finish TRoE for PATCH /entities/{entityId} (need a change in postgres subAttributes table - add opcode)
• GET /types?options=details and pagination - it's complex ...
• NULL RHS and removal
• Look at Is POST of entityOperations/upsert deleting the entity temporarily ? #1296
  • Load tests to test threadsafeness of mongoc library
• Use multicast for propagation of changes in subscriptions in an HA setup (and regs as well) ?
• Use DDS for propagation of changes in subscriptions in an HA setup (and regs as well), if supported
• Rename -experimental to -mongoc
• Entity snapshots (new in NGSI-LD v1.9?)
• NGSI-LD Distributed Operations for BATCH ops
• Support for Orion-LD without subscription cache (CLI -noCache might have stopped working #889)
• New subscriptions - more granular (notify on attribute deletion, update, creation ...)
• Notification problems after a while #1390
• FIWARE-Correlator #1358
• Attribute Groups?
• Logging in JSON format - see json-logging-best-practices
• Pagination - APIv2 way vs NGSI-LD way
• FIWARE-Correlator - do we use/need it?
• Custom notifications - Customized payload not supported #710
• Support for Registration subscriptions?
• throttling - keep latest and notify when throttling time times out (this one really for API v1.4.1)
• Entity Cache (would be great for "smaller" brokers)
• memory management for containers: Overload new/malloc and sleep if too much memory is allocated
• Forwarding for DELETE /subscriptions/{subscriptionId}
• Forwarding for POST /subscriptions
• Forwarding for PATCH /subscriptions/{subscriptionId}
• Forwarding for PUT /entities/{entityId} (1 week)
• TEST Geo-Queries with new GET /entities
• Metrics #1199
• Put some order in the payload check functions
• Error Handling for "@context" anywhere
• Test Subscriptions and "q" for:
  • q="name[en]==%22hello%22"
• Georel Equals for GET /entities + Notifications - postponed
• PUT /manage/metrics?state=off
  • Come up with something good with Stefan and Jason, for all types of management (traces, caches, ...)
  • Is it OK to have a PUT without payload body?
    • Yes, as long as Content-Length: 0 is present:
    • https://stackoverflow.com/questions/1233372/is-an-http-put-request-required-to-include-a-body
• Q: Implement mix and/or on same level (without need of parenthesis)
• Use variables from fieldPaths.h/cpp for all error handling - avoid stupid copies
  • Somehow give correct fieldPaths to common check functions (e.g. 'entIdVector' - used by both subs and regs)
• Ability to express Number as "1.23E4" - that's for kjson
• Websocket notifications
• Attribute Groups
• Reject attribute names for terms that have an @type in the context ?
• Prepare ETSI Presentation on Attribute Groups (kz and Francisco - Mr Roboto)
• Documentation for Release 1.0.0
• Make application/json the default Content-Type and Accept
• Subscriptions w/o id - issue Subscriptions w/o id #936 - not reproducible ... need input from Chris
• Websockets, full API support
• Registrations and their @context:
  • Store "as is" in mongo
  • Store cached context URL in mongo
  • on use, if cached context URL doesn't exist, parse and cache, with same name
  • See also issue Context for forwarding requests to be stored in the registration in mongo #860
• Subscriptions and their @context - just like registrations
• OPC UA support, starting with entity updates, perhaps PATCH on a single attribute?
• Add the version of the NGSI-LD spec to the output of GET /ngsi-ld/v1/info and also a field for "exceptions" -
• Move call to temporal functions to requestCompleted
• Secure MQTT
• Notification TRIGGERS
• Invalid notification on PATCH Entity, updating attributes #559+Possible problem with PATCH Attribute and missing notifications #556 Complete rewrite of PATCH Entity including native NGSI-LD notifications:
• Steal this idea from Orion repo: FIX CMakeLists.txt compiler flags bases in gcc veresion instead of distro version telefonicaid/fiware-orion#3627
• MHD returns urlencoded attribute names - can MHD be configured not to encode responses?
  - See MHD_OPTION_UNESCAPE_CALLBACK
• Disable mongoc counters?
  • WARNING: mongoc: Falling back to malloc for counters
  • mongoc-stat uses those counters - might be interesting for troubleshooting
  • Would be good to enable/disable in run-time (and CLI param)
• 30x Permanently moved + Location header for Mintaka
  • CLI -mintaka <ip:port>
  • in orionldGetTemporal*(): if (mintaka != NULL) 30x+Location else 501 Not Implemented
• In notifications, attributes with datasetId are sent as "complete arrays" - this needs to be implemented
• 504 for "forwarded message gave a timeout while waiting for response"
• geo+json notifications for subscriptions where notification::attributes doesn't contain the geometry-property
• Notifications for changes in datasetId (POST /entities/*/attrs)
• Do something about -disableCustomNotifs and HTTP headers ...
• Test performance for 'socket service' - implement Entity Creation, Update, Query, and Retrieval ?
• Issue GET /attributes, with details=true, and registrations #810
• options=expanded => return response in expanded form - for all requests?
• throttling - keep latest and notify when throttling time times out (this one really for API v1.4.1)
• Don't EVER expand "observedAt" - but be prepared for it to be expanded in DB (backward compatibility)
• Bug: createdAt of attribute is modified when updating using POST /entities/{EID}/attrs - Issue createdAt for attribute updated when updating using POST /entities/{EID}/attrs #496
• KZ: PR delete attribute using mongo C++ legacy driver #460 - attempt to do it with single access to DB - pending answer from SOF
• Check incoming payload for any null value of any field and return error if found, or, do whatever the spec states where null is allowed (to remove an attribute for example).
• New HTTP header: Orionld-Performance?
  • Returns a Orionld-Performance HTTP header with the number of seconds and nanoseconds that the command took
  • Then compare NGSI-LD "Retrieve Entity" with equivalent function for NGSIv2
• Notifications using OPC-UA ?
• Attribute Domains
• POST /ngsi-ld/v1/cSourceSubscriptions
• PATCH /ngsi-ld/v1/cSourceSubscriptions/{csub-id}
• GET /ngsi-ld/v1/cSourceSubscriptions
• GET /ngsi-ld/v1/cSourceSubscriptions/{csub-id}
• DELETE /ngsi-ld/v1/cSourceSubscriptions/{csub-id}
• One semaphore per database collection, once mongoBackend is gone
• Q-filter with "database variables" in the Right-Hand-Side of the expressions
• Advanced Notifications - let attributes of one entity trigger notifications for another entity (entities)
  • Seems like "Custom Notifications" is the way to go here
• Entity Cache (would be great for "small" brokers)
• Registration-Subscription Cache
• Detailed Implementation doc (the source code completely explained)
• Notifications for changes in datasetId (POST /entities/*/attrs)
• NGSI-LD Q filter for subscriptions
• X-Forwarded-For HTTP header to avoid recursion
• Memory leak when restarting broker populates the subscription cache
• Move call to temporal functions to requestCompleted
• Do something about -disableCustomNotifs and HTTP headers ...
• Test performance for 'socket service' - implement Entity Creation, Update, Query, and Retrieval ?
• Secure MQTT?

QA

• Make ALL ETSI test suite cases work
• Performance tests with -mongoc
• Load tests to test threadsafeness of mongoc library
• A few leaks, in pernot subs and one more place
• Full GE (awaiting input from Jason)