Skip to content

Add GeoJSON and DateTime, unitCode and observedAt NGSI-LD support #843

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 28 commits into from
Mar 4, 2020
Merged

Add GeoJSON and DateTime, unitCode and observedAt NGSI-LD support #843

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
a732017
Add `observedAt` support
jason-fox Feb 10, 2020
ff76dfe
Handle Date, DateTime and Time as mandated by the NGSI-LD specification
jason-fox Feb 10, 2020
3e02f3e
Handle GeoJSON parsing as mandated by the NGSI-LD specification
jason-fox Feb 10, 2020
d8a64e3
Correct type attribute
jason-fox Feb 10, 2020
637451d
Calculate temporal properties using the UTC timezone
jason-fox Feb 10, 2020
73a350c
Use single quote
jason-fox Feb 10, 2020
f525477
Calculate temporal properties using the UTC timezone for observedAt A…
jason-fox Feb 10, 2020
866d901
Add unitCode support.
jason-fox Feb 10, 2020
aab10c3
Standardize timestamps to UTC.
jason-fox Feb 10, 2020
caf1896
Merge branch 'feature/ngsi-ld-measure' into feature/time-and-geo
jason-fox Feb 11, 2020
ae20a71
Merge branch 'feature/ngsi-ld-measure' into feature/time-and-geo
jason-fox Feb 11, 2020
00878fa
Remove observedAt from Entity root.
jason-fox Feb 20, 2020
41342e6
Changes after review. (feature/ngsi-ld-measure)
jason-fox Feb 26, 2020
44eb834
Merge branch 'feature/ngsi-ld-measure' into feature/time-and-geo
jason-fox Feb 26, 2020
7d78950
More JavaDoc NGSI-v2 => NSGI-LD
jason-fox Feb 27, 2020
ec47097
Merge branch 'feature/ngsi-ld-measure' into feature/time-and-geo
jason-fox Feb 28, 2020
fd5c5d9
Merge branch 'feature/ngsi-ld-measure' into feature/time-and-geo
jason-fox Feb 28, 2020
c30d072
Merge commit '3cc306477f414c4600cf0f7f807c9574dd046cbb' into feature/…
jason-fox Mar 2, 2020
1a5aaae
More JavaDoc
jason-fox Mar 2, 2020
1da050b
Updated CNR
jason-fox Mar 2, 2020
fa90072
Adding basic documentation.
jason-fox Mar 2, 2020
60b5292
Update doc/architecture.md
jason-fox Mar 3, 2020
5aa3a0b
Update doc/architecture.md
jason-fox Mar 3, 2020
03bb0fe
Update doc/installationguide.md
jason-fox Mar 3, 2020
6cd5300
Update doc/api.md
jason-fox Mar 3, 2020
16455aa
Update doc/architecture.md
jason-fox Mar 3, 2020
7991863
Update doc/advanced-topics.md
jason-fox Mar 3, 2020
3d72455
Remove dead code - replace with comment
jason-fox Mar 3, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGES_NEXT_RELEASE
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,8 @@ Add NGSIv2 metadata support to device provisioned attributes
Fix: Error message when sending measures with unknown/undefined attribute
Add Null check within executeWithSecurity() to avoid crash (#829)
Basic NGSI-LD active measures support (#841)
Add GeoJSON and DateTime, unitCode and observedAt NGSI-LD support (#843)
- The NGSI v2 `TimeInstant` element has been mapped onto the NGSI-LD `observedAt` property
- The NGSI v2 `metadata.unitCode` attribute has been mapped onto the NGSI-LD `unitCode` property
Add NGSIv2 metadata support to attributeAlias plugin.
Add mongodb authentication: IOTA_MONGO_USER, IOTA_MONGO_PASSWORD and IOTA_MONGO_AUTH_SOURCE (#844)
51 changes: 48 additions & 3 deletions doc/advanced-topics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,50 @@ e.g.:
}
```

#### NGSI-LD data and metadata considerations

When provisioning devices for an NGSI-LD Context Broker, `type` values should typically correspond to one of the
following:

- `Property`, `Relationship`, `Geoproperty`
- Native JSON types (e.g. `String`, `Boolean`, `Float` , `Integer` `Number`)
- Temporal Properties (e.g. `Datetime`, `Date` , `Time`)
- GeoJSON types (e.g `Point`, `LineString`, `Polygon`, `MultiPoint`, `MultiLineString`, `MultiPolygon`)

Most NGSI-LD attributes are sent to the Context Broker as _properties_. If a GeoJSON type or native JSON type is
defined, the data will be converted to the appropriate type. Temporal properties should always be expressed in UTC,
using ISO 8601. This ISO 8601 conversion is applied automatically for the `observedAt` _property-of-a-property_ metadata
where present.

Data for any attribute defined as a _relationship_ must be a valid URN.

Note that when the `unitCode` metadata attribute is supplied in the provisioning data under NGSI-LD, the standard
`unitCode` _property-of-a-property_ `String` attribute is created.

Other unrecognised `type` attributes will be passed as NGSI-LD data using the following JSON-LD format:

```json
"<property_name>": {
"type" : "Property",
"value": {
"@type": "<property_type>",
"@value": { string or object}
}
}
```

`null` values will be passed in the following format:

```json
"<property_name>": {
"type" : "Property",
"value": {
"@type": "Intangible",
"@value": null
}
}
```

### Data mapping plugins

The IoT Agent Library provides a plugin mechanism in order to facilitate reusing code that makes small transformations
Expand Down Expand Up @@ -138,7 +182,7 @@ The library provides some plugins out of the box, in the `dataPlugins` collectio
use the `addQueryMiddleware` and `addUpdateMiddleware` functions with the selected plugin, as in the example:

```javascript
var iotaLib = require('iotagent-node-lib');
var iotaLib = require("iotagent-node-lib");

iotaLib.addUpdateMiddleware(iotaLib.dataPlugins.compressTimestamp.update);
iotaLib.addQueryMiddleware(iotaLib.dataPlugins.compressTimestamp.query);
Expand Down Expand Up @@ -166,8 +210,9 @@ events in the IoT Agent with the configured type name will be marked as events.

##### Timestamp Processing Plugin (timestampProcess)

This plugin processes the entity attributes looking for a TimeInstant attribute. If one is found, the plugin add a
TimeInstant attribute as metadata for every other attribute in the same request.
This plugin processes the entity attributes looking for a `TimeInstant` attribute. If one is found, for NGSI-v1/NGSIv2,
the plugin adds a `TimeInstant` attribute as metadata for every other attribute in the same request. With NGSI-LD, the
Standard `observedAt` property-of-a-property is used instead.

##### Expression Translation plugin (expressionTransformation)

Expand Down
36 changes: 18 additions & 18 deletions doc/api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,25 +203,25 @@ Note that there is a 1:1 correspondence between payload fields and DB fields (bu
The table below shows the information held in the Device resource. The table also contains the correspondence between
the API resource fields and the same fields in the database model.

| Payload Field | DB Field | Definition | Example of value |
| ------------------------- | -------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------- |
| `device_id` | `id` | Device ID that will be used to identify the device. | UO834IO |
| `service` | `service` | Name of the service the device belongs to (will be used in the fiware-service header). | smartGondor |
| `service_path` | `subservice` | Name of the subservice the device belongs to (used in the fiware-servicepath header). | /gardens |
| `entity_name` | `name` | Name of the entity representing the device in the Context Broker | ParkLamplight12 |
| `entity_type` | `type` | Type of the entity in the Context Broker | Lamplights |
| `timezone` | `timezone` | Time zone of the sensor if it has any | America/Santiago |
| `timestamp` | `timestamp` | Optional flag about whether or not to addthe TimeInstant attribute to the device entity created, as well as a TimeInstant metadata to each attribute, with the current timestamp | true |
| `apikey` | `apikey` | Optional Apikey key string to use instead of group apikey |
| Payload Field | DB Field | Definition | Example of value |
| ------------------------- | -------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------- |
| `device_id` | `id` | Device ID that will be used to identify the device. | UO834IO |
| `service` | `service` | Name of the service the device belongs to (will be used in the fiware-service header). | smartGondor |
| `service_path` | `subservice` | Name of the subservice the device belongs to (used in the fiware-servicepath header). | /gardens |
| `entity_name` | `name` | Name of the entity representing the device in the Context Broker | ParkLamplight12 |
| `entity_type` | `type` | Type of the entity in the Context Broker | Lamplights |
| `timezone` | `timezone` | Time zone of the sensor if it has any | America/Santiago |
| `timestamp` | `timestamp` | Optional flag about whether or not to add the `TimeInstant` attribute to the device entity created, as well as a `TimeInstant` metadata to each attribute, with the current timestamp. With NGSI-LD, the Standard `observedAt` property-of-a-property is created instead. | true |
| `apikey` | `apikey` | Optional Apikey key string to use instead of group apikey |
| 9n4hb1vpwbjozzmw9f0flf9c2 |
| `endpoint` | `endpoint` | Endpoint where the device is going to receive commands, if any. | http://theDeviceUrl:1234/commands |
| `protocol` | `protocol` | Name of the device protocol, for its use with an IoT Manager. | IoTA-UL |
| `transport` | `transport` | Name of the device transport protocol, for the IoT Agents with multiple transport protocols. | MQTT |
| `attributes` | `active` | List of active attributes of the device | `[ { "name": "attr_name", "type": "Text" } ]` |
| `lazy` | `lazy` | List of lazy attributes of the device | `[ { "name": "attr_name", "type": "Text" } ]` |
| `commands` | `commands` | List of commands of the device | `[ { "name": "attr_name", "type": "Text" } ]` |
| `internal_attributes` | `internalAttributes` | List of internal attributes with free format for specific IoT Agent configuration | LWM2M mappings from object URIs to attributes |
| `static_attributes` | `staticAttributes` | List of static attributes to append to the entity. All the updateContext requests to the CB will have this set of attributes appended. | `[ { "name": "attr_name", "type": "Text" } ]` |
| `endpoint` | `endpoint` | Endpoint where the device is going to receive commands, if any. | http://theDeviceUrl:1234/commands |
| `protocol` | `protocol` | Name of the device protocol, for its use with an IoT Manager. | IoTA-UL |
| `transport` | `transport` | Name of the device transport protocol, for the IoT Agents with multiple transport protocols. | MQTT |
| `attributes` | `active` | List of active attributes of the device | `[ { "name": "attr_name", "type": "Text" } ]` |
| `lazy` | `lazy` | List of lazy attributes of the device | `[ { "name": "attr_name", "type": "Text" } ]` |
| `commands` | `commands` | List of commands of the device | `[ { "name": "attr_name", "type": "Text" } ]` |
| `internal_attributes` | `internalAttributes` | List of internal attributes with free format for specific IoT Agent configuration | LWM2M mappings from object URIs to attributes |
| `static_attributes` | `staticAttributes` | List of static attributes to append to the entity. All the updateContext requests to the CB will have this set of attributes appended. | `[ { "name": "attr_name", "type": "Text" } ]` |

#### Attribute lists

Expand Down
15 changes: 8 additions & 7 deletions doc/architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ basis preprovisioning the devices). Device measures can have three different beh
The following sequence diagram shows the different NGSI interactions an IoT Agent makes with the Context Broker,
explained in the following subsections (using the example of a OMA Lightweight M2M device).

![General ](./img/ngsiInteractions.png 'NGSI Interactions')
![General ](./img/ngsiInteractions.png "NGSI Interactions")

Be aware that the IoT Agents are only required to support NGSI10 operations `updateContext` and `queryContext` in their
standard formats (currently in JSON format; XML deprecated) but will not answer to NGSI9 operations (or NGSI convenience
Expand Down Expand Up @@ -207,21 +207,22 @@ the concrete IoT Agent implementations will be to map between the native device
The following figure offers a graphical example of how a COAP IoT Agent work, ordered from the registration of the
device to a command update to the device.

![General ](./img/iotAgentLib.png 'Architecture Overview')
![General ](./img/iotAgentLib.png "Architecture Overview")

### The `TimeInstant` element

As part of the device to entity mapping process the IoT Agent creates and updates automatically a special timestamp.
This timestamp is represented as two different properties of the mapped entity::

- An attribute metadata named `TimeInstant` per dynamic attribute mapped, which captures as an ISO8601 timestamp when
the associated measurement (represented as attribute value) was observed.
- With NGSIv1/NGSI-v2, an attribute metadata named `TimeInstant` (per dynamic attribute mapped, which captures as an
ISO8601 timestamp when the associated measurement (represented as attribute value) was observed. With NGSI-LD, the
Standard `observedAt` property-of-a-property is used instead.

- An entity attribute named `TimeInstant` which captures as an ISO8601 timestamp when the last measurement received
from the device was observed.
- For NGSIv1/NGSI-v2 only, an additional attribute `TimeInstant` is added to the entity which captures as an
ISO8601 timestamp when the last measurement received from the device was observed.

If no information about the measurement timestamp is received by the IoT Agent, the arrival time of the measurement will
be used to generate a `TimeInstant` for both the entity and the attribute's metadata.
be used to generate a `TimeInstant` for both the entity attribute and the attribute metadata.

Take into account that:

Expand Down
11 changes: 6 additions & 5 deletions doc/installationguide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,9 +149,9 @@ used for the same purpose. For instance:

- **mongodb**: configures the MongoDB driver for those repositories with 'mongodb' type. If the `host` parameter is a
list of comma-separated IPs, they will be considered to be part of a Replica Set. In that case, the optional
property `replicaSet` should contain the Replica Set name. If the database requires authentication, username
(`username`), password (`password`) and authSource (`authSource`) can be set. For The MongoBD driver will retry the
connection at startup time `retries` times, waiting `retryTime` seconds between attempts, if those attributes are
property `replicaSet` should contain the Replica Set name. If the database requires authentication, username
(`username`), password (`password`) and authSource (`authSource`) can be set. For The MongoBD driver will retry the
connection at startup time `retries` times, waiting `retryTime` seconds between attempts, if those attributes are
present (default values are 5 and 5 respectively). E.g.:

```javascript
Expand Down Expand Up @@ -206,9 +206,10 @@ used for the same purpose. For instance:
any unexpected error.
- **singleConfigurationMode**: enables the Single Configuration mode for backwards compatibility (see description in
the Overview). Default to false.
- **timestamp**: if this flag is activated, the IoT Agent will add a 'TimeInstant' metadata attribute to all the
attributes updated from device information. This flag is overwritten by `timestamp` flag in group or device
- **timestamp**: if this flag is activated:
- For NGSIv1/NGSIv2, the IoT Agent will add a `TimeInstant` metadata attribute to all the attributes updated from device information. This flag is overwritten by `timestamp` flag in group or device
provision.
- With NGSI-LD, the standard `observedAt` property-of-a-property is created instead.
- **defaultResource**: default string to use as resource for the registration of new Configurations (if no resource is
provided).
- **defaultKey**: default string to use as API Key for devices that do not belong to a particular Configuration.
Expand Down
Loading