Data Platform Access Module

• Data Platform Access Module
  • Overview
  • Module Structure
  • Usage
    • Basic Usage Logic
    • Data platform access module methods
      • Input paramters
      • Functionalities
    • Code-Based Usage Example
    • Module Test Execution

Overview

This repository contains code for the data platform access module (dpam). The module can be used as a python library to simplify interaction of custom python software with a data platform. The module supports to do CRUD (Create - Update - Read - Delete) operations to access platform data via NGSI-LD API to request data and optionally token-based access control via OIDC & OAuth2.0 authentication and authorization idenity. The dpam module is developed and tested against a FIWARE broker using Kong API Gatway and Keycloak identity provider.

Module Structure

The code is composed out of three types of classes:

1. controller: The controller folder contains classes that control the logic of the module. For the moment this is the platformController.py to create, read, update or delete data at data platform.
2. model: The model folder contains all data classes that model input parameters or return values in form of structured python objects. The message object is used as structured return value of the tool (see message.py). Furthermore, there are python objects that contain a set of input parameters to request a token tokenRequest.py or create a query at data platform platformQuery.py. Finally the operationResult object (see operationResult.py) is used in a project of multi-agent system representation (Link published soon!) and reflects asset administration shell model snippets from submodel-operation model (https://github.com/smart-data-models/dataModel.AAS/tree/master/I4SubmodelElementOperation).
3. utils: The utils folder contains a bunch of helper classes that handle certains tasks such as handle errors in unified ways (see errorHandler.py), get an authentication token (see tokenHandler.py), or send http requests to data platform (see requestsHandler.py).

Furthermore, there is a test.py class that executes some tests with the module.

The image below shows a class diagram of the code structure to display the structure graphically.

Usage

To use the module you need to understand how the module works in general (see Basic Usage Logic) and how you can use it in your own python project (see Code-Based Usage Example). Furthermore, it might be useful to test the module from command line to test the connection with data platform in general or test the functionality of the module for its further development (see Module Test Execution).

Basic Usage Logic

This section describes how to use the module in general from a high-level perspective.

The code in this repository should be used as a library to make CRUD operations at data platform. This means that the code should be imported into the own python project to use the implemented module functionality. The most important class is the platform controller (see PlatformController.py) that has to be initiated first and can then be used to access the data platform.

To access data platform the user of the module might need to provide a token for secured data platform access and can then make requests to the data platform. Both tasks, requesting a token and making platform requests, can be done in simplified way via data platform access module (see Functionalities).

Data platform access module methods

To use the data platform access module you will need to understand which functionalities it provides and how the functionalities can be used.

Input paramters

Before the module methods are described you need to understand the python classes tokenRequest.py and platformQuery.py. Both classes are used model a set of input parameters that are needed for the data platform access module functionality:

• tokenRequest.py: Class that represents all variables that are needed to request a token if your data platform access is secured. It has following instance variables:
  1. enabled (mandatory): Enable or disable token-based access. To query and provide a token via dpam module configure true otherwise false.
  2. url (mandatory): URL of the identity provider to request a token
  3. username (mandatory): Username of the user who requests a token
  4. password (mandatory): Password of the user who requests a token
  5. client_id (mandatory): Client id of the identity provider client or application that should be accessed
  6. client_secret (mandatory): Client secret of the identity provider client or application that should be accessed
• platformQuery.py: Class that represents all variables that are needed to paramterize queries to data platform for all types of CRUD operations. This class has following instance variables:
  1. entity_id: Entity id of the entity that should be read or updated eg. 'urn:ngsi-ld:WeatherObserved:Dummy-01'.
  2. body: JSON body with the entity to be created or updated. The JSON has to contain a context tag to determine the context the entity belongs to.
  3. options: Options in terms of 'keyValues' or 'normalized' formatting of entity for read operations. If no options are provided 'keyValues' formatting is being used by default.
  4. attrs (optional): List of attributes that should be read eg ["temperature", "windSpeed"]. If no attributes are provided, the whole entity is read.
  5. query_parameters (optional): Additional query parameters that are added to the url
  6. fiware_service (mandatory): FIWARE tenant name where the entity should be created, updated, deleted or read from
  7. fiware_path (mandatory): FIWARE tenant service path where the entity should be created, updated, deleted or read from
  8. context (mandatory): Link of the FIWARE context that should be used to read the data

Functionalities

The data platform access module provides two types of functionalities:

1. Request a token: To access the data platform you first need to request a token as described in section Basic Usage Logic. The token has to be requested from identity provider and can then be used for a limited time to access the data platform. The method platformController.get_token(tokenRequest) allows to request a token from data platform whenever needed. The user of the module has the responsiblity to check the validity of the token for data platform access and request a new token after the old one expired or has become invalid. To call the method platformController.get_token(tokenRequest) the user has to create a dpam.model.tokenRequest object as input parameter as specified above.
2. Access data platform: Furthermore, the user can make queries to execute CRUD operations at data platform. If token-based access is enabled, all methods expect a valid token and a platformQuery object as input parameter to define which type of query should be made:
   1. read_single_entity(token, platformQueryParameter): Can be used to read data from a single entity.
   2. read_entities_by_type(token, platformQueryParameter): Can be used to read possibly multiple entities using filter mechnisms, for example, filter by type or creation date
   3. create_single_entity(token, platformQueryParameter): Can be used to create a new entity.
   4. create_new_attributes(token, platformQueryParameter): Can be used to add new attributes to an existing entity
   5. update_single_entity(token, platformQueryParameter): Can be used to update some or all attributes of an existing entity.
   6. delete_single_entity(token, platformQueryParameter): Can be used to delete an existing entity.

The table below provides an overview which parameter of the platformQuery.py object should be provided for which data platform access method. Thereby 'x' means mandatory, '(x)' means optional and '-' means not neccessary for this type of operation.

Parameter	create_single_entity	create_new_attributes	read_single_entity	update_single_entity	delete_single_entity	read_entities_by_type
url	x	x	x	x	x	x
fiware_service	x	x	x	x	x	x
fiware_service_path	x	x	x	x	x	x
entity_id	-	x	x	x	x	-
body	x	x	-	x	-	-
options	-	-	(x)	-	-	(x)
attributes	-	-	(x)	-	-	(x)
context	-	-	x	-	x	x

Remark: In the current implementation the body of create / update operations always needs a @context field instead of using a http header. So please make sure to include the field in your provided payload.

...
  "@context": [
    "http://ld-context/ld-context/datamodels.context-ngsild.jsonld"
  ]
...

Code-Based Usage Example

This section provides a concrete example on how to use the module from your individual python project.

First, you will need to install the module as a dependency in your python environment. As an example you can include the module inside your requirements.txt file and install it together with other dependencies using pip install -r requirements.txt:

...
dpam @ git+https://github.com/DLR-SF/DPAM-Module.git
...

Second, you can import the platformController and models for the input and output objects of the dpam module in python files or classes that should use the module to access the data platform:

...
from dpam.controller.platformController import platformController
from dpam.model.platformQuery import platformQuery
from dpam.model.tokenRequest import tokenRequest
...

Third, before making requests to data platform you will need to request an authentication token. Therefore, create a tokenRequest object and use the platformController to request the token.:

# 1) Instantiate variables
# -> platform controller
pc = platformController()
# -> authentication variables
url_idp = "http://host.org/identity-provider"
username = "user"
password = "secure-password"
client_id = "client"
client_secret = "1234"

# 2) create tokenRequest object
tr = tokenRequest(url_idp, username, password, client_id, client_secret)

# 3) use platform controller to request token
token_result = pc.get_token(tr)
if token_result.messageType == "info":
   print("Successfully requested token: {}".format(token_result.text))
else:
   print("Token request has not been successful. Reason: {}".format(token_result.text))

Fourth, you can execute data platform CRUD operations (create - read - update - delete) by using the platformController and the requested token:

# 1) Instantiate platform controller from dpam module with variables that do not change
# -> platform controller
pc = platformController()
# -> query parameters
url_broker = "http://host.org/orion-ld"
fiware_service = "rest_weather"
fiware_service_path = "/demo"
context = "http://ld-context/datamodels.context-ngsild.jsonld"
pq = platformQuery(url_broker, fiware_service, fiware_service_path, context)

# 1) Example Write Operation
# -> Create JSON body of WeatherObserved entity (see model specification here: https://github.com/smart-data-models/dataModel.Weather/blob/master/WeatherObserved/README.md)
pq.body = '{"id":"urn:ngsi-ld:WeatherObserved:Dummy-01","type":"WeatherObserved","atmosphericPressure":{"type":"Property","value":938.9},precipitation":{"type":"Property","value":0},"relativeHumidity":{"type":"Property","value":1},"temperature":{"type":"Property","value":10},"windDirection":{"type":"Property","value":135},"windSpeed":{"type":"Property","value":2},"@context":["http://host.org/ld-context/datamodels.context-ngsild.jsonld"]}'

# -> use platform controller to create entity at data platform
result_c1 = pc.create_single_entity(token, pq)


# 2) Example Read Operation
# -> use platform controller to read complete created WeatherObserved entity in normalized data format
pq.entity_id = "urn:ngsi-ld:WeatherObserved:Dummy-01"
pq.options = "normalized"
pq.attributes = None
result_r1 = pc.read_single_entity(token, pq)

# -> use platform controller to read attributes temperature and windSpeed of created WeatherObserved entity in short keyValues format
pq.options = "keyValues"
pq.attributes = ["temperature", "windSpeed"]
result_r2 = pc.read_single_entity(token, pq)


# 3) Example Update Operation
# -> create JSON body to update precipitation, snowHeight, temperature and windSpeed of WeatherObserved entity
pq.body = '{"dateObserved":{"type":"Property","value":"2024-11-25T12:22:00.000Z"},"precipitation":{"type":"Property","value":7},"snowHeight":{"type":"Property","value":52},"temperature":{"type":"Property","value":21},"windSpeed":{"type":"Property","value":9},"@context":["http://host.org/ld-context/datamodels.context-ngsild.jsonld"]}'

# use platform controller to update entity at data platform
result_u1 = pc.update_single_entity(token, pq)

An example on how the methods for CRUD operations can be called can be found in test.py.

Fith, the return value of the dpam module is a message object if the operation has been successful or a normal error occured. The message object contains of:

• messageType: Type of the message. This can be 'info', 'error', 'warn', or 'exception'.
• text: Text of the message. In case of an error this is the error message. In case of successful operation execution this is the result message or entity in case data is queried from the platform.
• code: Code of the message. In case of error this is 1. In case of successful operation execution this is the http status code, for example, 200 (successfully read entity) or 404 (entity not found)

You can access the field of the message return object for example like that:

# Example read entity
result = pc.read_single_entity("urn:ngsi-ld:WeatherObserved:Dummy-01", "keyValues", ["temperature", "windSpeed"], "rest_weather", "/demo", "http://ld-context/ld-context/datamodels.context-ngsild.jsonld")

if result.messageType == "info":
   weather_entity = result.text
else:
   raise Exception(result.text)

In case of a success message the entity is read from the result object. In case of an error message object an exception is raised with the text of the error.

A full example on how the module can be used in a python project can be found here: https://github.com/DLR-SF/Agent-Dummy. In this example a dummy-agent uses the dpam module to update agent runtime information at the data platform. Furthermore, it shows how to request a token and use it until it becomes invalid because of its age. In this example code the token information is read from a configuration file using the configuration handler, so that sensible authentication information is not committed into git repository together with the code.

Module Test Execution

Furthermore, the module can be used from command line to execute some tests.

Therefore, install the module into the python environment as a module as explained above (see Code-Based Usage Example) or build the module locally. To build the module locally:

1. Install the requirements from requirements.txt file at your python environment: pip install -r requirements.txt
2. Install your package: pip install -e .
3. Build your package: python -m build or python -m build --wheel

Afterwards copy test-config.example-system.yaml and name it 'test-config.test-system.yaml' or 'test-config.prod-system.yaml' to test the module at test or productive system. Adopt the configuration file according to your actual configuration. Afterwards adopt the beginning of test.py, so that your created configuration file is being used:

def test():
    print("\nStart Test CRUD operations via Module")

    # get variables
    # -> dpamtest
    config_path = "dpam" + os.sep + "test" + os.sep + "test-config.test-system.yaml"

Afterwards, you can call dpamtest from command line:

(.venv) PS C:\path\to\data_platform_access_module> dpamtest       

Start Test CRUD operations via Module

Create Tests:

- Result of Test Create 1 (Successfully create full single entity):
{'messageType': 'info', 'text': '', 'code': 200}

Read Tests:

...

End Test CRUD operations via Module.

To execute the tests successfully you will need to have a connection and correct configuration to access data platform. Otherwise the tests will fail because data platform is not reachable. As an example:

{'messageType': 'error', 'text': 'Error during patch request to create platform data. Reason: HTTPConnectionPool(host=\'host.org\', port=80): Max retries exceeded with url: /orion-ld/ngsi-ld/v1/entities (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x00000289C0719450>: Failed to resolve \'host.org\' ([Errno 11001] getaddrinfo failed)")).', 'code': 1}