Special attribute type to skip forbidden chars

[fgalan]

Context Broker doesn't allow forbidden charts (https://fiware-orion.readthedocs.io/en/master/user/forbidden_characters/index.html) in attribute values. This can be restrictive in some scenarios.

We could define an special attributeType (for instance, "TextUnrestricted") so ContextBroker broker will not do checks on it. Eg:

"description": {
   "type": "TextUnrestricted",
   "value": "I'm a description (and I'm using forbidden chars)"
}

Of course, the user should know the potential security implications of this and assume them in his client code.

This would be backward compatible and enhance Orion.

[fgalan]

Implemented by PR #3592

[pcanoesp]

Hi, fgalan.
We also have this problem. We are trying to use "TextUnrestricted" attribute type, but it doesn't work. Could you confirm us if it is still valid this solution or not? We are trying to use this attribute type to store a base64-coded payload.

Thanks in advance.

[fgalan]

@pcanoesp could you provide the request you are sending to the CB, pls? verb + url + payload

[pcanoesp]

We are using MQTT connection. Deployment is as follows: Network Server <=> Mosquitto MQTT broker <=> IoT Agent JSON <=> Orion Context Broker (MongoDB has been also deployed).

As a simple example, if we send a simple JSON as follows in a MQTT message:

{
  "prueba":"PRUEBA_DEVICE_TEXTO_QQQ=="
 }

The context broker ignores the message and entity fields are not updated ("prueba").

Here you have the entity and device definitions in FIWARE platform:
ENTITY:

{
    "id": "TouchTag:70b3d54b1c006813",
    "type": "TouchTag",           
    "PRUEBA":{
        "type": "TextUnrestricted",
        "value": "xxx"
    }     
}

DEVICE:

{
    "devices": [
        {
            "device_id": "70b3d54b1c006813",
            "entity_name": "TouchTag:70b3d54b1c006813",
            "entity_type": "TouchTag",
            "protocol": "IoTA-JSON",
            "transport": "MQTT",
            "timezone": "Europe/Madrid",
            "attributes": [
                {
                    "object_id": "prueba",
                    "name": "PRUEBA",
                    "type": "TextUnrestricted"                                      
                }                              
            ],
            "static_attributes": [
                {
                    "name": "refStore",
                    "type": "Relationship",
                    "value": "urn:ngsi-ld:Store:001"
                }
            ]
        }
    ]
}

Please, tell me if there is some missing information.
Thank you, @fgalan , for your quick response.

[pcanoesp]

In our real case of use, payload filed is the one including forbiden characters (base64 coding)

[fgalan]

Let's try to see the problem in terms of Orion Context Broker (the repo in which this issue has been opened ;), leaving apart the other components in your e2e case. In this sense:

• Which exact request is sending IOTA to Orion? Hint: IOTA logs and CB logs (in INFO level) could provide info about this
• Which problem do you get? I mean, response error (which one), log trace, etc.

[pcanoesp]

Hi again, @fgalan.
This is the error in the IoT Agent logs:

time=2021-10-19T15:45:32.122Z | lvl=DEBUG | corr=f5468e31-5b79-423e-a4a8-72210c9e14d2 | trans=f5468e31-5b79-423e-a4a8-72210c9e14d2 | op=IoTAgentNGSI.Entities-v2 | from=n/a | srv=everynet | subsrv=/atc | msg=Unknown error executing update operation | comp=IoTAgent
time=2021-10-19T15:45:32.122Z | lvl=ERROR | corr=f5468e31-5b79-423e-a4a8-72210c9e14d2 | trans=f5468e31-5b79-423e-a4a8-72210c9e14d2 | op=IoTAgentJSON.commonBinding | from=n/a | srv=n/a | subsrv=n/a | msg=MEASURES-002: Couldn't send the updated values to the Context Broker due to an error: {"name":"ENTITY_GENERIC_ERROR","message":"Error accesing entity data for device: TouchTag:70b3d54b1c006813 of type: TouchTag","details":{"error":"BadRequest","description":"Invalid characters in attribute value"},"code":400} | comp=IoTAgent
time=2021-10-19T15:45:32.123Z | lvl=DEBUG | corr=f5468e31-5b79-423e-a4a8-72210c9e14d2 | trans=f5468e31-5b79-423e-a4a8-72210c9e14d2 | op=IoTAgentNGSI.DomainControl | from=n/a | srv=everynet | subsrv=/atc | msg=response-time: 743831953 | comp=IoTAgent

So it seems that IoT Agent is filtering the message. The JSON message that I sent to IoT Agent is the following:

{
    "url": "http://localhost:1026/v2/entities/TouchTag:70b3d54b1c006813/attrs?type=TouchTag",
    "method": "PATCH",
    "headers": {
        "fiware-service": "everynet",
        "fiware-servicepath": "/atc"
    },
    "json": {
        "PRUEBA": {
            "type": "TextUnrestricted",
            "value": "PRUEBA_DEVICE_TEXTO_TTT==",
            "metadata": {
                "TimeInstant": {
                    "type": "DateTime",
                    "value": "2021-10-19T15:45:32.118Z"
                }
            }
        },
        "refStore": {
            "type": "Relationship",
            "value": "urn:ngsi-ld:Store:001",
            "metadata": {
                "TimeInstant": {
                    "type": "DateTime",
                    "value": "2021-10-19T15:45:32.118Z"
                }
            }
        },
        "TimeInstant": {
            "type": "DateTime",
            "value": "2021-10-19T15:45:32.118Z"
        }
    }
}

[fgalan]

Which CB version are you using?

[pcanoesp]

2.2.0 on a CentOS 7. We installed this version because we are not using docker. We are going to install a newer version and check again. I will keep you informed about this.

[fgalan]

TextUnrestricted was introduced in Orion 2.4.0.

2.2.0 is pretty old (February 2019). I strongly recommend you to upgrade to latest Orion release (3.2.0)

[pcanoesp]

We are upgrading to the latest version. I will keep you informed with the results. Thank you very much for your support.

[pcanoesp]

@fgalan , after upgrading to version 3.2.0, "TextUnrestricted" type is working properly for our case of use.
Thank you very much for your support.