[HELP] Example of working configuration for the REQUEST connector?

[kastayou]

Describe the issue
I try to use the request connector without success.

My gateway is connected to my thingsboard instance (demo.thingsboard.io).
The gateway is shown as active in thingsboard but no messages are received

I have error messages in the gateway logs (connector.log and converter.log)

For my test, I am using a API mockup:
https://tb-gateway.free.beeceptor.com/my/api/path

Note: this endpoint my not be further available due to API rate limits.
In that case the endpoint is not available anymore, this api returns the following JSON

{"deviceName":"sensor 01", "deviceType":"temperature sensor", "temperature":20, "serial":"ABCD"}

I have reviewed the documentation but could not find a working configuration.
Do you have any example working configuration that works ?

Configuration (Attach your configuration file)
Notate: Remove Access token from file if you wanna attach tb_gateway.yaml
Here is my tb_gateway.yaml

thingsboard:
  host: demo.thingsboard.io
  port: 1883
  remoteConfiguration: false
  security:
    accessToken: MYACCESTOKEN
storage:
  type: memory
  read_records_count: 100
  max_records_count: 100000
#  type: file
#  data_folder_path: ./data/
#  max_file_count: 10
#  max_read_records_count: 10
#  max_records_per_file: 10000
connectors:

  -
    name: REQUEST Connector
    type: request
    configuration: request.json

Here is my request.json configuration

{
  "host": "https://tb-gateway.free.beeceptor.com",
  "SSLVerify": true,
  "security": {
    "type": "anonymous"
  },
  "mapping": [
    {
      "url": "my/api/path",
      "httpMethod": "GET",
      "httpHeaders": {
        "ACCEPT": "application/json"
      },
      "allowRedirects": true,
      "timeout": 0.5,
      "scanPeriod": 60,
      "converter": {
        "type": "json",
        "deviceNameJsonExpression": "${deviceName}",
        "deviceTypeJsonExpression": "${deviceType}",
        "attributes": [
          {
            "key": "serialNumber",
            "type": "string",
            "value": "${serial}"
          }
        ],
        "telemetry": [
          {
            "key": "temperature",
            "type": "int",
            "value": "${temperature}"
          }
        ]
      }
    }
  ]
}

Connector name (If you need help with some connector/converter):
connector is REQUEST connecter and the converter is JSON

Error traceback (If it was raised):

With the above configuration, when I start the gateway, I have the following errors

converter.log

""2020-06-24 23:34:13" - ERROR - [json_request_uplink_converter.py] - json_request_uplink_converter - 38 - The expression for looking "deviceType" not found in config {"url": "my/api/path", "httpMethod": "GET", "httpHeaders": {"ACCEPT": "application/json"}, "allowRedirects": true, "timeout": 0.5, "scanPeriod": 60, "converter": {"type": "json", "deviceNameJsonExpression": "${deviceName}", "deviceTypeJsonExpression": "${deviceType}", "attributes": [{"key": "serialNumber", "type": "string", "value": "${serial}"}], "telemetry": [{"key": "temperature", "type": "int", "value": "${temperature}"}]}}"

connector.log

""2020-06-24 23:34:13" - ERROR - [request_connector.py] - request_connector - 237 - 'NoneType' object has no attribute 'get'"
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/thingsboard_gateway/connectors/request/request_connector.py", line 231, in __process_data
    self.__gateway.send_to_storage(self.get_name(), data_to_send[device])
  File "/usr/lib/python3/dist-packages/thingsboard_gateway/gateway/tb_gateway_service.py", line 303, in send_to_storage
    if not TBUtility.validate_converted_data(data):
  File "/usr/lib/python3/dist-packages/thingsboard_gateway/tb_utility/tb_utility.py", line 51, in validate_converted_data
    if error is None and not data.get("deviceName"):
AttributeError: 'NoneType' object has no attribute 'get'

If I try to replace the following lines in request.conf

        "deviceNameJsonExpression": "Test device",
        "deviceTypeJsonExpression": "default",

With this new configuration, I still do not get message arriving in Thingsboard and I have the following error messages:

converter.log

""2020-06-25 00:03:56" - ERROR - [json_request_uplink_converter.py] - json_request_uplink_converter - 38 - The expression for looking "deviceType" not found in config {"url": "my/api/path", "httpMethod": "GET", "httpHeaders": {"ACCEPT": "application/json"}, "allowRedirects": true, "timeout": 0.5, "scanPeriod": 60, "converter": {"type": "json", "deviceNameJsonExpression": "Test device", "deviceTypeJsonExpression": "default", "attributes": [{"key": "serialNumber", "type": "string", "value": "${serial}"}], "telemetry": [{"key": "temperature", "type": "int", "value": "${temperature}"}]}}"

connector.log

""2020-06-25 00:03:56" - ERROR - [request_connector.py] - request_connector - 237 - 'NoneType' object has no attribute 'get'"
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/thingsboard_gateway/connectors/request/request_connector.py", line 231, in __process_data
    self.__gateway.send_to_storage(self.get_name(), data_to_send[device])
  File "/usr/lib/python3/dist-packages/thingsboard_gateway/gateway/tb_gateway_service.py", line 303, in send_to_storage
    if not TBUtility.validate_converted_data(data):
  File "/usr/lib/python3/dist-packages/thingsboard_gateway/tb_utility/tb_utility.py", line 51, in validate_converted_data
    if error is None and not data.get("deviceName"):
AttributeError: 'NoneType' object has no attribute 'get'

Versions (please complete the following information):

• OS: [e.g. Ubuntu 18.04]
  ThingsBoard IoT gateway version: 2.4.0" (installed from deb package)
  Python 3.8.2
  Distributor ID: Ubuntu
  Description: Ubuntu 19.10
  Release: 19.10
  Codename: eoan

[imbeacon]

Hi @kastayou

Thank you for your interest in the ThingsBoard IoT gateway. The configuration is correct, but by some reasons the response from the server doesn't contain any information. I will try to repeat this situation and write you.

[kastayou]

Hi, thank you for your feedback,

The server is a free mockup service with a rate limit, that's why it does not answer anymore.

I have found a different server but I still have the same issue
The server:
https://bitbay.net/API/Public/BTC/ticker.json

The answer from the server:
{"max":9610.01,"min":9200,"last":9250,"bid":9200,"ask":9250,"vwap":9250,"average":9250,"volume":1.52347255}
my request.json

{
  "host": "https://bitbay.net",
  "SSLVerify": true,
  "security": {
    "type": "anonymous"
  },
  "mapping": [
    {
      "url": "API/Public/BTC/ticker.json",
      "httpMethod": "GET",
      "httpHeaders": {
        "ACCEPT": "application/json"
      },
      "allowRedirects": true,
      "timeout": 0.5,
      "scanPeriod": 60,
      "converter": {
        "type": "json",
        "deviceName": "Test device",
        "deviceType": "default",
        "attributes": [
          {
            "key": "serialNumber",
            "type": "int",
            "value": "${max}"
          }
        ],
        "telemetry": [
          {
            "key": "temperature",
            "type": "int",
            "value": "${average}"
          }
        ]
      }
    }
  ]
}

converter.log

""2020-06-25 07:57:36" - ERROR - [json_request_uplink_converter.py] - json_request_uplink_converter - 38 - The expression for looking "deviceType" not found in config {"url": "API/Public/BTC/ticker.json", "httpMethod": "GET", "httpHeaders": {"ACCEPT": "application/json"}, "allowRedirects": true, "timeout": 0.5, "scanPeriod": 60, "converter": {"type": "json", "deviceName": "Test device", "deviceType": "default", "attributes": [{"key": "serialNumber", "type": "int", "value": "${max}"}], "telemetry": [{"key": "temperature", "type": "int", "value": "${average}"}]}}"

and connector.log

""2020-06-25 07:57:36" - ERROR - [request_connector.py] - request_connector - 237 - 'NoneType' object has no attribute 'get'"
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/thingsboard_gateway/connectors/request/request_connector.py", line 231, in __process_data
    self.__gateway.send_to_storage(self.get_name(), data_to_send[device])
  File "/usr/lib/python3/dist-packages/thingsboard_gateway/gateway/tb_gateway_service.py", line 303, in send_to_storage
    if not TBUtility.validate_converted_data(data):
  File "/usr/lib/python3/dist-packages/thingsboard_gateway/tb_utility/tb_utility.py", line 51, in validate_converted_data
    if error is None and not data.get("deviceName"):
AttributeError: 'NoneType' object has no attribute 'get'

[imbeacon]

It was a bug with request connector and with configuration:

To solve it - please install the version from the master branch (You can install it using this guide) and use the following configuration(Difference in deviceNameJsonExpression and deviceTypeJsonExpression names of the parameters):

{
  "host": "https://bitbay.net",
  "SSLVerify": true,
  "security": {
    "type": "anonymous"
  },
  "mapping": [
    {
      "url": "API/Public/BTC/ticker.json",
      "httpMethod": "GET",
      "httpHeaders": {
        "ACCEPT": "application/json"
      },
      "allowRedirects": true,
      "timeout": 0.5,
      "scanPeriod": 60,
      "converter": {
        "type": "json",
        "deviceNameJsonExpression": "Test device",
        "deviceTypeJsonExpression": "default",
        "attributes": [
          {
            "key": "serialNumber",
            "type": "int",
            "value": "${max}"
          }
        ],
        "telemetry": [
          {
            "key": "temperature",
            "type": "int",
            "value": "${average}"
          }
        ]
      }
    }
  ]
}

[kastayou]

Thank you for the quick feedback, I have installed the fix and it is now working. The device is created and I have the telemetry and attributes values.
Thank you for pointing the error in the last version of my request.conf file, I have tried so many different version of my configuration file.