Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The agent does not send data to the Orion Context Broker with a request payload of 566 KB #827

Closed
ludilorenz opened this issue May 6, 2024 · 9 comments

Comments

@ludilorenz
Copy link

IoT Agent JSON version the issue has been seen with

3.2.0

Bound or port used (API interaction)

Northbound (Provision API and NGSI Interactions)

NGSI version

NGSIv2, NGSI-LD

Are you running a container?

Yes, I am using a contaner (Docker, Kubernetes...)

Image type

normal

Expected behaviour you didn't see

Sending data from the agent to the Context Broker and subsequent creation/update of the entity

Unexpected behaviour you saw

Orion does not receive the request (there are no logs from Orion) and an error code 500 is returned with the following message:

{
"name": "PayloadTooLargeError",
"message": "request entity too large"
}

Steps to reproduce the problem

No response

Configs

environment:
  - IOTA_CB_HOST=orion
  - IOTA_CB_PORT=1026
  - IOTA_NORTH_PORT=4041
  - IOTA_REGISTRY_TYPE=mongodb
  - IOTA_TIMESTAMP=true
  - IOTA_CB_NGSI_VERSION=v2
  - IOTA_AUTOCAST=true
  - IOTA_MONGO_HOST=mongodb
  - IOTA_MONGO_PORT=27017
  - IOTA_MONGO_DB=iotagent-json-v2
  - IOTA_HTTP_PORT=7896
  - IOTA_PROVIDER_URL=http://iot-agent-v2:4041
  - IOTA_DEFAULT_RESOURCE=/iot/json

Log output

2024-05-06 11:56:09 time=2024-05-06T09:56:09.245Z | lvl=DEBUG | corr=3951c9c6-1404-4c14-b204-e3cc2942c8e5 | trans=3951c9c6-1404-4c14-b204-e3cc2942c8e5 | op=IOTAJSON.HTTP.Binding | from=n/a | srv=fpn | subsrv=/FP4/battery_metrics | msg=Error PayloadTooLargeError handling request: request entity too large | comp=IoTAgent
2024-05-06 11:56:09 time=2024-05-06T09:56:09.842Z | lvl=DEBUG | corr=09b1fb75-89fc-4418-9e02-a9d516ec0902 | trans=09b1fb75-89fc-4418-9e02-a9d516ec0902 | op=IoTAgentNGSI.GenericMiddlewares | from=n/a | srv=fpn | subsrv=/FP4/battery_metrics | msg=Request for path [/iot/about] query [{}] from [localhost:4041] | comp=IoTAgent
2024-05-06 11:56:09 time=2024-05-06T09:56:09.842Z | lvl=DEBUG | corr=09b1fb75-89fc-4418-9e02-a9d516ec0902 | trans=09b1fb75-89fc-4418-9e02-a9d516ec0902 | op=IoTAgentNGSI.DomainControl | from=n/a | srv=n/a | subsrv=n/a | msg=response-time: 1 | comp=IoTAgent
@fgalan
Copy link
Member

fgalan commented May 6, 2024

msg=Error PayloadTooLargeError handling request: request entity too large

could be due to ContextBroker, as seen here

https://github.com/telefonicaid/fiware-orion/blob/5fe943b970599949e8f769bf7750e0e01f312edc/src/lib/rest/HttpStatusCode.cpp#L46

However, ContextBroker limits requests to 1MB size (see https://github.com/telefonicaid/fiware-orion/blob/master/doc/manuals/user/known_limitations.md#request-maximum-size) and the size mentioned in the issue is smaller (566KB). Could you provide the 566KB request you are using in curl format, please (maybe with a link to https://pastebin.com)?

In addition, increasing the log level in Orion to INFO level may provide extra info (this link may be useful: https://github.com/telefonicaid/fiware-orion/blob/master/doc/manuals/admin/management_api.md#log-configs-and-trace-levels)

@fgalan fgalan removed the bug label May 6, 2024
@fgalan
Copy link
Member

fgalan commented May 6, 2024

I'm removing the "bug" label due to this is (by the moment) expected behaviour.

@fgalan
Copy link
Member

fgalan commented May 6, 2024

In addition, could you provide the docker-compose.yml you are using, please? It may be needed.

@ludilorenz
Copy link
Author

ludilorenz commented May 6, 2024

msg=Error PayloadTooLargeError handling request: request entity too large

could be due to ContextBroker, as seen here

https://github.com/telefonicaid/fiware-orion/blob/5fe943b970599949e8f769bf7750e0e01f312edc/src/lib/rest/HttpStatusCode.cpp#L46

However, ContextBroker limits requests to 1MB size (see https://github.com/telefonicaid/fiware-orion/blob/master/doc/manuals/user/known_limitations.md#request-maximum-size) and the size mentioned in the issue is smaller (566KB). Could you provide the 566KB request you are using in curl format, please (maybe with a link to https://pastebin.com)?

In addition, increasing the log level in Orion to INFO level may provide extra info (this link may be useful: https://github.com/telefonicaid/fiware-orion/blob/master/doc/manuals/admin/management_api.md#log-configs-and-trace-levels)

I had already tried to increase the maximum request size of Orion with the command inReqPayloadMaxSize within the docker compose, but I received the same error.

I tried setting the log level to INFO, but Orion does not show me any messages when I try to send data.

Regarding the data, unfortunately I cannot share it, I can say that it is a CSV file of 1147 rows and 16 columns converted to a JSON and then sent to the agent. The payload sent to the agent have 20633 rows, I don't know if this information is useful.

@ludilorenz
Copy link
Author

ludilorenz commented May 6, 2024

In addition, could you provide the docker-compose.yml you are using, please? It may be needed.

This is the docker-compose.yml I am using:

version: "3.1"

services:

  iot-agent-v2:
    image: fiware/iotagent-json:3.2.0
    hostname: iot-agent-v2
    container_name: fiware-iot-agent-v2
    depends_on:
      - mongodb
    expose:
      - "4041"
      - "7896"
    ports:
      - "4041:4041"
      - "7896:7896"
    environment:
      - IOTA_CB_HOST=orion
      - IOTA_CB_PORT=1026
      - IOTA_NORTH_PORT=4041
      - IOTA_REGISTRY_TYPE=mongodb
      - IOTA_TIMESTAMP=true
      - IOTA_CB_NGSI_VERSION=v2
      - IOTA_AUTOCAST=true
      - IOTA_MONGO_HOST=mongodb
      - IOTA_MONGO_PORT=27017
      - IOTA_MONGO_DB=iotagent-json-v2
      - IOTA_HTTP_PORT=7896
      - IOTA_PROVIDER_URL=http://iot-agent-v2:4041
      - IOTA_DEFAULT_RESOURCE=/iot/json
      
  mongodb:
    image: mongo:4.2
    hostname: mongodb
    container_name: mongo-db
    ports:
      - "27017:27017"
    command: --bind_ip_all 
    volumes:
      - mongo-db:/data/db
      - mongo-config:/data/configdb

  orion:
    image: fiware/orion:3.11.0
    hostname: 27017
    container_name: fiware-orion
    depends_on:
      - mongodb
    expose:
      - "1026"
    ports:
      - "1026:1026"
    command: -dbhost mongo-db -logLevel INFO

volumes:
  mongo-db: ~
  mongo-config: ~
  synchdata: ~

@fgalan
Copy link
Member

fgalan commented May 6, 2024

After investigation, it seems is related with express middlewre in IOTA (i.e. it's not related with Orion Context Broker). This PR would fix the issue telefonicaid/iotagent-node-lib#1608

@fgalan
Copy link
Member

fgalan commented May 6, 2024

Additional PRs: PR #828 (IOTA-JSON) and PR telefonicaid/iotagent-ul#675 (IOTA-UL)

@fgalan
Copy link
Member

fgalan commented May 6, 2024

@ludilorenz the issue should be now solved in the latest docker available at telefonicaiot/iotagent-json:latest. Could you test again with it and provide your feedback in a comment in this issue, please?

@ludilorenz
Copy link
Author

@ludilorenz the issue should be now solved in the latest docker available at telefonicaiot/iotagent-json:latest. Could you test again with it and provide your feedback in a comment in this issue, please?

I tried to send the data and everything went well, I was able to retrieve the data from Orion correctly. I will proceed to resolve the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants