BUG: Performance degradation for Application Screen with LoRaWAN devices

[bkd231]

We are experiencing a performance degradation issue that progressively worsens over time, which affects the loading of application data/screen related to LoRaWAN IoT devices. Initially (just after the os2iot backend was started), upon accessing the application screen, data related to LoRaWAN IoT devices is displayed immediately. However, after 1-2 hours of operation, the loading process becomes significantly slower, eventually resulting in timeouts and failure to display any data. Performing a service restart temporarily resolves the issue,. It is important to note that screens or applications without LoRaWAN devices load without any issues. The performance does return to normal after a service restart but degrades again after a short period.

The request from the frontend that times out calls the endpoint: /api/v1/application/{applicationId}/iot-devices?limit=5

The OS2IoT Backend processes the request and eventually needs to retrieve LoRaWAN devices' information from Chirpstack. To do this, the OS2IoT backend makes calls to Chirpstack for each device individually. This occurs within the findDevicesForApplication function in the ApplicationService, where there is a loop invoking the enrichLoRaWANDevice function of the ChirpstackDeviceService. Within this function, Chirpstack is queried twice for each device (devices/${deviceId}/keys and devices/${id}). We monitored the ingress to track requests from OS2IoT to Chirpstack for these endpoints. Our observations revealed that Chirpstack consistently handles the requests quickly. However, the interval between each request made by OS2IoT, while processing a single batch of sensor data, increases over time. Initially, the delay is only milliseconds after OS2IoT starts, but it can extend to several seconds (sometimes even 10-20 seconds) as the system continues to operate.

[ramogens-OS2]

Hi @bkd231
Thanks for reporting this frustrating bug. We have a new version of OS2iot coming up, where the calls to Chirpstack have been rewritten. Hopefully this solves the issue. At least we'll look into this being a first try for a solution.

[ramogens-OS2]

Hi @bkd231
Did you get to installing v1.6.0 and if so, have you seen any changes to this bug?

[ramogens-OS2]

Hi @bkd231
We hear from some of your customers on vers. 1.7, that they don't experience the performance degrading anymore. So I guess this issue seems to be solved, right?

[bkd231]

Hi @ramogens
Yes, after the update (we have updated to 1.7.0), we are no longer experiencing performance degradation issues. Thanks for your help in addressing this!

[ramogens-OS2]

Good to hear @bkd231 !
I'm closing this issue then.