Store device-sessions in database to simplify backups and disaster-recovery #74
Comments
|
Yes, true. Losing device-sessions is a real disaster. |
|
Losing device-sessions is something that we need to avoid so storing it in the PostgreSQL database could prevent devices stop communicating for a period of time. So if this could be fixed / changed that would be a great addition. |
|
@brocaar what is your opinion on this? Is there any architectural reason why device sessions are in redis now? |
|
Redis is going to be a lot faster than Postgres for repeated lookups with a predetermined key. I imagine Mr. Brocaar will speak to the full rationale for this design, but personally, I don't think backing up Redis is that big a lift. You should be doing it already if you're in production. |
Is this an assumption or did you actually test this? In my experience with production-size PostgreSQL databases and Redis databases the difference on primary-key-based reads is actually negligible between Redis and Postgres. Writing is another story due to the integrity guarantees of Postgres, however I guess the PostgreSQL performance would still be sufficient. |
|
Just to let you know, I am doing some testing currently to understand the performance impact of this: This might be the best solution to solve #362 as all device-sessions matching a DevAddr can be retrieved in a single query rather than retrieving the DevAddr -> DevEUI set, and then performing one query per key (as we can't use a Redis pipeline as each key might be on a different node in case of Redis Cluster). |
This migrates the device-sessions from Redis into PostgreSQL. This fixes a performance issue in case the same DevAddr is reused many times (e.g. devices rejoining very often or a NetID with small DevAddr space). There were two issues: The Redis key containing the DevAddr -> DevEUIs mapping could contain DevEUIs that no longer used the DevAddr. This mapping would only expire from the Redis database after none of the devices would use the DevAddr for more than the configured device_session_ttl. The other issue with the previous approach was that on for example a Type 7 NetID, a single DevAddr could be re-used multiple times. As each device-session could be stored on a different Redis Cluster instance, there was no option to retrieve all device-sessions at once. Thus a high re-usage of a single DevAddr would cause an increase in Redis queries. Both issues are solved by moving the device-session into PostgreSQL as the DevAddr is a column of the device record and thus filtering on this DevAddr would always result in the devices using that DevAddr, as well all device-sessions for a DevAddr can be retrieved by a single query. Note that to migrate the device-sessions, you must run: chirpstack -c path/to/config migrate-device-sessions-to-postgres A nice side-effect is that a PostgreSQL backup / restore will also restore the device connectivity. Closes #362 and #74.
|
This has been implemented in the above commit. |
|
Hi @brocaar. Thank you for fixing this. Will test and report back. Just to confirm, there's no need to back-up Redis anymore after this change, right? |
|
@timcooijmans there is still some data in Redis, like pending mac-commands which could case some issues if not restored (especially when there are RX1Delay, RX1DROffset, RX2DR or RX2Freq changes). Maybe these should be merged into the device-session at some point as well to have everything at one place. However, in the case where all devices are in sync restoring a PostgreSQL backup should recover device connectivity. |
Currently device-sessions are only stored in redis. When the device-sessions are lost, devices have to re-join, something that is not automated on all devices. The result is that the redis 'database' should be considered persistent-data and should backed-up in addition to the PostgreSQL database.
It would be easier if the device-sessions would be backed up in the database so only a database-restore is needed as disaster-recovery.
The text was updated successfully, but these errors were encountered: