Huge amount of rentained messages and restarting

[skinkie]

The last action upon a restart is saving retained messages to /var/lib/flashmq/retained.db, would it be possible to do this in a different way, since it takes quite long to write it. In addition to it, it looks that flashmq needs a whopping 6GB of RAM, and my expectation is that this is mostly due to these retained messages, any chance to offload this?

[halfgaar]

For both issues, it would probably require integration with some kind of storage specific software, like something SQL based (Sqlite maybe). And then in such a way so that it's not a build-time requirement. Also, some kind of change tracking would have to be implemented.

I'll think about some solutions, perhaps intermediate ones as well.

[ben-auo]

+1

Persistence like https://nanomq.io/docs/en/latest/rule/config-file.html#data-persistence-with-sqlite with the option to republish a topic would be awesome

[bigsmoke]

Just a note: there are a couple of new contenders for in-process databases that might be worth checking out before defaulting to SQLite:

• Column-oriented: https://duckdb.org/
• Key-value: https://rocksdb.org/
• Key-value: https://github.com/google/leveldb
• http://www.lmdb.tech/doc/ is a bit older, but might also be quite sufficient and more performant than SQLite, with the downside of the datafiles being platform-dependent (which may not be a big deal with FlashMQ's singular Linux x64 focus).

This is a preliminary list, without having, for instance, done any license compatibility analysis.

[skinkie]

I am kind of a heavy DuckDB user ;-) The problem with DuckDB is that it is in principle single threaded access. That is not a problem if the inter process communication is handled within the application and the database access can be handled with just one thread. That having said, DuckDB has transparant compression support, which is pretty neat.

[ben-auo]

worth checking out before defaulting to SQLite

Oh yes, definitely. That was just a quick example that came to mind. Some other ideas:

• https://github.com/rqlite/rqlite
• https://github.com/Expensify/Bedrock
• https://github.com/apple/foundationdb/
• https://github.com/bloomberg/comdb2

[bigsmoke]

I've spoke with @halfgaar about this a couple of times recently, and persistence options for retained messages are something that he very much wants to maul over for some time before committing to any particular approach. This is something to get right off the bat, especially because one of the architectural goals for FlashMQ is to keep the number of dependencies to an absolute minimum. (Recently, we even got rid of an external DocBook→groff tool dependency for generating the man pages.)