Make TokyoCabinet available as a message store

[slact]

That would be awesome, distributed, and persistent!

[slact]

We get TokyoTyrant for free once Memcached storage is implemented (same protocol), but it would be nice to have a native interface as well.

[mmaunder]

Just MHO, but I think persistence should be handled by the server-side application. NHPM is a real-time event passing or message passing system. The client should load an initial data set from the server and then receive new items as they arrive in real-time. The only persistence that NHPM should implement should be to deal with slow clients. e.g. Guys internet connection dropped for 10 seconds because he's on wifi and then recovers and he needs to catch up. If you need persistence because you want an audit log or history, just CC each publish request via your back-end 1 Gbit ethernet switch to a local server running Tyrant or MySQL.

[slact]

Your argument is sound. Maybe it wouldn't be so awesome for persistence after all.
But people might still want disk space use over RAM for message storage.

[mmaunder]

I can think of two reasons to use disk:

1. A browser has fetched data and is expecting events to be passed via nginx. It is receiving events and the server goes down. Browser keeps retrying. Server comes back up and because events were stored in memory they were lost. So browser misses those events and only receives new ones. So for guaranteed delivery, disk may be a good option.
2. For applications with very large queues running on servers with not much memory on slow links, disk may be a good option if the queues won't fit in memory.

In both cases there's a performance penalty when using disk and as far as I can guess it'll be O(2n) because there's 1 disk stat() for published events being written and another when they're read back for clients. So I can't say I'm super excited about using a 7200rpm disk in a highly concurrent environment. But I'm sure there's a little grey man in a little grey suit out there that's going to want to know where his data goes when the server dies. ;-)

[slact]

Consider all the people running blogs on dedicated virtual hosts with very little RAM to play with. Should someone happen to write a neat little "every post gets its own chatroom" wordpress plugin, I wouldn't want those little bloggers to feel left out because their plan gives a puny amount of RAM.

Otherwise you're spot on.

[mmaunder]

True.

[slact]

meh, redis has persistence. berkeleydb may be added for persistence support, if there's enough interest. Closing this for now.