Flirting with the idea of writing a Redis module that would enable updates and batches of updates, via reflection, of a flatbuffer stored as binary string at some key.
The surface area of the task appears fairly limited.
Currently these types of delta updates are impossible to achieve with cached flatbuffers.
The fact that right now you’d have to pull down the original flatbuffer, change it, and then overwrite the cache entry, race conditions between multiple clients trying to update the same flatbuffer, would result in updates being totally lost.
I currently use Mongo push a whole bunch of updates into arrays of update type inside nested tables of users. Then when requested by their target user, have to pull down this structure and serialize it into a flatbuffer. So of course I would prefer this flatbuffer to already be created, and simply push it off on the network.
My initial concern would be that constantly resizing flatbuffer vectors via reflection to make room for new updates, one at a time, will be a very inefficient / resource intensive process and that I’d be better off just creating the flatbuffer on demand.
But this relies on the assumption that the mongo process of accomplishing the same is much more efficient / performant, which may or may not be true. I’d hope they leave some empty space for an array to grow into, and then allocate another big chunk when that’s exhausted.
Maybe we could create some new… “mutable database mode”… where vectors are allocated with lots of extra memory to grow into? And then a stitch function that would delete all this empty space when pulled out of the database, making the flatbuffer tight again, ready for the network. This would make vector updates performant.
And the binary schemas needed for reflection can be stored right in Redis at some key, or statically loaded right into the module code. (Feeding it in via the network on demand would defeat the whole point).
Wondering if anyone has any input?
Flirting with the idea of writing a Redis module that would enable updates and batches of updates, via reflection, of a flatbuffer stored as binary string at some key.
The surface area of the task appears fairly limited.
Currently these types of delta updates are impossible to achieve with cached flatbuffers.
The fact that right now you’d have to pull down the original flatbuffer, change it, and then overwrite the cache entry, race conditions between multiple clients trying to update the same flatbuffer, would result in updates being totally lost.
I currently use Mongo push a whole bunch of updates into arrays of update type inside nested tables of users. Then when requested by their target user, have to pull down this structure and serialize it into a flatbuffer. So of course I would prefer this flatbuffer to already be created, and simply push it off on the network.
My initial concern would be that constantly resizing flatbuffer vectors via reflection to make room for new updates, one at a time, will be a very inefficient / resource intensive process and that I’d be better off just creating the flatbuffer on demand.
But this relies on the assumption that the mongo process of accomplishing the same is much more efficient / performant, which may or may not be true. I’d hope they leave some empty space for an array to grow into, and then allocate another big chunk when that’s exhausted.
Maybe we could create some new… “mutable database mode”… where vectors are allocated with lots of extra memory to grow into? And then a stitch function that would delete all this empty space when pulled out of the database, making the flatbuffer tight again, ready for the network. This would make vector updates performant.
And the binary schemas needed for reflection can be stored right in Redis at some key, or statically loaded right into the module code. (Feeding it in via the network on demand would defeat the whole point).
Wondering if anyone has any input?