-
Notifications
You must be signed in to change notification settings - Fork 616
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add ability to store message IDs persistent #541
Comments
Quick thoughts, redis seems like overkill. (and isn't really made for persistency afaik) As long as it's optional (and simple) it's ok for me. |
Bolt seems cool but I'd really prefer something that continues to work easily on heroku, where I have a few instances running :) I'll think on it. I think you're right about redis not being for persistence. Perhaps postgres. Free plan has 10,000 rows, which exactly matches how much history our slack instance has anyhow...! |
Why do you think bolt wouldn't work? |
Unless I'm misunderstanding, no heroku service addons offer bolt. Heroku is stateless without a file system, so it needs an add-on provider for its ability to have services (databases, caches, etc) as push-button |
Ok, so on further investigation, redis actually seems a step in the right direction. I don't think we need full persistency perhaps, but we need to be able to query based on more than primary key (original message timestamp), which is only what the lru cache allows. So assume an example of wanting to rethread messages. We instead need something that would allow us to query for a mID value in BrMsgID from the lru cache (using a ThreadTimestamp from thread reply ), and get the key (which is the original Timestamp). We could then know all the other messages to update from there. For now, I suppose I could just skirt the change by looping through the lru cache. Curious your thoughts on a larger solution though |
We should use https://github.com/abronan/valkeyrie if this gets implemented. |
perrrrfect. I'll prob work on this next. Getting to crunch time on another project, so might not be as soon as I'd like |
Potentially helpful link: https://medium.com/@nevio/proper-way-how-to-marshal-and-unmarshal-golang-structs-within-the-redis-cache-855423902787 Was curious about using msgpack (which redis-cache golang lib uses), but seems marshalling as json should work fine: https://www.reddit.com/r/golang/comments/476fdi/what_is_best_serialization_format_for_boltdb_for/ |
The valkeyrie library should handle all that. There shouldn't be any redis or bolt specific code in matterbridge. |
No, valkeyrie doesn't do marshalling -- just accepts byte slices as far as I can tell. Anyhow, I have a code spike here: Busy week coming up, so that's likely all I'll get done, but anyone is welcome to squash my commits and then fix it up :) EDIT: Works for edits and threads, though file cache is still using a more rudimentary connection approach from earlier in my experimentation |
@patcon okay nice! |
+1 for this feature, and also for the ability to use a local/embeddable database that is created automatically when needed. Not requiring any other services to be running on the machine is a big advantage of this program. |
May be try to use gokv - Simple key-value store abstraction and implementations for Go (Redis, Consul, etcd, bbolt, BadgerDB, LevelDB, Memcached, DynamoDB, S3, PostgreSQL, MongoDB, CockroachDB and many more) |
@patcon hello! Can you share your code with valkeyrie implementation for persistent message store? I want to implement this thing and any help will be useful :) |
@42wim hello! Can you share your thoughts on implementing this feature? I want to realize this feature. For now in code i see:
IMHO there is two possible implementations for persistent storing message IDs:
And we can initialize cache once and use it in all required places. |
@42wim ping? :) |
A-ha! I realized that I transferred the repo to another org, and my personal fork was a more recent one without my old branches. The code is really old, but it's here 🎉 Anyhow, good luck and godspeed @sas1024 ! I'm rooting for you! |
Hey there! Any updates on this? I just implemented Matterbridge to synchronize messages between Slack and Mattermost but it tends to crash twice a week. This is not a problem in itself, except it breaks all threads (which is quite annoying) so I'd be very happy if persistent threads were in the roadmap. |
@sas1024 did you ever get anywhere on writing a solution for this feature? The code link @patcon posted does not appear to exist anymore either, hoping you have it saved somewhere?
|
@yousefmansy1 I'm sorry, but I haven't started working on this feature yet :( |
Ok I see, no worries. |
No, I don't have this code, this link was broken several months. |
No worries, thanks for the response. |
Resolves 42wim#541
Resolves 42wim#541
Resolves 42wim#541
Resolves 42wim#541
oy... i'm sorry, i'm seeing that I must have confused to clipboard, and shared the same link again. It can't be merged anymore without conflict resolution, but here's the branch with the work: |
So that between Slack instance, we can preserve threading (#529) and send emojis (#526) with more consistent behaviour :)
Thoughts? I could be pretty simple, and totally optional for people who choose to stay utterly stateless.
With
matterbridge-heroku
, adding persistence of relationships could be as simple as:Curious your thoughts on this very Slack-specific addition
The text was updated successfully, but these errors were encountered: