Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

If QueueEventsOptions lastEventId is not set to `$`, it replays the full (all-time) event history on process restart #76

Open
jbr opened this issue Nov 21, 2019 · 4 comments · May be fixed by #82

Comments

@jbr
Copy link
Contributor

@jbr jbr commented Nov 21, 2019

Possible solutions:

  • Mark stream events as processed with XACK
  • Set the default lastEventId to "$" instead of "0-0", only streaming new events. Downside of this approach: Events are ignored during QueueEvents listener unavailability
@stansv

This comment has been minimized.

Copy link
Member

@stansv stansv commented Nov 22, 2019

Hmm, I thought we've already replaced "0-0" to "$" (#29)..

@jbr

This comment has been minimized.

Copy link
Contributor Author

@jbr jbr commented Nov 22, 2019

@stansv https://github.com/taskforcesh/bullmq/blob/master/src/classes/queue-events.ts#L29 is 0-0 and I don’t see any other redis commands than the XREAD


update: I found

this.client.xtrim(key, 'MAXLEN', '~', 100);
, but this still means that the last 100 events will always be replayed on process restart, which would only be appropriate for some uses of global events. I'm not super familiar with the streams api, but it seems like the intended use is to use XREADGROUP and XACK to consume events on behalf of a read group.

In my use case, replaying those events immediately in a loop on process restart actually blocks a server from binding to a port until those hundred events have been processed. It's probably a different Issue that event processing should be on a setTimeout(,0)/setImmediate/process.nextTick, not in a blocking loop in new QueueEvents(...)

@jbr

This comment has been minimized.

Copy link
Contributor Author

@jbr jbr commented Nov 27, 2019

Would a PR that uses XREADGROUP be welcome?

@manast

This comment has been minimized.

Copy link
Contributor

@manast manast commented Nov 27, 2019

@jbr the code above regarding XTRIM is for internal delayed jobs events, 100 is a extremely safe value so that we do not need to rely on watchdogs as we had in Bull3.

Regarding QueueEVents, the workaround is to pass '$' to only receive new events. I thought I had changed that already but for some reason is not the default value yet.

@jbr jbr linked a pull request that will close this issue Dec 6, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.