Delay agent skipping events

[crazycaveman]

I'm having some trouble with the delay agent. I'm using that to rate limit how many requests are sent to a webhook at once (5 at a time with an interval of 2 minutes). Sometimes, the same number of events in do not get emitted from the delay agent to the post agent (see screenshot). Is there a way to figure out why this is failing intermittently? TBH, I haven't kept a close eye on it and just noticed it again today. Re-emitting a skipped event causes it to process correctly.

Screenshot and agent configs:

• Delay agent:

{
  "expected_receive_period_in_days": "3",
  "max_events": "500",
  "keep": "newest",
  "max_emitted_events": "5"
}

[dsander]

Are you using multiple delayed_job background processors? I think it is possible that an Event is dropped when the Agent is ran simultaneously on two processors. If that is the case the root cause would be similar to #2389

[TheChrisK]

Hi, I manage the installation that @crazycaveman is referring to.

Yes, we are using multiple delayed_job processes, 8 total. The VM has 4c and 4GB of RAM running on Ubuntu 16.04.5.

Was there a config change that resolved the other issue or just a different way of building the agent?

[dsander]

I think the only way to resolve the issue is to introduce a lock that ensures that Agents which modify the memory do not run on two different delayed job processes at the same time.