WFLY-19419 Distributed timer service should consolidate timeouts that would execute in the past

[pferraro]

https://issues.redhat.com/browse/WFLY-19419

Also solves: WFLY-19271 Distributed timer service drops timeout events if server is suspended
https://issues.redhat.com/browse/WFLY-19271

---

<--- THIS SECTION IS AUTOMATICALLY GENERATED BY WILDFLY GITHUB BOT. ANY MANUAL CHANGES WILL BE LOST. --->

Wildfly issue links:

• WFLY-19514

<--- END OF WILDFLY GITHUB BOT REPORT --->

More information about the wildfly-bot[bot]

[bstansberry]

@tadamski @rachmatowicz Please review.

[pferraro]

Adding the hold label for now, as there are few things requiring further discussion:

• Yesterday, I discovered that, based on the implementations here vs here, there seems to be a discrepancy in behavior of missed timeouts between interval vs ScheduleExpression-based timers. While the old timer service implementation coalesces missed timeouts into a single event for interval timers, it does not appear to do the same for ScheduleExpression-based timers. If this is indeed the case, while the proposed changes would bring the behavior of the distributed timer service in line with the old timer service wrt interval timers, it would cause the same behavior wrt ScheduleExpression-timers to diverge.
• @rachmatowicz suggested that we expose a management attribute to allow users to enable/disable the coalescing of missed timeouts, which I thought was a good idea. If the previous point is valid, then we might actually want to consider an attribute per timer type - which would offer the greatest amount of flexibility wrt compatibility.

[pferraro]

Removing hold label.

I've included a reproducer and fix for WFLY-19514 to address the issue from my previous comment, where, in the local timer service, interval vs calendar timers behave differently wrt missed timeouts. With this change, both timer services will behave consistently for both interval and calendar-based timers.
I've logged a feature request (https://issues.redhat.com/browse/WFLY-19515) to expose ejb3 subsystem configuration for enabling/disabling the coalescing of missed timeouts, as well as deployment descriptor support to enable/disable this behavior per timer service.
Additionally, fixing the behavior of both timer service implementations wrt suspension will be handled separately.

[rachmatowicz]

@bstansberry @pferraro FYI, actively working on the review. Should be done Friday 12 July.

[bstansberry]

/retest

[bstansberry]

I'm going to defer this to 34.

[rachmatowicz]

I can see that the tests pass...so my understanding is just not deep enough at this stage to fully digest everything that is going on, but left some comments for consideration.

[rachmatowicz]

Met with Paul and discussed how LocalScheduler.run() and TimerScheduler.InvokeTask() work together to schedule and execute timeouts in such a way that any timeout callbacks that occur when the TimerService is not started are coalesced into one representative timer firing when the TimerService is restarted, so this looks good to me.

[pferraro]

Thanks @rachmatowicz !