Skip to content

Conversation

NicoK
Copy link
Contributor

@NicoK NicoK commented Mar 29, 2018

What is the purpose of the change

In a ProcessFunction, registering timers for each event via ctx.timerService().registerEventTimeTimer() using timestamps like ctx.timestamp() + timeout will get a millisecond accuracy and may thus create one timer per millisecond which may lead to some overhead in the TimerService.

This problem can be mitigated by using timer coalescing if the desired accuracy of the timer can be larger than 1ms. A timer firing at full seconds only, for example, can be realised like this:

long coalescedTime = ((ctx.timestamp() + timeout) / 1000) * 1000;
ctx.timerService().registerEventTimeTimer(coalescedTime);

As a result, only a single timer may exist for every second since we do not add timers for timestamps that are already there.

Please note that this PR includes #5788 and should also be merged into 1.3 and 1.4 docs to which it applies as well.

Brief change log

  • document timer coalescing for ProcessFunction

@NicoK
Copy link
Contributor Author

NicoK commented Mar 29, 2018

Actually, more advanced schemes using current watermark + 1 (which fires with the next watermark) for the event time timer should also go into the documentation. I'll extend the PR ...

each upcoming millisecond. Even if you do not do any processing for outdated timers in `onTimer`
(as above), this may put a significant burden on the Flink runtime.

Since there is only one timer per key and timestamp, however, you may coalesc timers by reducing the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo: coalesc -> coalesce

@NicoK
Copy link
Contributor Author

NicoK commented Apr 3, 2018

thanks - fixed and extended with an example for "schedule with the next watermark" for event-time timers

@twalthr
Copy link
Contributor

twalthr commented Apr 4, 2018

Thank you @NicoK. I will merge this...

@asfgit asfgit closed this in 7b0fc58 Apr 4, 2018
asfgit pushed a commit that referenced this pull request Apr 4, 2018
asfgit pushed a commit that referenced this pull request Apr 4, 2018
sampathBhat pushed a commit to sampathBhat/flink that referenced this pull request Jul 26, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants