Timers are pretty unscalable ...

[przygienda]

OK, profiled memory on my application and I need 100K timers or so. Unfortunately, MIO timers just explode on the heap since it looks like the whole pool of the timers is pre-alloc'ed so it's not a feasible solution.

I'm thinking:

a) either I back off MIO timers completely albeit it's a pity. I would need a heap or something
b) we think about writing something where the timers are allocated in slabs when they are being added

[przygienda]

yepp, looked @ the implementation. That would need something more along of

http://www.cs.columbia.edu/%7Enahum/w6998/papers/ton97-timing-wheels.pdf

while the Slab allocation is very good though.

Additionally, under very heavy load a system may limit the # of timers it can expire on a tick and with that the remaining timers would have to be efficiently moved to the next tic. It's timer slip and not unusual in heavily loaded large-scale systems ...

[andrewjstone]

I wrote ferris based on a number of the ideas in that paper. While it is missing the optimization to limit the number of expired timers, and doesn't migrate timers across wheels, it may be good enough for your use case. It shouldn't be too hard to add these features, I just haven't had the time or incentive to do so.

I'd be willing to accept any PR making ferris more complete.

[carllerche]

Thanks for the report. The Mio timer is deprecated in favor of implementing one in a third party crate.

That being said, you can "back off" on triggering timers by not polling timers off of the queue (and deferring to another tick). Also, re: heap allocation, I'm not sure what your specific case is, but generally if you are saying you need 100k timers, you should have that memory reserved to ensure capacity.

Since only critical bug fixes for the timer will be merged in at this point, I'm going to close this issue in favor of (somebody else 😄) maintaining a timer in another crate.

[przygienda]

Carl, Ok, point taken. Yes, I saw ferris but it looks early, have to check whether it even supports Evented. I see they refer to same paper ;-) There was an even better, but too complex work on calendar scheduling AFAIR having written these kind of things in the 90s multiple times.

And nope, the 100K is not really how an embedded system @ large scale works normally, i.e. people do NOT pre-alloc everything to peak-utility, the memory requirement would be unsustainable. I may need UP to 100K timers while I'd like to start with maybe couple K and then add couple K every time I blow the limit ...

I'll look @ Ferris in more detail ...

[przygienda]

hmm, ferris driven by a MIO timer tick ;-) ?

[hicqu]

@przygienda , I am working with Timer too. could you help me on this #541 ?
Thank you.

[andrewjstone]

Yup, that's precisely how I use it elsewhere. Just drive it by a timer tick :)

…

On Tue, Feb 28, 2017 at 11:41 PM, przygienda ***@***.***> wrote: hmm, ferris driven by a MIO timer tick ;-) ? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#553 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAcf-fC9HW7hRHCqzxA3Wt8NrWZoou5Vks5rhPb_gaJpZM4MN_xd> .

[przygienda]

Ok, makes absolute sense. I'll encaps it into something that has a high-prio and low-prio wheel with max amount timers and the rest will be 'slipped' into slots after that ... That's how you deal with very heavy load where timers have to back off if there are too many in a slot to process in a tick. I'll see whether I'll be allowed to upstream it ...

BTW, insert functions would benefit from some documentation ;-)

[andrewjstone]

Cool. Best of luck. Let me know how it goes (good or bad). This is my first timer wheel and I haven't really done any measurements on it yet individually. Any feedback, patches etc.. you can share would be appreciated. It's nice to have someone experienced in this area playing with it. Thank you.

…

On Wed, Mar 1, 2017 at 12:05 AM, przygienda ***@***.***> wrote: Ok, makes absolute sense. I'll encaps it into something that has a high-prio and low-prio wheel with max amount timers and the rest will be 'slipped' into slots after that ... That's how you deal with very heavy load where timers have to back off if there are too many in a slot to process in a tick. I'll see whether I'll be allowed to upstream it ... — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#553 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAcf-e3t76QkZFIClpTPZDCLz4WGoghbks5rhPyggaJpZM4MN_xd> .

[przygienda]

My pleasure. Wrote a couple of those using different approaches. Deltas were good & the ultimate was calendar scheduling AFAIR but damn'd complex ... If you run large amounts of those, slabbing would be good just like MIO is doing it. It increases performance dramatically when you run huge number timers (but given jemalloc does slabbing internally, not clear) ... That approach is also very good

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.23.6219&rep=rep1&type=pdf

Previous mentee of mine ;-)

[andrewjstone]

Excellent!

…

On Wed, Mar 1, 2017 at 12:16 AM, przygienda ***@***.***> wrote: My pleasure. Wrote a couple of those using different approaches. Deltas were good & the ultimate was calendar scheduling AFAIR but damn'd complex ... If you run large amounts of those, slabbing would be good just like MIO is doing it. It increases performance dramatically when you run huge number timers (but given jemalloc does slabbing internally, not clear) ... That approach is also very good http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1. 23.6219&rep=rep1&type=pdf Previous mentee of mine ;-) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#553 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAcf-XOjenQpOqdktCK9BmdBZs5mgntOks5rhP88gaJpZM4MN_xd> .