Faust silent failure on startup introduced with PR #403

[austinnichols101]

I have observed faust intermittently failing silently on startup with the 1.8.x series and have been able to trace it back to PR #403, commit c0daac1.

Note: special thanks to @patkivikram for helping me track this down...

Faust failing silently on startup

[2019-10-19 18:22:32,085] [46808] [INFO] [^---Recovery]: Resuming flow... 
[2019-10-19 18:22:32,085] [46808] [INFO] [^---Recovery]: Seek stream partitions to committed offsets. 
[2019-10-19 18:22:35,510] [46808] [INFO] [^--Fetcher]: Starting... 
[2019-10-19 18:22:35,510] [46808] [INFO] [^---Recovery]: Worker ready 
[2019-10-19 18:22:35,511] [46808] [INFO] [^Worker]: Ready 
[1]    46808 killed     faust --app streampunk.app worker --loglevel info

The _add_gap function in faust/transport/consumer.py is being called with a VERY large offset_from / offset_to delta. In the screenshot below a list is populated from 1 to 2,288,752,002 which results in python running out of memory and triggering the failure. Note that in the screenshot I added an exception handler to see if I could trap the error (I could not - python fails silently).

PyCharm debugger output

Resource Monitor: Memory Usage

PyCharm debugger: Values passed to _add_gap

Versions

• Python version 3.7.4
• Faust version 1.8.1
• Kafka version 2.3

[ask]

This is fixed in master, and will be part of 1.9 release

[austinnichols101]

I'm still seeing this problem in 1.9.0

The gap_for_tp list grows beyond available memory.

2,951,296,758 offset_from
3,136,952,810 offset_to

185,656,052 target length of gap_for_tp (OOM)

[austinnichols101]

state before calling _add_gap