Skip to content

Commit

Permalink
fix: pessimistically lookup the earliest key
Browse files Browse the repository at this point in the history
The earliest_offset function in the consumer seems to not always return
all tps which have been passed in but presumabely only the ones it has
data for.

After longer runtime it seems that the response MAY return a dict
not filled with all topic partitions which have been added. To bypass
this and not lead faust to crash, use the .get function on the returned
mapping and default to None on return.
  • Loading branch information
cbrand committed Mar 24, 2022
1 parent c230076 commit 221689f
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion faust/tables/recovery.py
Expand Up @@ -691,9 +691,10 @@ async def _build_offsets(
earliest = await consumer.earliest_offsets(*tps)
# FIXME To be consistent with the offset -1 logic
earliest = {tp: offset - 1 for tp, offset in earliest.items()}

for tp in tps:
last_value = destination[tp]
new_value = earliest[tp]
new_value = earliest.get(tp, None)

if last_value is None:
destination[tp] = new_value
Expand Down

0 comments on commit 221689f

Please sign in to comment.