Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix control batch bug #73

Merged
merged 2 commits into from
Aug 21, 2023

Conversation

gkousouris
Copy link

@gkousouris gkousouris commented Aug 18, 2023

Problem

_unpack_message_set was modified in #72 to skip over transactional control batches.

Because of this, unpacked = _unpack_message_set may return an empty list.
However, later in the code we reference the last element of that list, unpacked[-1], to find how many messages have been consumed. This causes a list index out of range exception

Solution

To solve this, we do not want to reference unpacked[-1], but skip the message.

Testing

I have manually edited the file to this from within the venv and this error was fixed:

{...}/venv/lib/python3.10/site-packages/kafka/consumer/fetcher.py", line 830, in _parse_fetched_data
    last_offset = unpacked[-1].offset
IndexError: list index out of range
exception 'IndexError('list index out of range')' encountered running paastorm.{..}`

@gkousouris gkousouris force-pushed the u/georgios/KAFKA-36342_fix_control_batches_bug branch from ae6cace to c4ee776 Compare August 18, 2023 15:17
@gkousouris gkousouris requested a review from JayH5 August 18, 2023 15:18
@JayH5
Copy link

JayH5 commented Aug 21, 2023

LGTM. Do you want to bump the version in this PR?

@gkousouris
Copy link
Author

LGTM. Do you want to bump the version in this PR?

ah, might as well bump it now

@gkousouris gkousouris merged commit 5e508ed into master Aug 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants