Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Consumer.partitions_for_topic requires metadata has already been fetched #1774

Closed
ershashu opened this issue Apr 1, 2019 · 4 comments
Closed

Consumer.partitions_for_topic requires metadata has already been fetched #1774

ershashu opened this issue Apr 1, 2019 · 4 comments

Comments

@ershashu
Copy link

ershashu commented Apr 1, 2019

Consumer 'partitions_for_topic' method is not returning the proper value.

We have used the method : partitions_for_topic(topic) in our code, it was working fine with version 1.4.4 but getting an error in latest one (1.4.5)
@dpkp
Copy link
Owner

dpkp commented Apr 2, 2019

partitions_for_topic() is currently non-blocking and will return empty if the client has not yet fetched metadata. It is likely that in the 1.4.4 you had already fetched metadata in the cache. I would suggest calling consumer.topics() to make a blocking metadata request prior to calling partitions_for_topic()

@dpkp dpkp changed the title Consumer unable to fetch partitions_for_topic Consumer.partitions_for_topic requires metadata has already been fetched Apr 2, 2019
@Baisang
Copy link
Contributor

Baisang commented Apr 2, 2019 •
edited
Loading

hey @dpkp, I recently ran into this issue as well in some scripts using version 1.4.2. The fix you described (fetching metadata with consumer.topics() works, but we never ran into the issue before (and it seems like the issue is more common now?). Not sure if something else changed (feels like it shouldn't have?) to explain this behavior.

Interestingly enough, it seems to only affects our integration tests, not actual usage of our scripts

@dpkp
Copy link
Owner

dpkp commented Apr 3, 2019

I believe the java client has been updated to make a blocking request for metadata in partitionsFor -- we probably should do the same.

@Baisang Baisang mentioned this issue Apr 7, 2019
Merged
@gablank gablank mentioned this issue May 14, 2019
Closed
@jeffwidman
Copy link
Collaborator

Fixed by #1781

@ghost ghost mentioned this issue Jun 9, 2019
Open
@jeffwidman jeffwidman mentioned this issue Jul 11, 2019
Closed
@jeffwidman jeffwidman mentioned this issue Aug 17, 2019
Merged
ofek pushed a commit to DataDog/integrations-core that referenced this issue Aug 20, 2019
@jeffwidman @ofek
Force initial population of the cluster cache (#4394)
5a5e533 
In `kafka-python` >= 1.4.5, `partitions_for_topic()` becomes a
non-blocking call, which can cause problems if the local metadata cache
is accessed before its populated:
dpkp/kafka-python#1774

So we proactively make sure the metadata request/response cycles has
finished and the cluster metadata cache is populated as part of the
kafka client creation process.

Also includes a few other minor fixups.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jeffwidman @dpkp @Baisang @ershashu