Skip to content
This repository has been archived by the owner on Mar 24, 2021. It is now read-only.

data of my topic in kafka is so big,when i get a consumer,i cost so much time and memoery #984

Open
SuperLight-007 opened this issue Dec 10, 2019 · 1 comment

Comments

@SuperLight-007
Copy link

How can i get the last message in topic ,whe topic have 180000 messages per seconds

@SuperLight-007
Copy link
Author

D:\software\IDE\python\python27\python.exe D:/workplace/health/app/platform/itoa-components/docker/itoa-gpkafka/resources/gpkafka_listener/job/kafka2GP/test.py
Traceback (most recent call last):
File "D:/workplace/health/app/platform/itoa-components/docker/itoa-gpkafka/resources/gpkafka_listener/job/kafka2GP/test.py", line 22, in
consumer = topic.get_simple_consumer(auto_offset_reset=OffsetType.LATEST,reset_offset_on_start=True)
File "D:\software\IDE\python\python27\lib\site-packages\pykafka\topic.py", line 265, in get_simple_consumer
**kwargs)
File "D:\software\IDE\python\python27\lib\site-packages\pykafka\simpleconsumer.py", line 244, in init
self.start()
File "D:\software\IDE\python\python27\lib\site-packages\pykafka\simpleconsumer.py", line 296, in start
self.reset_offsets()
File "D:\software\IDE\python\python27\lib\site-packages\pykafka\simpleconsumer.py", line 728, in reset_offsets
response = broker.request_offset_limits(reqs)
File "D:\software\IDE\python\python27\lib\site-packages\pykafka\broker.py", line 44, in wrapped
raise SocketDisconnectedError
pykafka.exceptions.SocketDisconnectedError

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant