You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Docker container sentry_onpremise_snuba-subscription-consumer-events_1 restarting all the time. Into logs I am looking into repeating messages:
New partitions assigned: {Partition(topic=Topic(name='events'), index=0): 4}
Caught OffsetOutOfRange('KafkaError{code=OFFSET_OUT_OF_RANGE,val=1,str="Broker: Offset out of range"}'), shutting down...
Traceback (most recent call last):
File "/usr/local/bin/snuba", line 33, in
sys.exit(load_entry_point('snuba', 'console_scripts', 'snuba')())
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/usr/src/snuba/snuba/cli/subscriptions.py", line 224, in subscriptions
batching_consumer.run()
File "/usr/src/snuba/snuba/utils/streams/processing/processor.py", line 109, in run
self._run_once()
File "/usr/src/snuba/snuba/utils/streams/processing/processor.py", line 139, in _run_once
self.__message = self.__consumer.poll(timeout=1.0)
File "/usr/src/snuba/snuba/subscriptions/consumer.py", line 120, in poll
message = self.__consumer.poll(timeout)
File "/usr/src/snuba/snuba/utils/streams/synchronized.py", line 217, in poll
message = self.__consumer.poll(timeout)
File "/usr/src/snuba/snuba/utils/streams/backends/kafka.py", line 400, in poll
raise OffsetOutOfRange(str(error))
snuba.utils.streams.backends.abstract.OffsetOutOfRange: KafkaError{code=OFFSET_OUT_OF_RANGE,val=1,str="Broker: Offset out of range"}
The logs you shared for the install step are partial and they don't contain any useful information to see what might be going wrong. If this is truly a fresh install, and you don't have any concerns about data loss, simply run the following commands and then run ./install.sh again:
after deleting data into kafka directory I've solved my problem, thanks. Now sentry looks fine all containers under monitoring and didn't report problems during the whole day.
Version Information
Sentry 20.11.0eb42277
Description
Docker container sentry_onpremise_snuba-subscription-consumer-events_1 restarting all the time. Into logs I am looking into repeating messages:
New partitions assigned: {Partition(topic=Topic(name='events'), index=0): 4}
Caught OffsetOutOfRange('KafkaError{code=OFFSET_OUT_OF_RANGE,val=1,str="Broker: Offset out of range"}'), shutting down...
Traceback (most recent call last):
File "/usr/local/bin/snuba", line 33, in
sys.exit(load_entry_point('snuba', 'console_scripts', 'snuba')())
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/usr/src/snuba/snuba/cli/subscriptions.py", line 224, in subscriptions
batching_consumer.run()
File "/usr/src/snuba/snuba/utils/streams/processing/processor.py", line 109, in run
self._run_once()
File "/usr/src/snuba/snuba/utils/streams/processing/processor.py", line 139, in _run_once
self.__message = self.__consumer.poll(timeout=1.0)
File "/usr/src/snuba/snuba/subscriptions/consumer.py", line 120, in poll
message = self.__consumer.poll(timeout)
File "/usr/src/snuba/snuba/utils/streams/synchronized.py", line 217, in poll
message = self.__consumer.poll(timeout)
File "/usr/src/snuba/snuba/utils/streams/backends/kafka.py", line 400, in poll
raise OffsetOutOfRange(str(error))
snuba.utils.streams.backends.abstract.OffsetOutOfRange: KafkaError{code=OFFSET_OUT_OF_RANGE,val=1,str="Broker: Offset out of range"}
2020-12-08 14:33:49,175 New partitions assigned: {Partition(topic=Topic(name='events'), index=0): 4}
2020-12-08 14:33:49,178 Caught OffsetOutOfRange('KafkaError{code=OFFSET_OUT_OF_RANGE,val=1,str="Broker: Offset out of range"}'), shutting down...
I've already seen into #478 and that sollution didn't help me. Can someone have a look into it? During install I had the following logs:
Creating sentry_onpremise_snuba-api_run ... done
%3|1605784974.331|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 2ms in state CONNECT)
%3|1605784975.329|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2020-11-19 11:22:55,329 Connection to Kafka failed (attempt 0)
Traceback (most recent call last):
File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
%3|1605784976.331|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1605784977.331|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2020-11-19 11:22:57,332 Connection to Kafka failed (attempt 1)
Traceback (most recent call last):
File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
%3|1605784978.335|FAIL|rdkafka#producer-3| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1605784979.335|FAIL|rdkafka#producer-3| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2020-11-19 11:22:59,336 Connection to Kafka failed (attempt 2)
Traceback (most recent call last):
File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
client.list_topics(timeout=1)
cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
2020-11-19 11:23:01,312 Topic ingest-sessions created
2020-11-19 11:23:01,312 Topic events created
2020-11-19 11:23:01,312 Topic cdc created
2020-11-19 11:23:01,312 Topic errors-replacements created
2020-11-19 11:23:01,312 Topic outcomes created
2020-11-19 11:23:01,312 Topic event-replacements created
2020-11-19 11:23:01,312 Topic snuba-commit-log created
Creating sentry_onpremise_snuba-api_run ...
Creating sentry_onpremise_snuba-api_run ... done
The text was updated successfully, but these errors were encountered: