Sentry Problems

https://develop.sentry.dev/self-hosted/troubleshooting/

Kafka Offsets

From https://develop.sentry.dev/self-hosted/troubleshooting/#kafka and the linked issue https://github.com/getsentry/self-hosted/issues/478

ssh to sentry.galaxyproject.org as root and cd /opt/sentry

1. There will probably be containers crashing, check uptime with docker compose ps.

2. Use docker compose -n 100 logs <service>, where you will probably see errors like:

   arroyo.errors.OffsetOutOfRange: KafkaError{code=_AUTO_OFFSET_RESET,val=-140,str="fetch failed due to requested offset not available on the broker: Broker: Offset out of range (broker 1001)"}
   

   Above the traceback you should also find topic name(s). Group name(s) will? be similar to the service name but can be found with the first 2 steps from the troubleshooting:

   docker compose run --rm kafka kafka-consumer-groups --bootstrap-server kafka:9092 --list
   docker compose run --rm kafka kafka-consumer-groups --bootstrap-server kafka:9092 --group snuba-consumers --describe

   Replace snuba-consumers if it's a different group like snuba-events-subscriptions-consumers.

3. Stop the failing containers and try to reset offsets:

   docker compose run --rm kafka kafka-consumer-groups --bootstrap-server kafka:9092 --group snuba-consumers --topic events --reset-offsets --to-latest --execute

   If you get:

   Error: Assignments can only be reset if the group 'snuba-post-processor' is inactive, but the current state is Stable.

   Then stop all containers and start up only zookeeper and kafka, then try again:

   docker compose down
   docker compose up -d zookeeper kafka