snuba spans consumer is going crazy "Dropping invalid message"

[williamdes]

Self-Hosted Version

25.9.0

CPU Architecture

x86_64

Docker Version

28.4.0

Docker Compose Version

2.39.2

Machine Specification

• My system meets the minimum system requirements of Sentry

Steps to Reproduce

I have no idea

Expected Result

Stop the logs spam

Actual Result

snuba-spans-consumer-1                 | 2025-09-19 16:04:03,892 Dropping invalid message
snuba-spans-consumer-1                 | Traceback (most recent call last):
snuba-spans-consumer-1                 |   File "/usr/src/snuba/snuba/consumers/rust_processor.py", line 257, in poll
snuba-spans-consumer-1                 |     self.__inner.poll()
snuba-spans-consumer-1                 |   File "/.venv/lib/python3.11/site-packages/arroyo/processing/strategies/run_task_with_multiprocessing.py", line 817, in poll
snuba-spans-consumer-1                 |     self.__check_for_results(timeout=0)
snuba-spans-consumer-1                 |   File "/.venv/lib/python3.11/site-packages/arroyo/processing/strategies/run_task_with_multiprocessing.py", line 650, in __check_for_results
snuba-spans-consumer-1                 |     self.__check_for_results_impl(
snuba-spans-consumer-1                 |   File "/.venv/lib/python3.11/site-packages/arroyo/processing/strategies/run_task_with_multiprocessing.py", line 708, in __check_for_results_impl
snuba-spans-consumer-1                 |     raise message
snuba-spans-consumer-1                 | arroyo.dlq.InvalidMessage: (Partition(topic=Topic(name='snuba-spans'), index=0), 371943177, True)
snuba-spans-consumer-1                 | 2025-09-19 16:04:03,897 Dropping invalid message
snuba-spans-consumer-1                 | Traceback (most recent call last):
snuba-spans-consumer-1                 |   File "/usr/src/snuba/snuba/consumers/rust_processor.py", line 257, in poll
snuba-spans-consumer-1                 |     self.__inner.poll()
snuba-spans-consumer-1                 |   File "/.venv/lib/python3.11/site-packages/arroyo/processing/strategies/run_task_with_multiprocessing.py", line 817, in poll
snuba-spans-consumer-1                 |     self.__check_for_results(timeout=0)
snuba-spans-consumer-1                 |   File "/.venv/lib/python3.11/site-packages/arroyo/processing/strategies/run_task_with_multiprocessing.py", line 650, in __check_for_results
snuba-spans-consumer-1                 |     self.__check_for_results_impl(
snuba-spans-consumer-1                 |   File "/.venv/lib/python3.11/site-packages/arroyo/processing/strategies/run_task_with_multiprocessing.py", line 708, in __check_for_results_impl
snuba-spans-consumer-1                 |     raise message
snuba-spans-consumer-1                 | arroyo.dlq.InvalidMessage: (Partition(topic=Topic(name='snuba-spans'), index=0), 371943178, True)
snuba-spans-consumer-1                 | 2025-09-19 16:04:03,901 Dropping invalid message
snuba-spans-consumer-1                 | Traceback (most recent call last):
snuba-spans-consumer-1                 |   File "/usr/src/snuba/snuba/consumers/rust_processor.py", line 257, in poll
snuba-spans-consumer-1                 |     self.__inner.poll()
snuba-spans-consumer-1                 |   File "/.venv/lib/python3.11/site-packages/arroyo/processing/strategies/run_task_with_multiprocessing.py", line 817, in poll
snuba-spans-consumer-1                 |     self.__check_for_results(timeout=0)
snuba-spans-consumer-1                 |   File "/.venv/lib/python3.11/site-packages/arroyo/processing/strategies/run_task_with_multiprocessing.py", line 650, in __check_for_results
snuba-spans-consumer-1                 |     self.__check_for_results_impl(
snuba-spans-consumer-1                 |   File "/.venv/lib/python3.11/site-packages/arroyo/processing/strategies/run_task_with_multiprocessing.py", line 708, in __check_for_results_impl
snuba-spans-consumer-1                 |     raise message
snuba-spans-consumer-1                 | arroyo.dlq.InvalidMessage: (Partition(topic=Topic(name='snuba-spans'), index=0), 371943179, True)

Endlessly

Event ID

No response

[aldy505]

Are you still experiencing this? Do you have missing feature flags?

[williamdes]

Are you still experiencing this? Do you have missing feature flags?

Absolutely, and no missing flags.
I have the same config file with some early available flags added

[aldy505]

Are you still experiencing this? Do you have missing feature flags?

Absolutely, and no missing flags. I have the same config file with some early available flags added

Any wrong image tag on .env or .env.custom file?