Sentry loses connection to Kafka

[mikhno-s]

Version Information

There were few version with that bug
Version: 20.7.0 (lose connection in few hours after start and stops receiving new event)
Version: 20.8.0 (loses connection in few hours after start, but trying reconnect after - but loosing events)
Version: 20.9.0 (loses connection in few hours after start and stops receiving new event)

Description

Sentry services lose connection to kafka. Some people describes that problem in #629 (search for timed out: disconnect)

Steps to Reproduce

1. Setup sentry instance via install.sh
2. Create project and start sending there events
3. Wait for stopping receiving events and seeing following logs.

Logs

%3|1601458843.990|FAIL|rdkafka#producer-1| [thrd:sys-logs-kfk-2.42:6667/2637]: sys-logs-kfk-2.42:6667/2637: 1 request(s) timed out: disconnect
%3|1601458843.990|ERROR|rdkafka#producer-1| [thrd:sys-logs-kfk-2.42:6667/2637]: sys-logs-kfk-2.42:6667/2637: 1 request(s) timed out: disconnect

Checked with different kafka setups. Will check with Aws MSK.

[BYK]

Create project and start sending there events

How many events and at what rate please?

[BYK]

Closing due to inactivity.