You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Sending large number of large payloads via Producer.Send() shouldn't panic/crash
Actual behavior
Need some insights from you guys if this rings any bell or not. We've been using pure golang client for a few weeks now with good results until we just recently onboarded a new customer, who sends in large order files each of which contains many large records.
Typically record after protobuf encoding is about 150KB to 300KB. And we have millions of these records. Although the send is called in a single thread right now, when we start rapidly calling Send, our logs suddenly shows this:
.......
.......
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Error reading from connection"}
panic: send on closed channel
goroutine 3105 [running]:
github.com/apache/pulsar-client-go/pulsar/internal.(*rpcClient).Request.func1(0xc001e3bc00, 0x0, 0x0)
/go/pkg/mod/github.com/apache/pulsar-client-go@v0.0.0-20200316114055-4dc7855f99dc/pulsar/internal/rpc_client.go:97 +0xb6
github.com/apache/pulsar-client-go/pulsar/internal.(*connection).handleResponse(0xc0057945a0, 0x65, 0xc001e3bc00)
/go/pkg/mod/github.com/apache/pulsar-client-go@v0.0.0-20200316114055-4dc7855f99dc/pulsar/internal/connection.go:493 +0xb3
github.com/apache/pulsar-client-go/pulsar/internal.(*connection).internalReceivedCommand(0xc0057945a0, 0xc001e3bc00, 0x0, 0x0)
/go/pkg/mod/github.com/apache/pulsar-client-go@v0.0.0-20200316114055-4dc7855f99dc/pulsar/internal/connection.go:403 +0x2e4
github.com/apache/pulsar-client-go/pulsar/internal.(*connection).run(0xc0057945a0)
/go/pkg/mod/github.com/apache/pulsar-client-go@v0.0.0-20200316114055-4dc7855f99dc/pulsar/internal/connection.go:325 +0x304
github.com/apache/pulsar-client-go/pulsar/internal.(*connection).start.func1(0xc0057945a0)
/go/pkg/mod/github.com/apache/pulsar-client-go@v0.0.0-20200316114055-4dc7855f99dc/pulsar/internal/connection.go:195 +0x59
created by github.com/apache/pulsar-client-go/pulsar/internal.(*connection).start
/go/pkg/mod/github.com/apache/pulsar-client-go@v0.0.0-20200316114055-4dc7855f99dc/pulsar/internal/connection.go:192 +0x3f
Steps to reproduce
Not sure how this can be repro'ed on your side. I'm reaching out to your rep's slack channel within our company to investigate as well.
System configuration
Pulsar version: github.com/apache/pulsar-client-go v0.0.0-20200316114055-4dc7855f99dc Go version: 1.12
The text was updated successfully, but these errors were encountered:
Original Issue: apache#239
Expected behavior
Sending large number of large payloads via
Producer.Send()
shouldn't panic/crashActual behavior
Need some insights from you guys if this rings any bell or not. We've been using pure golang client for a few weeks now with good results until we just recently onboarded a new customer, who sends in large order files each of which contains many large records.
Typically record after protobuf encoding is about 150KB to 300KB. And we have millions of these records. Although the send is called in a single thread right now, when we start rapidly calling Send, our logs suddenly shows this:
Steps to reproduce
Not sure how this can be repro'ed on your side. I'm reaching out to your rep's slack channel within our company to investigate as well.
System configuration
Pulsar version: github.com/apache/pulsar-client-go v0.0.0-20200316114055-4dc7855f99dc
Go version: 1.12
The text was updated successfully, but these errors were encountered: