Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ISSUE-239: panic during rapid send of many large-sized payloads: "panic: send on closed channel" #56

Closed
sijie opened this issue May 5, 2020 · 0 comments
Closed

ISSUE-239: panic during rapid send of many large-sized payloads: "panic: send on closed channel" #56

sijie opened this issue May 5, 2020 · 0 comments
Labels
bug triage/week-19

Comments

@sijie
Copy link
Member

sijie commented May 5, 2020

Original Issue: apache#239

Expected behavior

Sending large number of large payloads via Producer.Send() shouldn't panic/crash

Actual behavior

Need some insights from you guys if this rings any bell or not. We've been using pure golang client for a few weeks now with good results until we just recently onboarded a new customer, who sends in large order files each of which contains many large records.

Typically record after protobuf encoding is about 150KB to 300KB. And we have millions of these records. Although the send is called in a single thread right now, when we start rapidly calling Send, our logs suddenly shows this:

.......
.......
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:46Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:47Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Error reading from connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnecting to broker in 100ms"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connecting to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"TCP connection established"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection is ready"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Resending 1 pending batches"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Reconnected producer to broker"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Failed to write on connection"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Connection closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"WARNING","message":"Connection was closed"}
{"timestamp":"2020-05-05T03:06:48Z","severity":"INFO","message":"Error reading from connection"}
panic: send on closed channel

goroutine 3105 [running]:
github.com/apache/pulsar-client-go/pulsar/internal.(*rpcClient).Request.func1(0xc001e3bc00, 0x0, 0x0)
	/go/pkg/mod/github.com/apache/pulsar-client-go@v0.0.0-20200316114055-4dc7855f99dc/pulsar/internal/rpc_client.go:97 +0xb6
github.com/apache/pulsar-client-go/pulsar/internal.(*connection).handleResponse(0xc0057945a0, 0x65, 0xc001e3bc00)
	/go/pkg/mod/github.com/apache/pulsar-client-go@v0.0.0-20200316114055-4dc7855f99dc/pulsar/internal/connection.go:493 +0xb3
github.com/apache/pulsar-client-go/pulsar/internal.(*connection).internalReceivedCommand(0xc0057945a0, 0xc001e3bc00, 0x0, 0x0)
	/go/pkg/mod/github.com/apache/pulsar-client-go@v0.0.0-20200316114055-4dc7855f99dc/pulsar/internal/connection.go:403 +0x2e4
github.com/apache/pulsar-client-go/pulsar/internal.(*connection).run(0xc0057945a0)
	/go/pkg/mod/github.com/apache/pulsar-client-go@v0.0.0-20200316114055-4dc7855f99dc/pulsar/internal/connection.go:325 +0x304
github.com/apache/pulsar-client-go/pulsar/internal.(*connection).start.func1(0xc0057945a0)
	/go/pkg/mod/github.com/apache/pulsar-client-go@v0.0.0-20200316114055-4dc7855f99dc/pulsar/internal/connection.go:195 +0x59
created by github.com/apache/pulsar-client-go/pulsar/internal.(*connection).start
	/go/pkg/mod/github.com/apache/pulsar-client-go@v0.0.0-20200316114055-4dc7855f99dc/pulsar/internal/connection.go:192 +0x3f

Steps to reproduce

Not sure how this can be repro'ed on your side. I'm reaching out to your rep's slack channel within our company to investigate as well.

System configuration

Pulsar version: github.com/apache/pulsar-client-go v0.0.0-20200316114055-4dc7855f99dc
Go version: 1.12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug triage/week-19
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@sijie