fix hang on sync send if the connection has been closed

[sclasen]

10 second timeout on sync sends, close broker producer and broker if send times out

Fixes the hang I reported in my comments on #73

closing the broker producer and broker results in the connection being established when the broker comes back

[eapache]

I think this is basically trying to solve the same problem as #55? That has some unresolved issues as well, but I think I prefer doing this generically in the broker rather than special-casing it in the producer just because the producer happens to be the first one we noticed with this problem. It does theoretically affect other parts of the code as well, so I'd rather fix it just once.

[eapache]

I believe this was done when #96 was merged, let me know if it's still an issue.

[sclasen]

hey @eapache I am still seeing this behavior with 3137586

[sclasen]

Here's where sarama is sitting, after 10 min. The default timeout is 1 min ya? do I have to set it explicitly somewhere for it to have effect?

8 @ 0x1fdc6 0xc412 0xc858 0x3ba09 0x3ba86 0x42dfc 0x3c323 0x3ed3e 0x3b9f2 0x3d908 0x3d61b 0x271b 0x1ff90
#   0x3ba09 github.com/Shopify/sarama.(*produceMessage).enqueue+0x1b9       /Users/sclasen/code/src/github.com/heroku/event-shuttle/Godeps/_workspace/src/github.com/Shopify/sarama/produce_message.go:31
#   0x3ba86 github.com/Shopify/sarama.(*produceMessage).reenqueue+0x46      /Users/sclasen/code/src/github.com/heroku/event-shuttle/Godeps/_workspace/src/github.com/Shopify/sarama/produce_message.go:38
#   0x42dfc github.com/Shopify/sarama.func·010+0x3c                    /Users/sclasen/code/src/github.com/heroku/event-shuttle/Godeps/_workspace/src/github.com/Shopify/sarama/producer.go:336
#   0x3c323 github.com/Shopify/sarama.produceRequestBuilder.reverseEach+0x53    /Users/sclasen/code/src/github.com/heroku/event-shuttle/Godeps/_workspace/src/github.com/Shopify/sarama/produce_message.go:99
#   0x3ed3e github.com/Shopify/sarama.(*brokerProducer).flushRequest+0x66e      /Users/sclasen/code/src/github.com/heroku/event-shuttle/Godeps/_workspace/src/github.com/Shopify/sarama/producer.go:339
#   0x3b9f2 github.com/Shopify/sarama.(*produceMessage).enqueue+0x1a2       /Users/sclasen/code/src/github.com/heroku/event-shuttle/Godeps/_workspace/src/github.com/Shopify/sarama/produce_message.go:30
#   0x3d908 github.com/Shopify/sarama.(*Producer).genericSendMessage+0x2c8      /Users/sclasen/code/src/github.com/heroku/event-shuttle/Godeps/_workspace/src/github.com/Shopify/sarama/producer.go:167
#   0x3d61b github.com/Shopify/sarama.(*Producer).SendMessage+0x8b          /Users/sclasen/code/src/github.com/heroku/event-shuttle/Godeps/_workspace/src/github.com/Shopify/sarama/producer.go:136
#   0x271b  main.(*KafkaDeliver).deliverEvents+0x16b                /Users/sclasen/code/src/github.com/heroku/event-shuttle/deliver.go:90

1 @ 0x13991 0x2c68d 0x1ff90
#   0x2c68d timerproc+0xbd  /usr/local/Cellar/go/1.2.2/libexec/src/pkg/runtime/time.goc:232

2 @ 0x1fdc6 0xc412 0xc8ab 0x30d36 0x42256 0x421e2 0x1ff90
#   0x30d36 github.com/Shopify/sarama.(*Broker).responseReceiver+0x86               /Users/sclasen/code/src/github.com/heroku/event-shuttle/Godeps/_workspace/src/github.com/Shopify/sarama/broker.go:359
#   0x42256 github.com/Shopify/sarama.*Broker.(github.com/Shopify/sarama.responseReceiver)·fm+0x26 /Users/sclasen/code/src/github.com/heroku/event-shuttle/Godeps/_workspace/src/github.com/Shopify/sarama/broker.go:112
#   0x421e2 github.com/Shopify/sarama.withRecover+0x32                      /Users/sclasen/code/src/github.com/heroku/event-shuttle/Godeps/_workspace/src/github.com/Shopify/sarama/utils.go:27

[eapache]

ahhhh, this is the same issue as #65

[eapache]

I do not believe this is the right fix for this issue, but we can continue this discussion on the other bug

[sclasen]

👍