KAFKA-3205 Support passive close by broker

[bondj]

An attempt to fix KAFKA-3205. It appears the problem is that the broker has closed the connection passively, and the client should react appropriately.

In NetworkReceive.readFrom() rather than throw an EOFException (Which means the end of stream has been reached unexpectedly during input), instead return the negative bytes read signifying an acceptable end of stream.

In Selector if the channel is being passively closed, don't try to read any more data, don't try to write, and close the key.

I believe this will fix the problem.

[fpj]

@bondj the change seems sensible to me, it avoids printing the error messages with the stack trace. I'm not sure why you're being flooded with those messages in your log. If the producer reconnects, then shouldn't it produce another error only x minutes later, for whatever timeout you have? I'm trying to understand if this fully solves the problem you're reporting.

[bondj]

Thanks, we're rolling this fix out across production at the moment. Each server connects to enough brokers and we have many many brokers that this message was the top 'error' message across all our applications.

[ijuma]

@bondj If this is a useful change, then you may consider submitting a PR with trunk as the target. That way, you will not have to patch again when you upgrade.

[bondj]

@ijuma Looking at the code, I think the problem is specific to 0.8 client talking 0.9 server. Hopefully, once we've upgraded our clients to 0.9 (which will take a while), we won't need the patch anymore.

[ijuma]

@bondj OK, great.