Don't call conn.Close on the event loop

[mstoykov]

From #44 (comment)

While I did look into this a while back and was left with the impression it is fine due to the doc

Close closes the underlying network connection without sending or waiting for a close message.

This still calls net.Conn#Close which likely while much faster then most other calls will be a blocking one.

I doubt this is as big of a problem as if we were writing messages, but it still better if we do not.

We also do it in a bunch more places so this is not the only place that needs fixing.