heka_tcp output plugin terminates if server is not listening anymore

[bbinet]

Today I noticed that if I temporarily stop my heka_tcp input plugin, then the heka_tcp ouput plugin that were sending data to that stopped plugin will terminate quickly with the following error in logs:

1484245022748499356 [error] output.heka_tcp terminated: process_message() instruction_limit exceeded

Note that my heka_tcp input/output plugin are configured with ssl support.

[trink]

I had one hang (possibly in socket.write(), process_message never returned) under very similar conditions. I was working with Ops and we were able to reproduce it without SSL but got no further. I am assuming this failure is related and expect to find something that stepped on memory (as there is nothing that would exhaust the instruction limit in that plugin).

[bbinet]

In fact it actually enters an endless loop because client:sendreturns error "Broken pipe", so the plugin tries a new send_message(msg, i):
https://github.com/mozilla-services/lua_sandbox_extensions/blob/master/socket/sandboxes/heka/output/heka_tcp.lua#L70

Patch to come.

[bbinet]

This is addressed in pull request #66

[trink]

Good catch, I just looked at our prod config and the instruction_limit was set to zero so that explains our hang as it was just infinitely looping.