Stray {ref, ok} message can kill the writer process

[lukebakken]

Describe the bug

Complete details can be found in this discussion:

#9803

Reproduction steps

No reproduction steps exist at this time.

Expected behavior

The source of the stray message is found, or the message is logged and then ignored.

Additional context

@gomoripeti says that there is code in the ra library that fits the bill for the format of the stray message (link). Waiting on him to point out what code it is. This may, of course, be a message from within OTP itself.

[michaelklishin]

@lukebakken should we simply ignore this message in the writer and try to log some context at debug level? I mean, the writer cannot do much about this message, and this may allow us to log relevant details about the sender.

[lukebakken]

This feels like a warning level sort of message, maybe?

[michaelklishin]

@lukebakken logging a warning works for me #9994.

Those who have a way to reproduce and are interested in digging in now have a place to add more logging and tracing.

[michaelklishin]

rabbit_writer interacts with

• A socket, which in Erlang is a source of messages
• A statistics timer, which uses a reference for identity
• It implements a few bits required by the sys module

I'd start with those three for most likely senders of these stray {ref(), ok} messages.

[lukebakken]

I'm going to re-open this issue to devote some time (at some point) to figure out the root cause.

@michaelklishin will having this issue open affect release notes? I can open a new issue if so.

[michaelklishin]

Don't worry about the release notes, we can leave it open

[kjnilsson]

I can't see any usages in Ra of {ref, ok} the closest I have is https://github.com/rabbitmq/ra/blob/ba3293a3ea29e6aa738726ba22e951f1ce8e7af4/src/ra_server_sup_sup.erl#L243
but that should be a proper reference() not the atom ref.

[lukebakken]

@kjnilsson yes, the error in the logs shows an actual reference, not the atom ref

[michaelklishin]

I've closed this because as of #9991, we hope that the writer process does not die. If someone wants to investigate where these messages originate from, you are welcome to do it.

It's just that this specific issue, as it is worded right now, should be addressed.