Inefficient use of buffers reduces potential throughput for basicPublish with small messages.

[fadams]

I've recently started experimenting with amqp091-go and I had in the back of my mind some past experience with AMQP in Rust, so I put together a very simple producer https://gist.github.com/fadams/28ddeb912d1421b01b6d4591cda62290 based on https://github.com/rabbitmq/rabbitmq-tutorials/blob/main/go/send.go that publishes a large number of relatively small messages in a loop. The performance was OK, but was around half the throughput of the roughly equivalent Rust version which didn't seem right.

I ended up profiling with pprof and noted that Syscall was the hottest point, so started looking at publishWithDeferredConfirmWithContenxt then send then sendOpen on Channel. What I noticed was that sendOpen in Channel does multiple send calls on Connection (methodFrame, headerFrame, N x bodyFrame), however the send on Connection calls WriteFrame on writer, but that method both writes to the buffer and flushes.

Flushing the write buffer for every Frame largely defeats the point of using a buffered writer, certainly for the case of small messages, and it is significantly more efficient to write the message by writing each Frame to the buffer then flushing the complete message.

This patch https://gist.github.com/fadams/7ab99557c67852121bbe4f1d8dcd2ee2 adds sendUnflushed() and flush() methods to Connection and replaces the ch.connection.send calls in Channel sendOpen with ch.connection.sendUnflushed and ends with an explicit ch.connection.flush call. The patch also moves the notifier used to reduce heartbeats to the flush method because all the frames in message are semantically related and there is no real advantage to sending per Frame vice per "group of related Frames" and for the case of small messages time.Now() is (relatively) expensive.

With this patch the observed throughput for small messages is around double the original observed throughput.

amqp091-go-patches.tar.gz

[lukebakken]

Please feel free to open a pull request. Thanks.

[fadams]

Done:
#142

[Zerpet]

Merged #142. Thank you!

[fadams]

You're very welcome. That's great. Thanks for picking this up so quickly.