Detecting ring buffer overflows?

[tmm1]

I'm using a simple ring buffer, as so:

buffer.NewRing(buffer.New(8<<20))

In order to size the ring buffer for my use-case appropriately, I would like to collect some stats on how often it overflows (causing old data to be overwritten). I'm happy to open a PR to this effect, but wanted to discuss a sane API for exposing this information first.

[djherbis]

Hey, sounds like a good idea to me. I'm going to be busy with my day job until later today but I'll be happy to share some ideas here after that.

[tmm1]

Thanks!

Looking at the implementation in ring.go, I'm also wondering if a lock is needed around ring.L if the buffer is being written and read from different goroutines.

[djherbis]

None of the buffers in this package are concurrent safe on their own. To write/read concurrently you should consider using nio for a buffered concurrent Pipe (using one of these buffers as the buffer).

Or, you can just create a new struct called something like SafeBuffer which is composed of a Mutex & another Buffer. Then just delegate all the calls to the inner Buffer, guarding them with the Mutex.

[tmm1]

I am using nio.NewReader with the ring buffer already, and I see now that nio.Pipe only allows a single read/write at a time.

[tmm1]

I went with the simplest possible option for now: a patch to add logging to the ring buffer on overflows:

diff --git a/ring.go b/ring.go
index 7da7f0a..389e49e 100644
--- a/ring.go
+++ b/ring.go
@@ -2,6 +2,7 @@ package buffer
 
 import (
 	"io"
+	"log"
 	"math"
 
 	"github.com/djherbis/buffer/wrapio"
@@ -45,6 +46,7 @@ func (buf *ring) Write(p []byte) (n int, err error) {
 	n, err = buf.WrapWriter.Write(p)
 	buf.L += int64(n)
 	if buf.L > buf.BufferAt.Cap() {
+		log.Printf("[WARN] Ring buffer %p overflowed by %d bytes", buf, buf.L-buf.BufferAt.Cap())
 		buf.L = buf.BufferAt.Cap()
 	}
 	return n, err

(I'm intermittently seeing some strange behavior in a complex network application using nio + buffer, which could be symptomatic of a buffer overflow. Since increasing the size of the buffer didn't seem to help, I'm hoping this patch will help me confirm whether the behavior is actually an overflow, vs some other bug in either the ring buffer or my application.)

[djherbis]

Do you have an example of the strange behavior you're seeing? (out of order data perhaps?)
I can try adding some more tests for the RingBuffer, I'll admit it's much less battle tested (and more convoluted) than all the other buffers.

I think your log trick is totally fine, if you'd prefer not to modify RingBuffer in your codebase, you can also just copy this buffer below, and drop it anywhere you need a 'log on overflow buffer'.

You could also detect an arbitrary Buffer is 'overflowing' via:

type OverflowStatBuffer struct {
  Buffer
}

func (buf *OverflowStatBuffer) Write(p []byte) (n int, err error) {
  oldLen := buf.Len()
  n, err = buf.Buffer.Write(p)
  if oldLen+int64(n) > buf.Len() {
    // bytes must have been dropped
    // record something? log something?
  }
  return n, err
}

So far I don't think there's any need to add this to the buffer package (since it's main use is debugging and it's easy to drop in and configure to each individual use case).

Hope that helps! If there is an issue with this package I'll help fix it fast for you, thanks for using it!

[djherbis]

Also, is there any reason you're preferring to drop old data instead of just allowing back-pressure to do it's thing?

[tmm1]

@djherbis Appreciate the help! I like the OverflowStatBuffer trick, it's quite a nice way to add logging without having to patch the underlying buffer implementation.

I've been using the ring buffer without issue across hundreds of mac/linux instances for the past few weeks, so I think the implementation is pretty solid. It took me a while to wrap my head around it, but now that I understand what's happening its actually quite elegant.

The issue I was investigating here was only manifesting on windows, and after extensive testing today I'm pretty sure it's not a problem with the buffer. (There appears to be some OS issue that impacts write throughput to a file when also trying to open/read the same file multiple times concurrently, apparently due to how anti-virus software hooks into file opens on windows).

Also, is there any reason you're preferring to drop old data instead of just allowing back-pressure to do it's thing?

Yes, the source of the data stream is very sensitive to back-pressure, and if I don't drain it fast enough it starts dropping packets aggressively.

[tmm1]

Also, is there any reason you're preferring to drop old data instead of just allowing back-pressure to do it's thing?

I think what you're saying here is that I could simply use buffer.New(8<<20) without Ring and the behavior would be exactly the same, except that back-pressure would be applied at the network layer after 8mb of buffering, instead of receiving and dropping data manually at the application level. (The latter might be preferable for applications where "new" data was prioritized over "old" data, but that's not the case for me.)

This seems obvious now, and would remove a lot of unnecessary code in my hot-path. I had assumed I needed Ring to be able to "re-use" available memory at front of buffer, but I think I see now how bytes.Buffer and the underlying byte slices take care of memory management.

Is my understanding correct?

[djherbis]

Hey, sorry for the slow reply.

Glad to know the ring buffer has been working for you!

I've also run into performance issues in windows with file writes (specifically when I'm using sync() to make sure the bytes land on disk and aren't sitting in the cache).

As for your second comment, you are correct. nio will do back pressure when used with buffer.New. The network layer will drop the new packets when the buffer is full [under pressure], and whenever space becomes available in the buffer, nio will re-use the space in the buffer.

Ring is only when you prefer to drop old data instead of new data as you said, which you don't seem to have a preference over for your use case.

I actually originally developed all of these buffers for a similar use case to yours. I wanted to have complete control on how my application responded under pressure. (apply back-pressure, drop old data, drop new data, or grow the buffer).

I probably should do a blog post or something explaining how this works, and how nio works with the buffers to apply the different pressure strategies.

[tmm1]

Thanks, I'm still trying to wrap my head around all the different types of buffers so your explanations are very helpful. I think memory buffers could probably use a bit more documentation, and an example in the README.

I realized after posting my last comment that I do have one reason not to apply back-pressure: I'm splitting my stream across multiple readers, and I don't want one slow one to stall the rest.

Currently I have a home-grown implementation similar to io.MultiWriter, but one that allows writers to be added and removed. I'm using a set of io.Pipes to distribute data, and then each reader is using an additional nio.Reader + buffer.Ring to drain its copy quickly as not to stall the rest.

However, looking now at djherbis/stream it appears to solve a very similar problem, so I'm trying to figure out how that fits in with buffer/nio and whether it makes sense for me to use that instead.

[djherbis]

Yeah djherbis/stream and djherbis/bufit were created for the fan-out style buffering (with two different use cases in mind).

Stream is for when you want the entire stream to be re-readable forever (best for finite-length streams), bufit is for when you want parts of the data stream which have been read by all the current readers to be dropped.

I'll add a todo for myself to do so more examples and better explanations.

[tmm1]

Thanks, it sounds like djherbis/bufit is what I've been looking for. I had come across it last week, but wasn't sure how it related to your other projects.

I have to say: all of these libraries are stellar, and have given me a new-found appreciation for the golang language and the power of the io interfaces.