Relation to Disruptor pattern?

[samuell]

I was wondering if you are using, or are aiming to make use of the principles in the disruptor pattern? (FWIW, there's a simplistic implementation in Go, of something like this, utilising a ring-buffer to get 6x speedup over plain channels)

[ezrosent]

This looks cool! Thanks for the links. When I get the chance I'll add these into the benchmarks (comparing Go to Go and Rust to any C/C++ implementations that may exist). Keep in mind that there a few senses in which these aren't apples-to-apples comparisons:

• It doesn't look like the Go implementation supports sending arbitrary types over a channel; the channels here (at least in Go) box their values, making many things slower. If this were implemented in the Go runtime, it could potentially be faster.

• The Go implementation essentially busy-waits (it does for ... { runtime.GoSched() }) when readers have out-run writers. This probably makes microbenchmarks perform better, but it seems wasteful for the case where a writer may just not be ready to send a value. Having a more principled blocking strategy where you just block until the writer is ready seems more reasonable; which is what we do here. It looks like there are many such "waiting strategies" supported in the original paper. As a result of this (and possibly also the underlying algorithm) I suspect that the go implementation will perform a bit better than the bounded channels here.

• Correct me if I'm wrong here (I've only skimmed the paper) but the Disruptor structure seems to target "communication graphs" with a small number of producers or consumers; the aim for the data-structures in this repo is to support (and scale with) arbitrary producers and consumers. Last time I ran some benchmarks on the Rust implementation (which is more optimized than the Go one), I think it performed worse at lower producer/consumer counts, but performed similarly or better at higher ones compared to the paper (which admittedly uses a Java implementation). It also seems to have a different API than a standard concurrent queue, which may make apples-to-apples comparisons difficult.

• The aim of the data-structures here is to support the full Go channel semantics. Among other things, this means supporting select (see 81099c5). While it may be possible to use a ring-buffer-like structure to support select, it isn't obvious to me how to do it. I'll post back here if it occurs to me in the meantime.

Does that sound reasonable to you?

P.S: The Rust implementation I'm talking about isn't the one on the rust_impl branch right now; but it will be posted soon.

[samuell]

Many thanks for the detailed explanations @ezrosent

I'm really quite new to all of this. Still learning.