Some issues with benchmark tool

[kozlovic]

Hello, I work on the nats.io team and was made aware of your project.
Thank you for integrating with NATS!

I noticed very low numbers for the NATS report and started to dig into it.
What I have found first is that you can't run the benchmark without first fixing the import:

https://github.com/ThreeDotsLabs/watermill-benchmark/blob/master/cmd/main.go#L10
Should be: "github.com/ThreeDotsLabs/watermill-benchmark/pkg"

Then I tracked down the biggest part of the slowness in the NATS consumer side to the unmarshaling (https://github.com/ThreeDotsLabs/watermill-nats/blob/master/pkg/nats/marshaler.go#L41). For 100,000 messages it is taking 2secs on my machine. This is something to look into.

I would recommend that you set the AckWait to higher number than 1sec, conversely, the default CloseTimeout is too high IMO.

You don't expose it seems the start position for the subscription. So I am not sure if this is why you stop/start the subscriptions at initialization time. With NATS Streaming you can pass options to specify where to start in the stream (DeliverAllAvailable() for all messages, etc..).

I was surprised with Kafka numbers compared to NATS. Should not the Kafka publisher require acks from the cluster? (https://github.com/ThreeDotsLabs/watermill-kafka/blob/master/pkg/kafka/publisher.go#L80) something like:

config.Producer.RequiredAcks = sarama.WaitForAll

Thanks for your attention and effort!

[kozlovic]

Also, upgrading the streaming server to latest (0.16.2) will help with file store performance on read.
But overall, there is a significant overhead with this lib compared to bare NATS streaming API. That is, whatever consumer performance you get, if you start a consumer with say the bench tool that we have (https://github.com/nats-io/stan.go/tree/master/examples/stan-bench), you should see a huge difference (even if I bypass the marshal/unmarshal in your nats implementation).

go run examples/stan-bench/main.go -np 1 -ns 0 -n 1000000 benchmark
go run examples/stan-bench/main.go -np 0 -ns 1 -n 1000000 benchmark

(this will send all first, then consume, similar to what your benchmark tool does).

[m110]

Hi @kozlovic! I'm sorry for such late response, I didn't receive notification of this issue for some reason.

Thanks a lot for pointing out possible issues, we will definitely look into it! I'll keep you updated.

[m110]

Import paths fixed in #4.