Resiliency tests

[sirupsen]

It'd be awesome if Sarama has resiliency tests with Toxiproxy. I'm working on a Go client currently which should make it fairly easy. It does put an extra dependency on the tests, but I definitely think it's worth it. It'd make me trust Sarama a lot more.

Tell me how I can make your life easier in terms of Toxiproxy for testing. :-) I'll be working on adding Toxics to the client this weekend, but it supports up/down currently. I'll write a README later for the Go client, however, it shouldn't be too hard to figure out.

(incomplete) to-do list:

• improve mockbroker as appropriate so that all of the below tests are possible (Add the possibility to introduce latency to the mock broker #295 and maybe others)
• producer with multiple retry levels flowing concurrently
• Producer hangs when Retry.Max is 0 #294 (test written, just needs a fix)
• client behaviour when refreshing metadata under various error scenarios
• Consumer can spin on misbehaving broker #261 (needs fixing and a test)
• DRY up the tests (there are a lot of dup. code fragments that can be extracted into helper methods)

@eapache @wvanbergen @drdee @yagnik

[wvanbergen]

I mainly want to pair with you to set up some initial tests that use
toxiproxy to get a feel for how to approach this.

Can we also influence internal cluster communication with it, i.e.
connections outside the scope of the go client?

On Sat, Feb 14, 2015 at 9:45 AM, Simon Eskildsen notifications@github.com
wrote:

It'd be awesome if Sarama has resiliency tests with Toxiproxy. I'm working
on a Go client
https://github.com/Shopify/toxiproxy/blob/master/client/client.go
currently which should make it fairly easy. It does put an extra dependency
on the tests, but I definitely think it's worth it. It'd make me trust
Sarama a lot more.

Tell me how I can make your life easier in terms of Toxiproxy for testing.
:-) I'll be working on adding Toxics to the client this weekend, but it
supports up/down currently. I'll write a README later for the Go client,
however, it shouldn't be too hard to figure out.

@eapache https://github.com/eapache @wvanbergen
https://github.com/wvanbergen

—
Reply to this email directly or view it on GitHub
#271.

[sirupsen]

@wvanbergen depends, is it possible to change the listening port for Kafka?

[wvanbergen]

Yeah, but I am not quite sure how it works, because of how it advertises itself in Zookeeper. Might require some experimentation.

[sirupsen]

oh yes... that may be a bit complicated then

[eapache]

I'd prefer we do the toxiproxy tests against mock brokers, as that should be much easier to manage. We're testing Sarama here, not kafka :)

[sirupsen]

@eapache Sarama is a client of ZK and Kafka though, how can you be sure the failover etc. logic is implemented correctly unless you test against the real thing?

[eapache]

Kafka's broker behaviour is very well-defined, and (in many cases) much easier to simulate with mockbrokers than to trigger in a real cluster (e.g. three-way network partition during failover). Testing against that behaviour is all we need to verify Sarama - if the cluster behaves differently, that's a kafka bug.

For example, in TestProducerFailureRetry we use mock brokers test what happens when leadership moves (e.g. due to a rebalance) and the broker in question returns NotLeaderForPartition. That test could be easily copied+tweaked to test what happens when a broker crashes (disappears) and the connection sends a TCP reset instead. We could also throw toxiproxy in the middle to test what happens when latency is present. Doing a similar test against a live broker cluster is much more complicated.

Which isn't to say that we shouldn't try to be resilient to the kafka cluster behaving outside of spec, but I'm much more comfortable with "best effort" in that scenario (especially since the brokers themselves have been so reliable for us).

Tangential point: sarama does not currently talk to zk at all.

[eapache]

Significant chunks of this have ended up done over the past year, and the rest have been effectively forgotten. The main toxiproxy tests (#444) still haven't been merged because they exposed an upstream kafka bug which hasn't been fixed yet, but our test suite is in a much better spot now regardless.