*: run network fault tests with proxy #9081

gyuho · 2018-01-02T18:08:57Z

Problem

Current functional-tester depends on Linux utilities: iptables, iproute2, and tc. Which complicates and slows down tester deployments (require root access, impossible to test locally, hard to containerize, etc.). Also we never test real network partitions in CIs. We should simulate common network conditions to simplify test workflow (ideally package everything in container image).

Solution (what this PR does)

Implements proxy/transport.Proxy that simulates network faults, also useful for clientv3 integration network partition tests(Test clientv3 balancer under network partitions, other failures #8711) with the future embed package migration(use embed in integration package #8416).
Proxy with very small overhead (<500μs per request), negligible for etcd use cases (see benchmark results in tools/etcd-test-proxy/README.md).
functional-tester routes incoming advertise URL traffic to listen URL, to interfere peer/client traffic (no more iptables).
Containerizes functional-tester components (now just docker pull and run).
CI injects network partitions to functional-tester cluster (previously couldn't).

Other Changes

When network faults are injected, we expect cluster disrupts after election timeouts. Currently there's no waiting before recovering injected failures e.g. add latency and remove right away; nothing was being tested. failureDelay wrapper was there but never used. This PR adds delays to trigger leader elections, and ensures cluster continue to operate under such circumstances.

Future Work

gyuho · 2018-01-12T17:51:11Z

@hexfusion @spzala Can you take a look when you have time?

The plan is merge this after 3.3 release (early February), so not urgent.

Thanks!

spzala · 2018-01-14T17:17:52Z

@gyuho thanks and sure, in mid of understanding couple other issues related work but will be reviewing afterwards considering no urgency here.

gyuho · 2018-01-19T19:27:59Z

Also cc @jpbetz. Can you take a look?

This is first step to make functional-tester deploy easier.
Once this gets merged after 3.3 release, I will document container-based deploy workflow.

jpbetz

Thanks for getting this done! A couple suggestions.

jpbetz · 2018-01-19T20:47:57Z

hack/scripts-dev/Makefile

@@ -23,11 +23,32 @@ clean:
 	rm -f ./clientv3/integration/127.0.0.1:* ./clientv3/integration/localhost:*
 	rm -f ./clientv3/ordering/127.0.0.1:* ./clientv3/ordering/localhost:*

+
+
 _GO_VERSION = 1.9.2
 ifdef GO_VERSION
 	_GO_VERSION = $(GO_VERSION)
 endif


In make, just GO_VERSION?=1.9.2 should be all you need to have a configurable env var with a default. The ifdef and _* var appear unnecessary here. Same for a few other of the below vars.

jpbetz · 2018-01-19T20:52:54Z

hack/scripts-dev/Makefile

+build-docker-functional-tester:
+	$(info GO_VERSION: $(_GO_VERSION))
+	$(info ETCD_VERSION: $(_ETCD_VERSION))
+	@cat ./Dockerfile-functional-tester | sed s/REPLACE_ME_GO_VERSION/$(_GO_VERSION)/ \


Just FYI, sed -i.bak 's|TEMPLATE_VAR|$(REPLACEMENT)|g' filename also works and is slightly more compact. Not sure if we care.

jpbetz · 2018-01-19T20:57:41Z

tools/functional-tester/README.md

+Or
+
+```bash
+$ rm -rf `pwd`/agent-1 && mkdir -p `pwd`/agent-1 && docker run \


This is quite a bit of script-style code. Should we maintain it in .sh files somewhere instead of in the README and reference the scripts from the README? Same for below.

jpbetz · 2018-01-19T21:06:55Z

tools/etcd-test-proxy/main.go

+var httpPort int
+var verbose bool
+
+func main() {


Do we care about TLS? Maybe future work?

jpbetz · 2018-01-19T21:08:07Z

tools/etcd-test-proxy/main.go

+	flag.StringVar(&to, "to", "localhost:2379", "Address URL to forward.")
+	flag.IntVar(&httpPort, "http-port", 2378, "Port to serve etcd-test-proxy API.")
+	flag.BoolVar(&verbose, "verbose", false, "'true' to run proxy in verbose mode.")
+	flag.Parse()


Add a short program description that is output with help?

gyuho · 2018-01-24T23:31:14Z

@jpbetz All fixed. PTAL.

Thanks!

Signed-off-by: Gyuho Lee <gyuhox@gmail.com>

spzala · 2018-02-05T17:46:17Z

@gyuho tons of changes :) ..looks good to me but I am gonna try to test using the readme you have provided, if that's helpful. Thanks!

gyuho · 2018-02-05T21:16:56Z

@spzala Easiest way to run is PASSES=functional ./test.

jpbetz

lgtm

Thanks!

spzala · 2018-02-06T20:19:29Z

@gyuho just fyi, I have run the test successfully in my env. Thanks!

....
Finished 'functional' pass at Tue Feb  6 20:16:33 UTC 2018
Success

gyuho requested a review from xiang90 January 2, 2018 18:09

gyuho mentioned this pull request Jan 12, 2018

grpc keepalive: test server-to-client HTTP/2 pings #8645

Closed

gyuho mentioned this pull request Jan 16, 2018

Make functional-tester deploy easy #9152

Closed

jpbetz suggested changes Jan 19, 2018

View reviewed changes

gyuho added the WIP label Jan 19, 2018

gyuho force-pushed the network-fault-test-with-proxy branch from 2e370f0 to 0915c00 Compare January 24, 2018 22:15

gyuho removed the WIP label Jan 24, 2018

gyuho added 15 commits January 24, 2018 17:37

pkg/transport: add "fixtures" for TLS tests

114a777

Signed-off-by: Gyuho Lee <gyuhox@gmail.com>

pkg/transport: implement "Proxy"

ab486e5

Signed-off-by: Gyuho Lee <gyuhox@gmail.com>

etcd-test-proxy: initial commit

849f88e

Signed-off-by: Gyuho Lee <gyuhox@gmail.com>

hack/scripts-dev: add "build-etcd-test-proxy" to Makefile

27d938f

Signed-off-by: Gyuho Lee <gyuhox@gmail.com>

functional-tester: remove old assets

8bba536

Signed-off-by: Gyuho Lee <gyuhox@gmail.com>

etcd-agent: remove "use-root"

affcaef

Signed-off-by: Gyuho Lee <gyuhox@gmail.com>

test: remove "use-root" in functional_pass

42c90ff

Signed-off-by: Gyuho Lee <gyuhox@gmail.com>

etcd-agent: use "pkg/transport.Proxy"

a912ddc

Signed-off-by: Gyuho Lee <gyuhox@gmail.com>

etcd-tester: set advertise ports, delay w/ network faults

a90695f

Signed-off-by: Gyuho Lee <gyuhox@gmail.com>

test: configure advertise ports in functional_pass

5960c75

Signed-off-by: Gyuho Lee <gyuhox@gmail.com>

gitignore: add ".Dockerfile-functional-tester"

0ca37a0

Signed-off-by: Gyuho Lee <gyuhox@gmail.com>

Dockerfile-functional-tester: initial commit

e33ef34

Signed-off-by: Gyuho Lee <gyuhox@gmail.com>

hack/scripts-dev: add "build-docker-functional-tester" to Makefile

e8242dd

Signed-off-by: Gyuho Lee <gyuhox@gmail.com>

hack/scripts-dev: clean up "Makefile"

1f58e72

Signed-off-by: Gyuho Lee <gyuhox@gmail.com>

tools/functional-tester: update README for local docker testing

c61b660

Signed-off-by: Gyuho Lee <gyuhox@gmail.com>

gyuho force-pushed the network-fault-test-with-proxy branch from 0915c00 to c61b660 Compare January 25, 2018 01:37

etcd-io deleted a comment from codecov-io Jan 25, 2018

jpbetz approved these changes Feb 6, 2018

View reviewed changes

gyuho merged commit 3bd0114 into etcd-io:master Feb 6, 2018

gyuho deleted the network-fault-test-with-proxy branch February 6, 2018 17:51

gyuho mentioned this pull request Feb 23, 2018

*: configure Raft Pre-Vote to reduce disruptive rejoining servers #9352

Merged

gyuho mentioned this pull request Apr 5, 2018

functional-tester: run network fault tests with snapshot trigger #9530

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: run network fault tests with proxy #9081

*: run network fault tests with proxy #9081

gyuho commented Jan 2, 2018 •

edited

Loading

gyuho commented Jan 12, 2018 •

edited

Loading

spzala commented Jan 14, 2018

gyuho commented Jan 19, 2018

jpbetz left a comment

jpbetz Jan 19, 2018

jpbetz Jan 19, 2018 •

edited

Loading

jpbetz Jan 19, 2018 •

edited

Loading

jpbetz Jan 19, 2018

jpbetz Jan 19, 2018

gyuho commented Jan 24, 2018

spzala commented Feb 5, 2018

gyuho commented Feb 5, 2018

jpbetz left a comment

spzala commented Feb 6, 2018 •

edited

Loading

*: run network fault tests with proxy #9081

*: run network fault tests with proxy #9081

Conversation

gyuho commented Jan 2, 2018 • edited Loading

Problem

Solution (what this PR does)

Other Changes

Future Work

gyuho commented Jan 12, 2018 • edited Loading

spzala commented Jan 14, 2018

gyuho commented Jan 19, 2018

jpbetz left a comment

Choose a reason for hiding this comment

jpbetz Jan 19, 2018

Choose a reason for hiding this comment

jpbetz Jan 19, 2018 • edited Loading

Choose a reason for hiding this comment

jpbetz Jan 19, 2018 • edited Loading

Choose a reason for hiding this comment

jpbetz Jan 19, 2018

Choose a reason for hiding this comment

jpbetz Jan 19, 2018

Choose a reason for hiding this comment

gyuho commented Jan 24, 2018

spzala commented Feb 5, 2018

gyuho commented Feb 5, 2018

jpbetz left a comment

Choose a reason for hiding this comment

spzala commented Feb 6, 2018 • edited Loading

gyuho commented Jan 2, 2018 •

edited

Loading

gyuho commented Jan 12, 2018 •

edited

Loading

jpbetz Jan 19, 2018 •

edited

Loading

jpbetz Jan 19, 2018 •

edited

Loading

spzala commented Feb 6, 2018 •

edited

Loading