Bump ectd client and fix initializing IP/Port allocators when etcd is not reachable

[fgrzadkowski]

This PR reintroduces newer version of go-etcd client library (reverted in #9447). It also adds additional debugging to service allocator code to help debug potential failures that I was not able to reproduce locally.

@wojtek-t @smarterclayton
@saad-ali (currentl oncall)

NOTE 1: If you see other places where logging might be useful please add a comment.
NOTE 2: Please do not merge this PR. I will do this tomorrow morning so that I can closely watch jenkins and revert it once I gather more data for debugging.

[k8s-bot]

GCE e2e build/test failed for commit cee8c22ea2d794c42a3e369756c1a3d97a0b43c0.

• Build Log
• Test Artifacts
• Internal Jenkins Results

[wojtek-t]

As discussed offline - it seems that the Jenkins failure is exactly the reproduction of the problem.

[mnemotiv]

I don't know where it should be put and whether it's related, but it would be very helpful if while using --etcd-config (go-etcd) flag it would output any errors to kube-apiserver, because for now - if it fails - no logs are being shown.

[fgrzadkowski]

@k8s-bot test this please

[k8s-bot]

GCE e2e build/test failed for commit cee8c22ea2d794c42a3e369756c1a3d97a0b43c0.

• Build Log
• Test Artifacts
• Internal Jenkins Results

[fgrzadkowski]

@k8s-bot test this please

[k8s-bot]

GCE e2e build/test failed for commit cee8c22ea2d794c42a3e369756c1a3d97a0b43c0.

• Build Log
• Test Artifacts
• Internal Jenkins Results

[fgrzadkowski]

@k8s-bot test this please

[k8s-bot]

GCE e2e build/test failed for commit b677cef5030af461fd26544748817e352b38adf9.

• Build Log
• Test Artifacts
• Internal Jenkins Results

[k8s-bot]

GCE e2e build/test failed for commit 985cb1cb5450f3c4c0ccb619fd8924a4bd8193a2.

• Build Log
• Test Artifacts
• Internal Jenkins Results

[k8s-bot]

GCE e2e build/test failed for commit cbe7207bf59d860161b3ee611e9a679c42bf47a4.

• Build Log
• Test Artifacts
• Internal Jenkins Results

[fgrzadkowski]

@k8s-bot test this please

[k8s-bot]

GCE e2e build/test failed for commit cbe7207bf59d860161b3ee611e9a679c42bf47a4.

• Build Log
• Test Artifacts
• Internal Jenkins Results

[k8s-bot]

GCE e2e build/test failed for commit 333c40b2b15e95333de5f74054eea4f4e2528365.

• Build Log
• Test Artifacts
• Internal Jenkins Results

[k8s-bot]

GCE e2e build/test failed for commit c2bf8f5abe8f5c51701becc47c75b3dc018504d3.

• Build Log
• Test Artifacts
• Internal Jenkins Results

[k8s-bot]

GCE e2e build/test passed for commit b7032521728b4c2d28ad36d546a74812899cc923.

• Build Log
• Test Artifacts
• Internal Jenkins Results

[fgrzadkowski]

@k8s-bot test this please

[k8s-bot]

GCE e2e build/test passed for commit b7032521728b4c2d28ad36d546a74812899cc923.

• Build Log
• Test Artifacts
• Internal Jenkins Results

[fgrzadkowski]

This is ready for review.

Assigning to @smarterclayton since one of the commits fixes a bug in allocator code.

[k8s-bot]

GCE e2e build/test passed for commit d06461f119904efb4be7aae5573a7bba043d41cf.

• Build Log
• Test Artifacts
• Internal Jenkins Results

[fgrzadkowski]

@davidopp
Results from density test with 30 pods per node in 100 node cluster with n1-standard-4 master:

INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.99 Latency:1.605515s}
INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.9 Latency:1.270142s}
INFO: Top latency metric: {Verb:LIST Resource:pods Quantile:0.5 Latency:529.251ms}
INFO: Top latency metric: {Verb:POST Resource:bindings Quantile:0.99 Latency:444.361ms}
INFO: Top latency metric: {Verb:PUT Resource:pods Quantile:0.99 Latency:426.816ms}

This is way better (~25%) than the usual run for density test.

[k8s-bot]

GCE e2e build/test passed for commit 6ecba7b364d385f4d6f5e71de79cb75aecc53fed.

• Build Log
• Test Artifacts
• Internal Jenkins Results

[fgrzadkowski]

Shippable complains about hack/run-gendocs.sh but I run it locally and it was a no-op :/

@smarterclayton I can't find any of your comments (even those I've already addressed). Is there anything else I should fix with this PR? It'd be great to have it merged today.

[smarterclayton]

Had one comment that got last.

[smarterclayton]

LGTM but I'm not sure what the shippable failure is.

[k8s-bot]

GCE e2e build/test passed for commit 4c0c7dd.

• Build Log
• Test Artifacts
• Internal Jenkins Results

[satnam6502]

Issue? Risk level? Milestone? Does it need to be merged anyway?

[fgrzadkowski]

@satnam6502 This fixes #9444 and will improve performance by reducing cpu usage on master machine. Based on discussion with @davidopp we want to merge this change before v1.0.

I'd say that risk is medium-high due to multiple changes in etcd client library since our last update.

I've run e2e tests multiple times (using jenkins and locally) and it looks good.

[davidopp]

Thanks for merging. I think this merits being included in 1.0 since it substantially improves performance.

[ghost]

See #10146. There is quite possibly a bug in this PR - it appears to be causing recovery after an etcd failure not to work reliably.