add a iptables failure detector, default off

[lambdai]

Add a iptables failure detector in go version of iptables

By default it will watch the inbound traffic and recover the original port(15002).
If iptables are set correctly it should see the inbound port as 15001 or 15006 but the original port is 15002.
It should work in both ipv4 and ipv6
The client and server are both in the init container and send traffic to itself through host ip. So it require istio >=1.3 to make this tcp connection capturable.

Add --skip-rule-apply and --skip-validation
A little bit more on --dry-run

1. golden test: --dry-run --skip-rule-apply --skip-validation
2. istio-init (non-cni) : --skip-validation (as default)
3. kubelet at cni: --skip-validation (as default)
4. istio-init at cni: --skip-rule-apply --run-validation and run as uid or gid 1337

An example log of spawn another istio-validation preinit container

  - command:
    - istio-iptables
    - -p
    - "15001"
    - -z
    - "15006"
    - -u
    - "1337"
    - --skip-rule-apply
    - --run-validation
    securityContext:
      runAsNonRoot: true
      runAsUser: 1337
    image: gcr.io/lambdai-experiment/proxyv2:cni6.125
    imagePullPolicy: IfNotPresent
    name: istio-validation
    resources:
      limits:
        cpu: 100m
        memory: 50Mi
      requests:
        cpu: 10m
        memory: 10Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File

$ k logs `k get pods|grep prod|cut -d " " -f 1` -c istio-validation
in new validator: 10.12.1.51
Listening on 127.0.0.1:15001
Listening on 127.0.0.1:15006
Error connecting to 127.0.0.6:15002: dial tcp 127.0.0.1:0->127.0.0.6:15002: connect: connection refused
local addr 127.0.0.1:15006
original addr 127.0.0.1:15002
validation passed

[rlenglet]

This is not correct. You can't setup iptables when Istio CNI is enabled, because 1) this container won't run with CAP_NET_ADMIN, and 2) trying to initialize iptables a 2nd time after Istio CNI has already done it will fail.

So your validator has to be in another command, not in the root command.

[lambdai]

The idea is to reuse the configurator defined here.

Ideally

1. cni executes the iptable setup in the kubelet context
2. istio-init executes the validation in the preinit pod
3. for non-cni, run both

WDYT?

[rlenglet]

3. for non-cni, run both

Why? I don't understand why we would ever need to check an iptables setup that succeeded just before.

We would ever need to do one or the other. Which is why I recommended to do the validation in another command, to make that clear.

[lambdai]

I don't have strong opinion we must validate in non-cni.

But I'd like to ship iptables setup in one binary. Easy to maintain, and adding very little overhead to the binary size.
Isn't that a fact that istio-iptables is already running in both kubelet (for cni) context and istio-init(for non-cni)?

[rlenglet]

But I'd like to ship iptables setup in one binary.

I never asked for another binary.
I asked for another Cobra command.

[lambdai]

@rlenglet Thanks!
I add --skip-rule-apply and --skip-validation.
See the summary

[mandarjog]

can you update the doc with what you want to do?
I had suggested running http server with loop back on the doc because you cannot use iptables.

[lambdai]

@mandarjog I will update the doc.
Http is over killing here.
The goal is to detect if iptables setup is done.
Since we assume the iptables setup is either all or nothing, we are utilizing the fact that
If the pod send a tcp request to itself using host ip, the tcp request should be captured.
To detect it,

1. the validation server listen on :15006 (inbound capture port)
2. the validation connect to 10.X.Y.Z:15002
   the validation server should receive the request where the local address port is 15006 but the original port is 15002. This could never happen without the istio iptables.

Below is the log I got from validation server.

Listening on :15006
local addr  127.0.0.1:15006
original addr  10.12.2.25:15002

[lambdai]

@rlenglet What is the blocker for this PR?

I disabled default detector by default in istio-iptables binary, so the cni should see no behavior change.

The follow PRs will be utilizing it with complete e2e test

[rlenglet]

/test unit-tests_istio

[rlenglet]

The code LGTM.
This still needs approval from @istio/wg-config-maintainers and @istio/wg-test-and-release-maintainers Thanks!

[lambdai]

/retest integ-istioio-k8s-tests_istio

[lambdai]

/wait

[lambdai]

/retest integ-istioio-k8s-tests_istio

[lambdai]

@howardjohn it take a little bit more than 1 second. Actually the 1 second is the retry since the first request usually fail. I can cut the 1 second in the follow up PR.

$ time /usr/local/bin/istio-iptables --skip-rule-apply --run-validation ; done
in new validator: 10.12.1.50
Error connecting to 127.0.0.6:15002: dial tcp 127.0.0.1:0->127.0.0.6:15002: connect: connection refused
Listening on 127.0.0.1:15001
Listening on 127.0.0.1:15006
local addr 127.0.0.1:15006
original addr 127.0.0.1:15002
validation passed

real    0m1.101s
user    0m0.002s
sys     0m0.003s

[howardjohn]

1s is pretty high. Any time we add to startup is not just slower pod start , for knative it is actually service latency One of the benefits of cni is improved startup speed so I hope we don't regress too much

…

On Fri, Jan 10, 2020, 12:39 AM Yuchen Dai ***@***.***> wrote: @howardjohn <https://github.com/howardjohn> it take a little bit more than 1 second. Actually the 1 second is the retry since the first request usually fail. I can cut the 1 second in the follow up PR. $ time /usr/local/bin/istio-iptables --skip-rule-apply --run-validation ; done in new validator: 10.12.1.50 Error connecting to 127.0.0.6:15002: dial tcp 127.0.0.1:0->127.0.0.6:15002: connect: connection refused Listening on 127.0.0.1:15001 Listening on 127.0.0.1:15006 local addr 127.0.0.1:15006 original addr 127.0.0.1:15002 validation passed real 0m1.101s user 0m0.002s sys 0m0.003s — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#19534?email_source=notifications&email_token=AAEYGXPGNHQYMQH47HPT6GTQ5AX4JA5CNFSM4JZXTKGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEITDWJI#issuecomment-572930853>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXJGT3XE2L2EQUOTA6TQ5AX4JANCNFSM4JZXTKGA> .

[mandarjog]

Why does happy path take 1s?
If everything is setup correctly it should take no time at all.
We want to ensure that happy path is optimized, the failure path is not so important.

[rlenglet]

+1 to @mandarjog's comment. Why does this code ever try to open a connection before it has opened the server socket? Opening the server socket should be the first action.

[mandarjog]

Another unfortunate side effect of this issue (not the pr specifically) is that cni may now be slower than non cni. CNi solution will have the same number of moving parts including scheduling the init container plus the time to execute cni initialization.

[lambdai]

@mandarjog @rlenglet @howardjohn Added a barrier: validation client runs only when server is ready to serve.
Now the validation would pass in 0.1 second if it should pass
I assumed it's not critical in this PR since I am not ready to validate in non-cni case.

I flush some further changes (reuseAddr) in this PR. Let me know if you think it should be excluded.

[lambdai]

@howardjohn

1s is pretty high. Any time we add to startup is not just slower pod start , for knative it is actually service latency One of the benefits of cni is improved startup speed so I hope we don't regress too much
…
On Fri, Jan 10, 2020, 12:39 AM Yuchen Dai @.***> wrote: @howardjohn https://github.com/howardjohn it take a little bit more than 1 second. Actually the 1 second is the retry since the first request usually fail. I can cut the 1 second in the follow up PR. $ time /usr/local/bin/istio-iptables --skip-rule-apply --run-validation ; done in new validator: 10.12.1.50 Error connecting to 127.0.0.6:15002: dial tcp 127.0.0.1:0->127.0.0.6:15002: connect: connection refused Listening on 127.0.0.1:15001 Listening on 127.0.0.1:15006 local addr 127.0.0.1:15006 original addr 127.0.0.1:15002 validation passed real 0m1.101s user 0m0.002s sys 0m0.003s — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#19534?email_source=notifications&email_token=AAEYGXPGNHQYMQH47HPT6GTQ5AX4JA5CNFSM4JZXTKGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEITDWJI#issuecomment-572930853>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEYGXJGT3XE2L2EQUOTA6TQ5AX4JANCNFSM4JZXTKGA .

Because validation is default off so knative is not impacted if the don't use cni (Do they?)
Anyway the extra 1s is optimized out

[howardjohn]

People use CNI + knative as far as I know, I think I have seen some mentions of it before

[howardjohn]

I ran this locally. ran it a couple 100 times, it seems very stable. 100 runs consistently takes 20s, so .2s each run

[mandarjog]

/lgtm

[rlenglet]

/test unit-tests_istio