fix mixed protocol issue for azure load balancer

[andyzhangx]

What type of PR is this?
/kind bug

What this PR does / why we need it:
fix mixed protocol issue for azure load balancer, with below config (service.beta.kubernetes.io/azure-load-balancer-mixed-protocols: "true"), azure provider will create both TCP and UDP rules for the service.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-mixed-protocols: "true"
  name: web
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: web
  sessionAffinity: None
  type: LoadBalancer

With this PR, you could see below both TCP and UDP rules are created for the service:

Which issue(s) this PR fixes:

Fixes #73849

Special notes for your reviewer:
Original PR(#67986) is not completed, I have no idea why I submitted a non-completed PR at that time...

Does this PR introduce a user-facing change?:

fix mixed protocol issue for azure load balancer

/kind bug
/assign @feiskyer
/priority important-soon
/sig azure

[justaugustus]

/test pull-kubernetes-e2e-gce

[feiskyer]

/test pull-kubernetes-e2e-aks-engine-azure

[feiskyer]

The changes LGTM, but the tests have failed. Let's fix the test and run again.

[andyzhangx]

@ritazh are you fixing this pull-kubernetes-e2e-aks-engine-azure testing issue?

W0219 01:54:20.056] 2019/02/19 01:54:20 process.go:155: Step 'bash -c . hack/lib/version.sh && KUBE_ROOT=. kube::version::get_version_vars && echo "${KUBE_GIT_VERSION-}"' finished in 7.243855407s
W0219 01:54:20.057] 2019/02/19 01:54:20 main.go:297: Something went wrong: failed to acquire k8s binaries: error starting /go/src/k8s.io/release/push-build.sh --nomock --verbose --noupdatelatest --bucket=kubernetes-release-pull --ci --gcs-suffix=/pull-kubernetes-e2e-aks-engine-azure --allow-dup: fork/exec /go/src/k8s.io/release/push-build.sh: no such file or directory
W0219 01:54:20.060] Traceback (most recent call last):
W0219 01:54:20.060]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 764, in <module>
W0219 01:54:20.090]     main(parse_args())
W0219 01:54:20.090]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 615, in main
W0219 01:54:20.091]     mode.start(runner_args)
W0219 01:54:20.091]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 262, in start
W0219 01:54:20.091]     check_env(env, self.command, *args)
W0219 01:54:20.091]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_e2e.py", line 111, in check_env
W0219 01:54:20.091]     subprocess.check_call(cmd, env=env)
W0219 01:54:20.091]   File "/usr/lib/python2.7/subprocess.py", line 186, in check_call
W0219 01:54:20.129]     raise CalledProcessError(retcode, cmd)

[feiskyer]

Refer kubernetes/test-infra#11355

[ritazh]

/test pull-kubernetes-e2e-aks-engine-azure

[justaugustus]

@andyzhangx @ritazh --

W0219 19:59:34.733] 2019/02/19 19:59:34 main.go:297: Something went wrong: starting e2e cluster: error building hyperkube error reading docker passoword file : open : no such file or directory.

[ritazh]

Here is a fix for that issue: kubernetes/test-infra#11378

[feiskyer]

/milestone 1.14

[k8s-ci-robot]

@feiskyer: The provided milestone is not valid for this repository. Milestones in this repository: [next-candidate, v1.10, v1.11, v1.12, v1.13, v1.14, v1.15, v1.16, v1.17, v1.18, v1.4, v1.5, v1.6, v1.7, v1.8, v1.9]

Use /milestone clear to clear the milestone.

In response to this:

/milestone 1.14

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[feiskyer]

/milestone v1.14

[andyzhangx]

/test pull-kubernetes-e2e-aks-engine-azure

[ritazh]

@andyzhangx the job won't work yet until a new image of kubekins-e2e is built and we need to update the job config after that

[andyzhangx]

/test pull-kubernetes-e2e-aks-engine-azure

[andyzhangx]

@ritazh the test error is different now, only 2 failed / 224 succeeded

test/e2e/framework/framework.go:704
Timed out after 60.000s.
Expected
    <*errors.errorString | 0xc0016eef10>: {
        s: "expected state to be terminated. Got pod status: {Phase:Pending Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-02-21 06:49:04 +0000 UTC Reason: Message:} {Type:Ready Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-02-21 06:49:04 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [bin-falsec6f69dbe-35a4-11e9-b498-2e54fb86f041]} {Type:ContainersReady Status:False LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-02-21 06:49:04 +0000 UTC Reason:ContainersNotReady Message:containers with unready status: [bin-falsec6f69dbe-35a4-11e9-b498-2e54fb86f041]} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2019-02-21 06:49:04 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:10.240.0.65 PodIP: StartTime:2019-02-21 06:49:04 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:bin-falsec6f69dbe-35a4-11e9-b498-2e54fb86f041 State:{Waiting:&ContainerStateWaiting{Reason:ContainerCreating,Message:,} Running:nil Terminated:nil} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:false RestartCount:0 Image:docker.io/library/busybox:1.29 ImageID: ContainerID:}] QOSClass:BestEffort}",
    }
to be nil
test/e2e/common/kubelet.go:123

[ritazh]

yes we are seeing flaky test failures

[feiskyer]

Thanks @ritazh @andyzhangx. The test failures are actually not related with this PR. Let's get this in first.

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andyzhangx, feiskyer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/cloudprovider/providers/azure/OWNERS [andyzhangx,feiskyer]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

[hargrave81]

I'm a n00b, how do i get this feature to work with my AKS?

[andyzhangx]

@hargrave81
This PR is cherry picked into following k8s versions:

v1.12.7
v1.13.4
v1.14.0

And AKS v1.12.7 is now available, you could try using the following example on v1.12.7:

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-mixed-protocols: "true"
  name: web
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: web
  sessionAffinity: None
  type: LoadBalancer

[mattkeeler]

Unless I'm missing something, the example above will indeed create an Azure load balancer with TCP and UDP rules for port 80, but Kubernetes itself will only expose the TCP port. I have yet to get UDP traffic working. #75831 appears to be the patch that's needed here.

[feiskyer]

@mattkeeler You're right, kube-proxy only opens TCP port for service for the above example.

[MPV]

KEP progress is being shared in #23880.