Revert "kubeadm: Create control plane with ClusterFirstWithHostNet dns policy"

[neolit123]

Reverts #68890

This is causing issues as described in the linked issue:
fixes kubernetes/kubeadm#1236

@kubernetes/sig-cluster-lifecycle-pr-reviews
/priority critical-urgent
/kind bug
/assign @timothysc @chuckha
/hold

[k8s-ci-robot]

@neolit123: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[neolit123]

/release-note-none

[timothysc]

/lgtm
/approve

/cc @andrewrynhard - we needed to revert this!

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: neolit123, timothysc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• cmd/kubeadm/OWNERS [timothysc]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[neolit123]

/hold cancel

[andrewrynhard]

@timothysc I see. Sorry for the troubles. I would still like to see DNSClusterFirstWithHostNet used. Any ideas here? I would gladly submit the PR. We could put DNSClusterFirstWithHostNet behind a flag/config option.

[mauilion]

@andrewrynhard Since oidc url has to be https anyway in tectonic we did this by standing up an ingress controller and configuring that as part of our install. Being able to reference internal svc urls for this stuff is kind of brittle.

[andrewrynhard]

@mauilion Thanks for the suggestion. I actually ran it like that for quite some time. In my current use case I am using cert-manager and some rewrite rules in CoreDNS:

rewrite stop {
    name regex dex.example.com dex.security.svc.cluster.local
    answer name dex.security.svc.cluster.local dex.example.com
}

I am able to handle OIDC using only a service. It has worked out well so far. I can get valid certs for my internal service, but this whole setup is dependent on DNSClusterFirstWithHostNet.

[mauilion]

nice! this is a neat idea. This is a hard thing to solve since the apiserver comes in so early in the process. The solution to this is to ensure that the underlying node uses some dns server with dns stub capability or to ensure that before the apiserver starts a hosts entry like

10.96.0.100 dex.example.com
is added to the /etc/hosts file on the master nodes.
You must do this before the apiserver is started cause on start the apiserver will receive a copy of the /etc/hosts file of the node.

TL;DR it's better to solve this problem without ClusterFirstWithHostNet because of the circular dependency.

Another possible and interesting solution would be to extend kubeadm to populate hostAliases on the control plane.
This would enable you to set the /etc/hosts file for the apiserver for the endpoint directly I can see how this would be useful for a number of edge cases.

as an example here is a modified /etc/kubernetes/manifests/kube-apiserver.yaml

~~ SNIP ~~
spec:
  hostAliases:
  - ip: "10.96.0.100"
    hostnames:
    - "dex.example.org"
  containers:
  - command:
    - kube-apiserver
    - --authorization-mode=Node,RBAC
    - --feature-gates=MountPropagation=true
~~ SNIP ~~

This results in a /etc/hosts on the apiserver pod that can resolv dex.example.org without resorting to ClusterFirstWithHostNet

[andrewrynhard]

@mauilion Awesome, those suggestions look like some potentially good alternatives, but I don't have a way of know the IP of the internal dex service.

[mauilion]

You can predefine it. You can specify the cluster ip when defining the Dex service.
Example:

kubectl create service clusterip dex --tcp=3000 --clusterip=10.96.0.5  -l k8s-app=dex -n kube-system --dry-run -o yaml

[hansenwg]

@andrewrynhard Did you find a way to resolve this? I have a similiar scenario where I want to extend kube-apiserver by using the webhook with a service. However, DNS cannot resolve the webhook.ns and I could not fix an IP address due to potential conflicts

[andrewrynhard]

@hansenwg I kinda gave up on the idea unfortunately because I got pulled into other things, but I still would like to run it this way.

[hansenwg]

@andrewrynhard Thanks for the reply, I will look for some other workarounds then, thanks anyways!