-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
More extensibility to CoreDNS configmap #462
Comments
Maybe it would be possible to allow some usage of |
Or maybe the system could allow defining a custom name for the CoreDNS ConfigMap? This way the default configmap could stay in the system and if custom configs are needed, the admin could just create a new ConfigMap and point the configuration to load the Corefile from there? |
Turns out there's (a bit hacky) way to have custom CoreDNS configs: the manifests in Question for Rancher Labs: can we rely on this behaviour and what is their vision of the optimal way to do this? |
I was going to say, that might not leave us in much of a better spot, but since it's just the CoreDNS ConfigMap, I suppose that could work. At least for the time being. |
We could probably use the import plugin to achieve this, am doing something similar with hosts on k3s where the current coredns configmap looks like the following: apiVersion: v1
data:
Corefile: |
.:53 {
errors
health
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
upstream
fallthrough in-addr.arpa ip6.arpa
}
hosts /etc/coredns/NodeHosts {
reload 1s
fallthrough
}
prometheus :9153
proxy . /etc/resolv.conf
cache 30
loop
reload
loadbalance
}
NodeHosts: |
10.135.135.100 k3s |
But in general manifest stuff could probably be improved, would depend on a properly named patch. |
The main issue with import is it doesn't really support graceful reloading so would require restarting coredns or updating Corefile. |
Related: coredns/coredns/issues/2633 |
@jait I don't believe it's reliable behavior. I've got an Ansible task that exports the config itself from the k8s resource via kubectl and writes it to |
@erikwilson have you made any progress on your PR here? erikwilson/coredns#1 |
Are there any news on this? The workaround mentioned above seems to work (for me) right now, but apparently it's a rather unstable situation. |
I opened a PR with coredns, please give it a thumbs up if it looks good: coredns/coredns#3068, might help to move it along. |
@erikwilson looks like your PR was merged, what do we need to do to get that work leveraged here? I'm willing to help with whatever I can. |
Another note on this. Even if you override the configmap, when you join a node to the cluster it overwrites the configmap with a node list. Open to any suggestions here. |
anyone has better solution. |
We are having the same issue here. Does anyone know what's causing the coredns manifest file to override on restart? |
All packaged components are re-deployed whenever a server node is restarted. If you want to replace coredns or any other component with your own, you can --disable it. |
@bradtopol .... I do not want to disable the coredns plugin, I just want to be able to save changes made to the configmap for multi cluster scenario's. What is your advise to achieve this? |
Could it be simpler if instead of generating a single ConfigMap containing both With this approach, it could be possible to adopt either of these solutions:
|
Maybe it's possible to do it similar to Azure Kubernetes? The coredns configuration on Azure looks like
and it mounts a config map with fixed name coredns-custom to folder custom. Depending on the files in the configmap *.override or *.server you are able to do a lot of customizings. Some examples are explained here: |
This feature would be great as we have an internal dns server set up to handle internal traffic and we don't want to foward all requests to this server. It seems a part of the feature is in the next release? |
Validations steps: Once server is ready, deployt this:
|
I see Based on @johgoe's comment and the corresponding documentation linked in this thread, the Azure implementation supports all coredns plugins via |
Due to the way docker works in non-native platforms, It is very hard to have a consistent experience across all platforms as we can't just use the [docker bridge netwrok IP's in non-native platforms](https://docs.docker.com/desktop/networking/). This means that users have to search their Host IP, and use It to get up and working [which we tried, but understand that it's not a good UX](https://github.com/gitpod-io/website/pull/2349). But users can use `127-0-0-1.nip.io` as the DOMAIN which resolves to `127.0.0.1` and is available in all platforms as its `localhost`. This works well and good for all user communication but internal communication fails as `127-0-0-1.nip.io` for them is something else. So, This PR fixes that by adding new coredns `gitpod.db` coredns config essentially asking to route all `127-0-0-1.nip.io` to `proxy.default.svc.cluster.local`. [As k3s does not yet support overriding coredns config in a sane-way](k3s-io/k3s#462) ,We instead skip the default coredns by adding `coredns.yaml.skip` file, and adding our own `custom-coredns.yaml` which is just plain `coredns.yaml` that comes with `k3s`, added with gitpod config. Signed-off-by: Tarun Pothulapati <tarun@gitpod.io>
Due to the way docker works in non-native platforms, It is very hard to have a consistent experience across all platforms as we can't just use the [docker bridge netwrok IP's in non-native platforms](https://docs.docker.com/desktop/networking/). This means that users have to search their Host IP, and use It to get up and working [which we tried, but understand that it's not a good UX](https://github.com/gitpod-io/website/pull/2349). But users can use `127-0-0-1.nip.io` as the DOMAIN which resolves to `127.0.0.1` and is available in all platforms as its `localhost`. This works well and good for all user communication but internal communication fails as `127-0-0-1.nip.io` for them is something else. So, This PR fixes that by adding new coredns `gitpod.db` coredns config essentially asking to route all `127-0-0-1.nip.io` to `proxy.default.svc.cluster.local`. [As k3s does not yet support overriding coredns config in a sane-way](k3s-io/k3s#462) ,We instead skip the default coredns by adding `coredns.yaml.skip` file, and adding our own `custom-coredns.yaml` which is just plain `coredns.yaml` that comes with `k3s`, added with gitpod config. Signed-off-by: Tarun Pothulapati <tarun@gitpod.io>
Due to the way docker works in non-native platforms, It is very hard to have a consistent experience across all platforms as we can't just use the [docker bridge netwrok IP's in non-native platforms](https://docs.docker.com/desktop/networking/). This means that users have to search their Host IP, and use It to get up and working [which we tried, but understand that it's not a good UX](https://github.com/gitpod-io/website/pull/2349). But users can use `127-0-0-1.nip.io` as the DOMAIN which resolves to `127.0.0.1` and is available in all platforms as its `localhost`. This works well and good for all user communication but internal communication fails as `127-0-0-1.nip.io` for them is something else. So, This PR fixes that by adding new coredns `gitpod.db` coredns config essentially asking to route all `127-0-0-1.nip.io` to `proxy.default.svc.cluster.local`. [As k3s does not yet support overriding coredns config in a sane-way](k3s-io/k3s#462) ,We instead skip the default coredns by adding `coredns.yaml.skip` file, and adding our own `custom-coredns.yaml` which is just plain `coredns.yaml` that comes with `k3s`, added with gitpod config. Signed-off-by: Tarun Pothulapati <tarun@gitpod.io>
Due to the way docker works in non-native platforms, It is very hard to have a consistent experience across all platforms as we can't just use the [docker bridge netwrok IP's in non-native platforms](https://docs.docker.com/desktop/networking/). This means that users have to search their Host IP, and use It to get up and working [which we tried, but understand that it's not a good UX](https://github.com/gitpod-io/website/pull/2349). But users can use `127-0-0-1.nip.io` as the DOMAIN which resolves to `127.0.0.1` and is available in all platforms as its `localhost`. This works well and good for all user communication but internal communication fails as `127-0-0-1.nip.io` for them is something else. So, This PR fixes that by adding new coredns `gitpod.db` coredns config essentially asking to route all `127-0-0-1.nip.io` to `proxy.default.svc.cluster.local`. [As k3s does not yet support overriding coredns config in a sane-way](k3s-io/k3s#462) ,We instead skip the default coredns by adding `coredns.yaml.skip` file, and adding our own `custom-coredns.yaml` which is just plain `coredns.yaml` that comes with `k3s`, added with gitpod config. Signed-off-by: Tarun Pothulapati <tarun@gitpod.io>
Due to the way docker works in non-native platforms, It is very hard to have a consistent experience across all platforms as we can't just use the [docker bridge netwrok IP's in non-native platforms](https://docs.docker.com/desktop/networking/). This means that users have to search their Host IP, and use It to get up and working [which we tried, but understand that it's not a good UX](https://github.com/gitpod-io/website/pull/2349). But users can use `127-0-0-1.nip.io` as the DOMAIN which resolves to `127.0.0.1` and is available in all platforms as its `localhost`. This works well and good for all user communication but internal communication fails as `127-0-0-1.nip.io` for them is something else. So, This PR fixes that by adding new coredns `gitpod.db` coredns config essentially asking to route all `127-0-0-1.nip.io` to `proxy.default.svc.cluster.local`. [As k3s does not yet support overriding coredns config in a sane-way](k3s-io/k3s#462) ,We instead skip the default coredns by adding `coredns.yaml.skip` file, and adding our own `custom-coredns.yaml` which is just plain `coredns.yaml` that comes with `k3s`, added with gitpod config. Signed-off-by: Tarun Pothulapati <tarun@gitpod.io>
Due to the way docker works in non-native platforms, It is very hard to have a consistent experience across all platforms as we can't just use the [docker bridge netwrok IP's in non-native platforms](https://docs.docker.com/desktop/networking/). This means that users have to search their Host IP, and use It to get up and working [which we tried, but understand that it's not a good UX](https://github.com/gitpod-io/website/pull/2349). But users can use `127-0-0-1.nip.io` as the DOMAIN which resolves to `127.0.0.1` and is available in all platforms as its `localhost`. This works well and good for all user communication but internal communication fails as `127-0-0-1.nip.io` for them is something else. So, This PR fixes that by adding new coredns `gitpod.db` coredns config essentially asking to route all `127-0-0-1.nip.io` to `proxy.default.svc.cluster.local`. [As k3s does not yet support overriding coredns config in a sane-way](k3s-io/k3s#462) ,We instead skip the default coredns by adding `coredns.yaml.skip` file, and adding our own `custom-coredns.yaml` which is just plain `coredns.yaml` that comes with `k3s`, added with gitpod config. Signed-off-by: Tarun Pothulapati <tarun@gitpod.io>
Due to the way docker works in non-native platforms, It is very hard to have a consistent experience across all platforms as we can't just use the [docker bridge netwrok IP's in non-native platforms](https://docs.docker.com/desktop/networking/). This means that users have to search their Host IP, and use It to get up and working [which we tried, but understand that it's not a good UX](https://github.com/gitpod-io/website/pull/2349). But users can use `127-0-0-1.nip.io` as the DOMAIN which resolves to `127.0.0.1` and is available in all platforms as its `localhost`. This works well and good for all user communication but internal communication fails as `127-0-0-1.nip.io` for them is something else. So, This PR fixes that by adding new coredns `gitpod.db` coredns config essentially asking to route all `127-0-0-1.nip.io` to `proxy.default.svc.cluster.local`. [As k3s does not yet support overriding coredns config in a sane-way](k3s-io/k3s#462) ,We instead skip the default coredns by adding `coredns.yaml.skip` file, and adding our own `custom-coredns.yaml` which is just plain `coredns.yaml` that comes with `k3s`, added with gitpod config. Signed-off-by: Tarun Pothulapati <tarun@gitpod.io>
Due to the way docker works in non-native platforms, It is very hard to have a consistent experience across all platforms as we can't just use the [docker bridge netwrok IP's in non-native platforms](https://docs.docker.com/desktop/networking/). This means that users have to search their Host IP, and use It to get up and working [which we tried, but understand that it's not a good UX](https://github.com/gitpod-io/website/pull/2349). But users can use `127-0-0-1.nip.io` as the DOMAIN which resolves to `127.0.0.1` and is available in all platforms as its `localhost`. This works well and good for all user communication but internal communication fails as `127-0-0-1.nip.io` for them is something else. So, This PR fixes that by adding new coredns `gitpod.db` coredns config essentially asking to route all `127-0-0-1.nip.io` to `proxy.default.svc.cluster.local`. [As k3s does not yet support overriding coredns config in a sane-way](k3s-io/k3s#462) ,We instead skip the default coredns by adding `coredns.yaml.skip` file, and adding our own `custom-coredns.yaml` which is just plain `coredns.yaml` that comes with `k3s`, added with gitpod config. Signed-off-by: Tarun Pothulapati <tarun@gitpod.io>
Is your feature request related to a problem? Please describe.
We have a desire to point requests from a specific domain (that doesn't line up with the current cluster domain schema) to the ingress in order to route requests from a web head pod to another pod on the same (single) node. For instance:
(webhead)
web.dev.be.lan
has an env var pointing to (other pod on same node)search.dev.be.lan
.We currently modify the rendered/written manifest at
To resemble:
The problem is, on k3s server start, this file gets overwritten:
https://github.com/rancher/k3s/blob/master/pkg/server/server.go#L133
Describe the solution you'd like
Ideally, there would be a way to augment the config map in a clean way so that when the file is updated/overwritten on server start, the changes persist.
Describe alternatives you've considered
Another option I considered was copying the CoreDNS manifest into our repo, adding the line, and specifying
--no-deploy coredns
but I didn't realize there were template variables in the file:%{CLUSTER_DOMAIN}%
and%{CLUSTER_DNS}%
so I'm reluctant to go the route as cluster DNS is very likely to not be consistent between environments/nodes.I'm not sure if there's another route I'm missing to either overwrite the config or reference it in another spot to survive restarts, but I'm definitely open to testing any solutions and experimenting a bit. Thank you for all the hard work on this project, it's great having a lightweight solution like k3s!!
The text was updated successfully, but these errors were encountered: