remove trailing dots from the parsed searches from host resolv.conf

[fcgravalos]

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change
/kind bug
/kind cleanup
/kind design
/kind documentation
/kind failing-test

/kind feature

/kind flake

What this PR does / why we need it:
In short: This will make kubelet ignore trailing dots . in search lines from host resolv.conf files, so when omitDuplicates function gets called, is de-duplicated and we don't hit validation limits. Please

Long story: We provision our Debian stretch based clusters (1.15.3) with kubespray, which supersedes search lines to add svc.clusterFqdn and default.svc.clusterFqdn to the host resolvconf conifg file (/etc/resolv.conf).

https://github.com/kubernetes-sigs/kubespray/blob/da50ed0936742bb633c6b48a84ffca98d8ab03ad/roles/kubernetes/preinstall/tasks/0040-set_facts.yml#L121

When a server restart, dhclient runs, adds trailing dots are added to those search lines.

When a pod with DNS policy ClusterFirst gets scheduled, kubelet will add svc.clusterFqdn and namespace.clusterFqdn to the pod resolv.conf and it will be merged with the host resolv.conf deduplicating entries and doing limit validations.

kubernetes/pkg/kubelet/network/dns/dns.go

Line 146 in 26cc580

func (c *Configurer) generateSearchesForDNSClusterFirst(hostSearch []string, pod *v1.Pod) []string {

kubernetes/pkg/kubelet/network/dns/dns.go

Line 85 in 26cc580

func omitDuplicates(strs []string) []string {

kubernetes/pkg/kubelet/network/dns/dns.go

Line 98 in 26cc580

func (c *Configurer) formDNSSearchFitsLimits(composedSearch []string, pod *v1.Pod) []string {

So, if the search line now contains trailing ., and you have things like svc.clusterFqdn. when scheduling a pod, svc.clusterFqdn and svc.clusterFqdn. are not deduplicated and you can hit one of the validation limits (MaxDNSSearchPaths) and the resulting line may be cut down to this limit, 6.

In our case it removed our search for services outside our kubernetes cluster, so dns request were not sent to upstream resolver and we had some timeouts affecting our production workload.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:
NOTE I'm not saying this is a bug. Adding trailing dots to search lines don't provide any values, but it is harmless, and I think it'd would be nice that kubelet can detect them and remove them so we don't have garbage in pod resolv.conf or we hit a validation limit.

Some facts:

• Kubernetes version: v1.15.3
• Kernel: 4.19.0-0.bpo.4-amd64 #1 SMP Debian 4.19.28-2~bpo9+1 (2019-03-27) x86_64 GNU/Linux
• Os release:

PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

• Cloud: NONE. bare-metal servers,
  Does this PR introduce a user-facing change?:

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[k8s-ci-robot]

Welcome @fcgravalos!

It looks like this is your first PR to kubernetes/kubernetes 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kubernetes has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

@fcgravalos: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

Hi @fcgravalos. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fcgravalos]

/sig network

[fcgravalos]

/release-note-none

[fcgravalos]

/assign @freehan

[aojea]

after this #74686 got merged maybe removing "all" the trailing dots in the search line is not the best solution because seems to change the behavior implemented in the PR mentioned previously.

However, based on the assumption that a domain with or without trailing dot in the search field is the same #74685 (comment), maybe we can consider as a duplicate a suffix with and without a trailing dot.

[fcgravalos]

after this #74686 got merged maybe removing "all" the trailing dots in the search line is not the best solution because seems to change the behavior implemented in the PR mentioned previously.

However, based on the assumption that a domain with or without trailing dot in the search field is the same #74685 (comment), maybe we can consider as a duplicate a suffix with and without a trailing dot.

Another option I considered was changin omitDuplicates to do so, but seems used for also server names, so I thought that it make sense when parsing the host resolv.conf

[thz]

after this #74686 got merged maybe removing "all" the trailing dots in the search line is not the best solution because seems to change the behavior implemented in the PR mentioned previously.

PR #74686 allows trailing dots in searches. It added the comment // it is fine to have a trailing dot and then removes the dot. This is okay because a trailing dot in a search list entry is just redundant. The search list entries always make the name to which they append to fully qualified (there is no second round of appending). So that trailing dot removal could be seen as a "normalization".

Removing all trailing dots in the searches is also just a "normalization" and not in conflict with #74686.

Although having a trailing dot in the searches is wrong already, it is good behavior to still accept and normalize (ignore) it.

[fcgravalos]

Hi @freehan !! 👋

Do you have an estimate about when this can be tested? I ran TestParseResolvConf locally and it did work, but I was curious about e2e tests as well

Thanks a lot!!

[mattjmcnaughton]

/ok-to-test
/lgtm

From my reading of the conversation, I think this looks good to me. Worth adding a comment explaining why we strip the "." (or alternatively extracting the code into a function with a descriptive name (i.e. removeRedundantTrailingPeriod)).

[fcgravalos]

/retest

[fcgravalos]

/retest

[fcgravalos]

tests failing because:

W1004 10:43:12.129] ERROR: (gcloud.compute.routers.create) Quota 'ROUTERS' exceeded. Limit: 10.0 globally.

I'll try again later

[johnbelamaric]

/retest
/lgtm

[fcgravalos]

It seems we're still hitting gcloud routers limits

[fcgravalos]

/retest

[fcgravalos]

Finally al tests passed @johnbelamaric :)

[johnbelamaric]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: fcgravalos, johnbelamaric

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/kubelet/network/dns/OWNERS [johnbelamaric]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[fcgravalos]

@johnbelamaric do you think it makes sense to cherry-pick this to recent releases (>=1.15)?

[johnbelamaric]

Oddly it's marked as kind/feature here, not kind/bug. But I would consider it a bug. It can certainly cause hard-to-identify DNS related production issues. So, I think it would makes sense to cherry pick.

[dannyk81]

@johnbelamaric is it possible to label it at this point as kind/bug? how would we go about with the cherry-pick now that it's merged?

[fcgravalos]

Certainly It is not a feature :). But on the other hand I was not sure of considering this a bug... I think I should have brought Up that discussion when I opened the PR.

[johnbelamaric]

/kind bug

[johnbelamaric]

/remove-kind feature

[johnbelamaric]

/priority important-soon

[johnbelamaric]

See https://github.com/kubernetes/community/blob/master/contributors/devel/sig-release/cherry-picks.md

[fcgravalos]

Thanks @johnbelamaric