Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

directly query dns and resolve it in the e2es for windows #97987

Merged
merged 1 commit into from Jan 15, 2021
Merged

directly query dns and resolve it in the e2es for windows #97987

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
99 changes: 70 additions & 29 deletions test/e2e/windows/dns.go
View file
Open in desktop
Expand Up @@ -18,7 +18,6 @@ package windows

import (
"context"
"regexp"
"strings"

v1 "k8s.io/api/core/v1"
Expand All @@ -37,58 +36,100 @@ var _ = SIGDescribe("[Feature:Windows] DNS", func() {
})

f := framework.NewDefaultFramework("dns")

ginkgo.It("should support configurable pod DNS servers", func() {

ginkgo.By("Getting the IP address of the internal Kubernetes service")

svc, err := f.ClientSet.CoreV1().Services("kube-system").Get(context.TODO(), "kube-dns", metav1.GetOptions{})
framework.ExpectNoError(err)

ginkgo.By("Preparing a test DNS service with injected DNS names...")
testInjectedIP := "1.1.1.1"
testSearchPath := "resolv.conf.local"

ginkgo.By("Creating a pod with dnsPolicy=None and customized dnsConfig...")
testUtilsPod := e2epod.NewAgnhostPod(f.Namespace.Name, "e2e-dns-utils", nil, nil, nil)
testUtilsPod.Spec.DNSPolicy = v1.DNSNone
testUtilsPod.Spec.DNSConfig = &v1.PodDNSConfig{
Nameservers: []string{testInjectedIP},
// the default service IP will vary from cluster to cluster, but will always be present and is a good DNS test target
testInjectedIP := svc.Spec.ClusterIP
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the default DNS for a pod is the kube-dns pod, if something isn't working with the custom DNS would these tests pass?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tests will fail if there is no valid DNS service. I figured thats actually a requirement - we can't test that we support configurable pod DNS unless we actually try to hit a named endpoint .

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any other test, which scans an implementation detail, wont actually measure the functionality, but rather, will just be an guess based on a heurisitc.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am wondering on the use of the kube-dns as the endpoint though. If the configuration is a noop for CNI's that don't implement the dns plugin then the default dns would be the cluster DNS (kube-dns) wouldn't it? That would mean they would pass the test even though they didn't really configure the DNS? I think we would need to use something like 1.1.1.1 to make sure custom DNS is working.

Agreed on actually hitting the endpoint. If the above is true then I think we should also keep the check that the DNS is actually set to what we expected and hit the endpoint (to make sure we are using the correct DNS).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, i see so you mean you could just 'get lucky' here, in cases where the dns isnt being overwritten

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good idea !

  • Confirmed that no-op CNIs, like EKS, indeed DO NOT by default resolve kubernetes on windows .... So were all good there... but ANYWAY, i stole your idea just bc its good to be extra careful :)
  • SO.... , just to be double careful, added a failure case, where 1.1.1.1 not being injected directly leads to failed test as well, that will weed out no-op DNS systems early on.
  • So now we have BOTH the 1.1.1.1 verification, as well as the other one. BTW the old regex seemed to not be restrictive enough, so i just did a regular string search, more readable and more precise.

results

  • On EKS, this test now FAILS FAST (because it notices that 1.1.1.1 is not injected)
  • On AKS w/ calico, the test passes .

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed that no-op CNIs, like EKS, indeed DO NOT by default resolve kubernetes on windows

Interesting. What is the default DNS configuration for EKS if not using kube-dns? On azure with flannel using no DNS pod spec I get:

ipconfig /all
  DNS Servers . . . . . . . . . . . : 10.96.0.10
   NetBIOS over Tcpip. . . . . . . . : Disabled
   Connection-specific DNS Suffix Search List :
                                       default.svc.cluster.local
                                       svc.cluster.local
                                       cluster.local

curl.exe -k https://kubernetes:443
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {

  },
  "code": 403
}

Where 10.96.0.10 is the kube-dns ip:

k get svc -n kube-system kube-dns              
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   20h

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

interestinggg
in that case its goood we kept the 1.1.1.1 testcase

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

Windows IP Configuration

   Host Name . . . . . . . . . . . . : windows-server-iis-675f9744d7-dr2qv
   Primary Dns Suffix  . . . . . . . :
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : omg.svc.cluster.local
                                       svc.cluster.local
                                       cluster.local

Ethernet adapter vEthernet (cid-5eae4daf98cf88ff695042b491c118892dcc853114f5353379c2b65c3e181d97):

   Connection-specific DNS Suffix  . : omg.svc.cluster.local
   Description . . . . . . . . . . . : Hyper-V Virtual Ethernet Adapter #5
   Physical Address. . . . . . . . . : 00-15-5D-AC-88-C9
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::b131:bcb8:c3fa:96db%34(Preferred)
   IPv4 Address. . . . . . . . . . . : 192.168.22.170(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.224.0
   Default Gateway . . . . . . . . . : 192.168.0.1
   DNS Servers . . . . . . . . . . . : 10.100.0.10
   NetBIOS over Tcpip. . . . . . . . : Disabled
   Connection-specific DNS Suffix Search List :
                                       omg.svc.cluster.local
                                       svc.cluster.local
                                       cluster.local```

thtas the default config of eks

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe its because your running in the default namespaces which is why you resolve kubernetews

otherwise, you wont be able to, hence why in general , the 2nd test is sufficient bc e2es allways run outside the default ns :)

testSearchPath := "default.svc.cluster.local"

ginkgo.By("Creating a windows pod with dnsPolicy=None and customized dnsConfig...")
testPod := e2epod.NewAgnhostPod(f.Namespace.Name, "e2e-dns-utils", nil, nil, nil)
testPod.Spec.DNSPolicy = v1.DNSNone
testPod.Spec.DNSConfig = &v1.PodDNSConfig{
Nameservers: []string{testInjectedIP, "1.1.1.1"},
Searches: []string{testSearchPath},
}
testUtilsPod.Spec.NodeSelector = map[string]string{
testPod.Spec.NodeSelector = map[string]string{
"kubernetes.io/os": "windows",
}
testUtilsPod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Create(context.TODO(), testUtilsPod, metav1.CreateOptions{})
testPod, err = f.ClientSet.CoreV1().Pods(f.Namespace.Name).Create(context.TODO(), testPod, metav1.CreateOptions{})
framework.ExpectNoError(err)
framework.Logf("Created pod %v", testUtilsPod)

ginkgo.By("confirming that the pod has a windows label")
jayunit100 marked this conversation as resolved.
Show resolved Hide resolved
framework.ExpectEqual(testPod.Spec.NodeSelector["kubernetes.io/os"], "windows")
framework.Logf("Created pod %v", testPod)
defer func() {
framework.Logf("Deleting pod %s...", testUtilsPod.Name)
if err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Delete(context.TODO(), testUtilsPod.Name, *metav1.NewDeleteOptions(0)); err != nil {
framework.Failf("Failed to delete pod %s: %v", testUtilsPod.Name, err)
framework.Logf("Deleting pod %s...", testPod.Name)
if err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Delete(context.TODO(), testPod.Name, *metav1.NewDeleteOptions(0)); err != nil {
framework.Failf("Failed to delete pod %s: %v", testPod.Name, err)
}
}()
framework.ExpectNoError(e2epod.WaitForPodNameRunningInNamespace(f.ClientSet, testUtilsPod.Name, f.Namespace.Name), "failed to wait for pod %s to be running", testUtilsPod.Name)
framework.ExpectNoError(e2epod.WaitForPodNameRunningInNamespace(f.ClientSet, testPod.Name, f.Namespace.Name), "failed to wait for pod %s to be running", testPod.Name)

ginkgo.By("Verifying customized DNS option is configured on pod...")
// This isn't the best 'test' but it is a great diagnostic, see later test for the 'real' test.
ginkgo.By("Calling ipconfig to get debugging info for this pod's DNS and confirm that a dns server 1.1.1.1 can be injected, along with ")
jayunit100 marked this conversation as resolved.
Show resolved Hide resolved
cmd := []string{"ipconfig", "/all"}
stdout, _, err := f.ExecWithOptions(framework.ExecOptions{
Command: cmd,
Namespace: f.Namespace.Name,
PodName: testUtilsPod.Name,
PodName: testPod.Name,
ContainerName: "agnhost-container",
CaptureStdout: true,
CaptureStderr: true,
})
framework.ExpectNoError(err)

framework.Logf("ipconfig /all:\n%s", stdout)
dnsRegex, err := regexp.Compile(`DNS Servers[\s*.]*:(\s*[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})+`)
framework.ExpectNoError(err)

if dnsRegex.MatchString(stdout) {
match := dnsRegex.FindString(stdout)
if !strings.Contains(stdout, "1.1.1.1") {
framework.Failf("One of the custom DNS options 1.1.1.1, not found in ipconfig /all")
}

if !strings.Contains(match, testInjectedIP) {
framework.Failf("customized DNS options not found in ipconfig /all, got: %s", match)
}
} else {
framework.Failf("cannot find DNS server info in ipconfig /all output: \n%s", stdout)
// We've now verified that the DNS stuff is injected... now lets make sure that curl'ing 'wrong' endpoints fails, i.e.
// a negative control, to run before we run our final test...

ginkgo.By("Verifying that curl queries FAIL for wrong URLs")

// the below tests use curl because nslookup doesnt seem to use ndots properly
// ideally we'd use the powershell native ResolveDns but, that is not a part of agnhost images (as of k8s 1.20)
// TODO @jayunit100 add ResolveHost to agn images

cmd = []string{"curl.exe", "-k", "https://kubernetezzzzzzzz:443"}
stdout, _, err = f.ExecWithOptions(framework.ExecOptions{
Command: cmd,
Namespace: f.Namespace.Name,
PodName: testPod.Name,
ContainerName: "agnhost-container",
CaptureStdout: true,
CaptureStderr: true,
})
if err == nil {
framework.Logf("Warning: Somehow the curl command succeeded... The output was \n %v", stdout)
framework.Failf("Expected a bogus URL query to fail - something is wrong with this test harness, cannot proceed.")
}

ginkgo.By("Verifying that injected dns records for 'kubernetes' resolve to the valid ip address")
cmd = []string{"curl.exe", "-k", "https://kubernetes:443"}
stdout, _, err = f.ExecWithOptions(framework.ExecOptions{
Command: cmd,
Namespace: f.Namespace.Name,
PodName: testPod.Name,
ContainerName: "agnhost-container",
CaptureStdout: true,
CaptureStderr: true,
})
framework.Logf("Result of curling the kubernetes service... (Failure ok, only testing for the sake of DNS resolution) %v ... error = %v", stdout, err)

// curl returns an error if the host isnt resolved, otherwise, it will return a passing result.
if err != nil {
framework.ExpectNoError(err)
}

// TODO: Add more test cases for other DNSPolicies.
})
})