Create ExperimentalClusterDNS feature flag

[justinsb]

This currently just turns off validation of the kubelet cluster dns
flag, which should allow for experimenting with more complicated DNS
configurations such as local proxies, which may address shortcomings
of DNS retries with UDP.

Issue #5584

[justinsb]

/retest

[dcowden]

Awesome! Looking forward to 1.11…!

[geojaz]

/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: geojaz, justinsb

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [geojaz,justinsb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[igoratencompass]

@justinsb I see an issue with the clusterDNS option for kubelet. When set under kubelet config like so:

spec:
  kubelet:
    clusterDNS: 169.254.20.10

it gets applied to master kubelets too:

  LaunchConfiguration/master-eu-west-1a.masters.cluster.name
  	UserData            
  	                    	...
  	                    	  set -o pipefail
  	                    	  
  	                    	+ NODEUP_URL=https://kubeupv2.s3.amazonaws.com/kops/1.12.0-beta.2/linux/amd64/nodeup
  	                    	- NODEUP_URL=https://kubeupv2.s3.amazonaws.com/kops/1.12.0-beta.1/linux/amd64/nodeup
  	                    	+ NODEUP_HASH=dc19d42f6580b0be5771004256e41b2ebe29beae
  	                    	- NODEUP_HASH=5c549c46519637485eca345e174d4060db13c75e
  	                    	  
  	                    	  export AWS_REGION=eu-west-1
  	                    	...
  	                    	    cgroupRoot: /
  	                    	    cloudProvider: aws
  	                    	+   clusterDNS: 169.254.20.10
  	                    	-   clusterDNS: 100.64.0.10

which is a problem since I don't have node-local-dns running on the master nodes. I tried to override the setting by putting clusterDNS: 100.64.0.10 in the masterKubelet section but the outcome was same. Shouldn't this setting for kubelet and masterKubelet be independent in case one wants to override it on one type of nodes only?

This is on kops-1.12-beta-2 by the way.