Support for CNI to use proxy

[akitekt]

Hi,
We are running a POC using the EKS preview.
Since we are behind a corporate proxy, we need to enable proxy for the CNI.
However, After the conversation with AWS support team, it seems the CNI container does not support proxy right now.
Is this going to be on the roadmap?

Thanks
Eric Liu

[bchav]

@akitekt Can you outline your configuration changes you've made on your cluster? IIRC you changed the kubelet and docker daemon config to route through the proxy?

[akitekt]

Yes, we changed both service file to use proxy to pull docker images and talk to EKS endpoint.
CNI is the only component left to enable proxy.
@bchav

[ewbankkit]

@akitekt We will also hit this problem (until we get our in-VPC EC2 service endpoints setup).
Interested to know if you are setting the HTTP_PROXY and associated environment variables for aws-k8s-agent somehow:

• Modifying the startup script to source /etc/environment, hard-code HTTP_PROXY, etc.
• Modifying the manifest to set the required environment variables

As the agent is using the AWS golang SDK, setting HTTP_PROXY should work.

[liwenwu-amazon]

@akitekt have you resolved the issue after modified the dns.yaml?

[akitekt]

After upgrading to EKS 1.9, The CNI container starts to working.
Same proxy changes are made on the kubelet, kube-proxy and docker engine.
There will be a new issue for kubedns to work.
The Fix is to modify dns.yaml to enable HTTP_PROXY for the kubedns container.
Do not worry about sidecar and dnsmasq, they will work after the kubedns fix above is made.

So EKS 1.9 + The proxy changes above ==> A private VPC version of EKS in business.
@ewbankkit @liwenwu-amazon @bchav

[liwenwu-amazon]

@akitekt thank for the update! can I close this issue now.

[ewbankkit]

I have the CNI plugin running behind an HTTP proxy by setting HTTP_PROXY etc. environment variables in the pod spec:

amazon-vpc-cni-k8s/misc/aws-k8s-cni.yaml

Line 30 in bd3297a

env:

[xdrus]

Hello,

@liwenwu-amazon I would like to extend this one with CIDR support as Kubernetes did:

• issue 23003
• some relevant code

While setting HTTP[S]_PROXY / NO_PROXY works well it is very handy to use CIDRs in NO_PROXY (e.g. I can specify my master subnet's CIDR and don't care regarding exact IP of my master). Otherwise, I need either put all ip addresses to NO_PROXY that is very messy or maintain proper DNS records and use DNS suffix (that can be cumbersome in some env).

Proxy is a strict requirement for many enterprise customers, so it would be nice to address this both for EKS and homegrown clusters with Amazon VPC CNI plugin.

Best,
Ruslan.

[sdavids13]

Hey @akitekt @ewbankkit @liwenwu-amazon could you please share what I might be missing? I created the following for both /etc/systemd/system/docker.service.d and /etc/systemd/system/kubelet.service.d:

ls docker.service.d/
http-proxy.conf  https-proxy.conf  no-proxy.conf
cat docker.service.d/*
[Service]
Environment="HTTP_PROXY=..........elb.us-east-1.amazonaws.com:3128"
[Service]
Environment="HTTPS_PROXY=..........elb.us-east-1.amazonaws.com:3128"
[Service]
Environment="NO_PROXY=localhost,127.0.0.1,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.elb.us-east-1.amazonaws.com,.elb.amazonaws.com,.s3.amazonaws.com,.s3.us-east-1.amazonaws.com,.dynamodb.us-east-1.amazonaws.com,169.254.169.254,.internal"

Updated kube-dns deployment via kubectl edit deployment.apps/kube-dns --namespace=kube-system to insert HTTPS_PROXY, HTTP_PROXY, and NO_PROXY in the path spec.template.spec.containers.args.env. I just edited the existing definition instead of running kubectl -f dns.yaml seeing how I have no idea where that file exists you were talking about.

I also did the similar thing for kubectl edit daemonset.apps/aws-node --namespace=kube-system

Unfortunately the kube-dns pods are stuck in ContainerCreating state and aws-node pod is in CrashLoopBackOff state. Kubelet is continually blowing with:

 dns.go:131] Nameserver limits were exceeded, some nameservers have been
 omitted, the applied nameserver line is: ...
 cni.go:259] Error adding network: rpc error: code = Unavailable desc =
grpc: the connection is unavailable
 cni.go:227] Error while adding to cni network: rpc error: code = Unavai
lable desc = grpc: the connection is unavailable
 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-7cff5cd864-gn2k2_kube-system" network: rpc error: code = Unavailable desc = grpc: the connection is unavailable
 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-dns-7cff5cd864-gn2k2_kube-system(ff4d7d10-712d-11e8-936a-0a9caac03b58)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-7cff5cd864-gn2k2_kube-system" network: rpc error: code = Unavailable desc = grpc: the connection is unavailable

Thanks for any help/pointers you can give to make this work.

**Edit: Please note that after adding the HTTPS_PROXY to the deployment & daemonset the proxy logs didn't show any additional traffic.

[ewbankkit]

@sdavids13 It may be that the container images for kube-dns can't be pulled on the worker nodes. The default location of the images is in ECR and access will be via the proxy. Can you do a kubectl describe pod on one of the pods stuck in ContainerCreating?

[ewbankkit]

@xdrus Ideally this would be done in the golang HTTP client: golang/go#16704. Otherwise the logic you linked to needs to be included in both the AWS SDK for Go and client-go.

[sdavids13]

@ewbankkit The worker node can reach out to download some of the containers, I ran docker images on the host with the following results:

REPOSITORY                                                          TAG                 IMAGE ID            CREATED             SIZE
602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/pause-amd64        3.1                 9e462c010bf3        2 weeks ago         742kB
602401143452.dkr.ecr.us-east-1.amazonaws.com/amazon-k8s-cni         1.0.0               7e6390decb99        2 weeks ago         347MB
602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/kube-proxy         v1.10.3             c6fc6eef666a        2 weeks ago         96.9MB

Having seen that the DNS docker images didn't get pulled on the host I went ahead and pulled the images by logging in via: aws ecr get-login --registry-ids 602401143452 --region us-east-1 --no-include-email then:

docker pull 602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/kube-dns/kube-dns:1.14.10
docker pull 602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/kube-dns/dnsmasq-nanny:1.14.10
docker pull 602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/kube-dns/sidecar:1.14.10

They were all able to be pulled through the proxy. As for the kubectl describe pods:

kubectl describe pod kube-dns-7547cfbccf-ms742 --namespace=kube-system
Name:           kube-dns-7547cfbccf-ms742
Namespace:      kube-system
Node:           <redacted>.ec2.internal/<redacted ip address>
Start Time:     Sat, 16 Jun 2018 06:27:18 +0000
Labels:         eks.amazonaws.com/component=kube-dns
                k8s-app=kube-dns
                pod-template-hash=3103796779
Annotations:    scheduler.alpha.kubernetes.io/critical-pod=
Status:         Pending
IP:
Controlled By:  ReplicaSet/kube-dns-7547cfbccf
Containers:
  kubedns:
    Container ID:
    Image:         602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/kube-dns/kube-dns:1.14.10
    Image ID:
    Ports:         10053/UDP, 10053/TCP, 10055/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      --domain=cluster.local.
      --dns-port=10053
      --config-dir=/kube-dns-config
      --v=2
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  170Mi
    Requests:
      cpu:      100m
      memory:   70Mi
    Liveness:   http-get http://:10054/healthcheck/kubedns delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:  http-get http://:8081/readiness delay=3s timeout=5s period=10s #success=1 #failure=3
    Environment:
      PROMETHEUS_PORT:  10055
      HTTPS_PROXY:      <redacted>.elb.us-east-1.amazonaws.com:3128
      HTTP_PROXY:       <redacted>.elb.us-east-1.amazonaws.com:3128
      NO_PROXY:         172.16.0.0/12,192.168.0.0/16,localhost,127.0.0.1,.elb.us-east-1.amazonaws.com,.elb.amazonaws.com,.s3.amazonaws.com,.s3.us-east-1.amazonaws.com,.dynamodb.us-east-1.amazonaws.com,169.254.169.254,.internal,10.0.0.0/8
    Mounts:
      /kube-dns-config from kube-dns-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-khlxk (ro)
  dnsmasq:
    Container ID:
    Image:         602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/kube-dns/dnsmasq-nanny:1.14.10
    Image ID:
    Ports:         53/UDP, 53/TCP
    Host Ports:    0/UDP, 0/TCP
    Args:
      -v=2
      -logtostderr
      -configDir=/etc/k8s/dns/dnsmasq-nanny
      -restartDnsmasq=true
      --
      -k
      --cache-size=1000
      --no-negcache
      --log-facility=-
      --server=/cluster.local/127.0.0.1#10053
      --server=/in-addr.arpa/127.0.0.1#10053
      --server=/ip6.arpa/127.0.0.1#10053
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:     150m
      memory:  20Mi
    Liveness:  http-get http://:10054/healthcheck/dnsmasq delay=60s timeout=5s period=10s #success=1 #failure=5
    Environment:
      HTTPS_PROXY:  <redacted>.elb.us-east-1.amazonaws.com:3128
      HTTP_PROXY:   <redacted>.elb.us-east-1.amazonaws.com:3128
      NO_PROXY:     172.16.0.0/12,192.168.0.0/16,localhost,127.0.0.1,.elb.us-east-1.amazonaws.com,.elb.amazonaws.com,.s3.amazonaws.com,.s3.us-east-1.amazonaws.com,.dynamodb.us-east-1.amazonaws.com,169.254.169.254,.internal,10.0.0.0/8
    Mounts:
      /etc/k8s/dns/dnsmasq-nanny from kube-dns-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-khlxk (ro)
  sidecar:
    Container ID:
    Image:         602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/kube-dns/sidecar:1.14.10
    Image ID:
    Port:          10054/TCP
    Host Port:     0/TCP
    Args:
      --v=2
      --logtostderr
      --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,SRV
      --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,SRV
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:     10m
      memory:  20Mi
    Liveness:  http-get http://:10054/metrics delay=60s timeout=5s period=10s #success=1 #failure=5
    Environment:
      HTTPS_PROXY:  <redacted>.elb.us-east-1.amazonaws.com:3128
      HTTP_PROXY:   <redacted>.elb.us-east-1.amazonaws.com:3128
      NO_PROXY:     172.16.0.0/12,192.168.0.0/16,localhost,127.0.0.1,.elb.us-east-1.amazonaws.com,.elb.amazonaws.com,.s3.amazonaws.com,.s3.us-east-1.amazonaws.com,.dynamodb.us-east-1.amazonaws.com,169.254.169.254,.internal,10.0.0.0/8
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-khlxk (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          False
  PodScheduled   True
Volumes:
  kube-dns-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kube-dns
    Optional:  true
  kube-dns-token-khlxk:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kube-dns-token-khlxk
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     CriticalAddonsOnly
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                  Age                    From                                   Message
  ----     ------                  ----                   ----                                   -------
  Normal   SandboxChanged          25m (x158398 over 2d)  kubelet, <redacted>.ec2.internal  Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  29s (x157567 over 2d)  kubelet, <redacted>.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-7547cfbccf-ms742_kube-system" network: rpc error: code = Unavailable desc = grpc: the connection is unavailable

Thanks for the help!

[sdavids13]

I was able to get it working, I needed to set the proxy environment variables for the kube-proxy daemonset as well. For a quick synopsis of my current configuration here are the necessary modifications:

Systemd settings:

Created three files put in both /etc/systemd/system/docker.service.d/ and /etc/systemd/system/kubelet.service.d/:

• http-proxy.conf

[Service]
Environment="HTTP_PROXY=<redacted>.elb.us-east-1.amazonaws.com:3128"

• https-proxy.conf

[Service]
Environment="HTTPS_PROXY=<redacted>.elb.us-east-1.amazonaws.com:3128"

• no-proxy.conf
  • Note: your NO_PROXY may vary depending on your custom domains, current AWS region, and enabled VPC Endpoints.

[Service]
Environment="NO_PROXY=localhost,127.0.0.1,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.elb.us-east-1.amazonaws.com,.elb.amazonaws.com,.s3.amazonaws.com,.s3.us-east-1.amazonaws.com,.dynamodb.us-east-1.amazonaws.com,169.254.169.254,.internal"

Kubernetes settings:

Store proxy settings in ConfigMap

Created proxy-environment-variables-config.yaml with:

apiVersion: v1
kind: ConfigMap
metadata:
  name: proxy-environment-variables
  namespace: kube-system
data:
  HTTPS_PROXY: <redacted>.elb.us-east-1.amazonaws.com:3128
  HTTP_PROXY: <redacted>.elb.us-east-1.amazonaws.com:3128
  NO_PROXY: localhost,127.0.0.1,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.elb.us-east-1.amazonaws.com,.elb.amazonaws.com,.s3.amazonaws.com,.s3.us-east-1.amazonaws.com,.dynamodb.us-east-1.amazonaws.com,169.254.169.254,.internal

Apply via kubectl apply -f proxy-environment-variables-config.yaml. See previous NO_PROXY caveat.

Inject environment variables into containers

Inject the proxy environment variables ConfigMap by adding the following snippet in the container definition:

envFrom:
- configMapRef:
  name: proxy-environment-variables

Edit the following to inject proxy environment variables:

• kubectl edit daemonset kube-proxy --namespace=kube-system
• kubectl edit daemonset aws-node --namespace=kube-system
• kubectl edit deployment kube-dns --namespace=kube-system
  • Note: there are 3 containers in the deployment definition only containers[name=kubedns] requires the environment variables.

I tried using the kubectl patch functionality to inject only the envFrom definition but wasn't able to get a good merge even with --type merge specified. If anyone happens to know how to make the patch work appropriately it may be useful for others.

[empath]

I tried all of that and I'm still in a state like this:

❯ kubectl get pods
NAME                                    READY     STATUS              RESTARTS   AGE
aws-node-7hgh2                          0/1       CrashLoopBackOff    5          3m
aws-node-9wt98                          0/1       CrashLoopBackOff    5          3m
aws-node-thlxc                          0/1       CrashLoopBackOff    5          3m
kube-dns-778cd8cd76-bpswt               0/3       ContainerCreating   0          4m
kube-dns-7cc87d595-mm5r4                0/3       ContainerCreating   0          4m
kube-proxy-4884z                        1/1       Running             0          4m
kube-proxy-bfb2l                        1/1       Running             0          4m
kube-proxy-qql65                        1/1       Running             0          4m
kubernetes-dashboard-7d5dcdb6d9-5qzkt   0/1       ContainerCreating   0          4m

I don't have any idea how to get logs or troubleshoot this.

I can exec into the proxy at least and it seems to have the correct environment variables.

The logs for aws-node are basically empty.

=====Starting installing AWS-CNI =========
=====Starting amazon-k8s-agent ===========
ERROR: logging before flag.Parse: W0711 17:14:25.328966      11 client_config.go:533] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.

[sdavids13]

Did you look at the kubelet logs on one of the host machines? Did you check your proxy access logs to see if any EKS traffic is being routed through the proxy? Also, after updating the various daemonsets and deployments defined above did you kill off all of the existing pods (which will then spawn new pods with the updated configuration)?

[empath]

Turns out the problem was that I needed to add the kube ip range to no_proxy.

…

On Wed, Jul 11, 2018 at 2:22 PM Steve D ***@***.***> wrote: Did you look at the kubelet logs on one of the host machines? Did you check your proxy access logs to see if any EKS traffic is being routed through the proxy? Also, after updating the various daemonsets and deployments defined above did you kill off all of the existing pods (which will then spawn new pods with the updated configuration)? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#49 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAXuge_2x4slaCJtW3UKksRqoiWewE-Zks5uFkJxgaJpZM4SpPuv> .

[sdavids13]

I thought I had all of the available private IP space available in the configuration above, did I miss one?

[empath]

Your directions were fine, I just had used our normal no_proxy string rather than yours.

…

On Wed, Jul 11, 2018 at 6:07 PM Steve D ***@***.***> wrote: I thought I had all of the available private IP space available in the configuration above, did I miss one? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#49 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAXugZ1BHXdAGjEcQChUSIQ6vaeB0ndZks5uFncpgaJpZM4SpPuv> .

[liwenwu-amazon]

@sdavids13 Thanks for providing the solution to this issue. Can you move them to a document in ./doc ? thanks

[kovalyukm]

These solutions for worker nodes with pods (first of all aws-node) behind proxy do not work, at least for current versions of k8s and eks clusters.
kube-system aws-node-hbmfx 0/1 CrashLoopBackOff 5 8m
pod restarts all time and there is no any logs inside pod
Error syncing pod ("aws-node-hbmfx_kube-s
ystem(ae5dada9-c0df-11e8-9935-02f3b8cfe516)"), skipping: failed to "StartContainer" for "aws-node" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=aws-node

[kovalyukm]

Faced with the same issue: first off to set the proxy environment variables for the kube-proxy daemonset to make services endpoints working, that aws-node pod uses.
BUT, I think this solution isn't for production use. Am I right?
AFAIU, daemonsets are managed by AWS, so proxy environment variables can disappear at any moment?
What is the AWS recommended way to have up and running worker nodes behind proxy?
Thanks!

[liwenwu-amazon]

@MykhailoKovaliuk, here is one way how people solve this in EKS:

• whitelist(no proxy) Pod's CIDR and worker IP on worker to Kubernetes service endpoints address.
  • you can find out kubernetes service endpoint addresses by kubectl describe service kubernetes

keep in mind, aws-node daemonset need to communicate to kubernetes service endpoint too. Without whitelist worker IP, aws-node will not able to communicate to kubernetes service endpoint.

[kovalyukm]

@liwenwu-amazon
Not understand, please, correct me if I'm wrong.
aws-node can communicate to kubernetes service endpoint when kube-proxy is up and running on the worker node, but kube-proxy communicate to API server endpoint of EKS cluster, that has internet addresses.

[bzon]

@liwenwu-amazon

• whitelist(no proxy) Pod's CIDR and worker IP on worker to Kubernetes service endpoints address.

  • you can find out kubernetes service endpoint addresses by kubectl describe service kubernetes

Do you mean no_proxy the POD's CIDR and Worker IP in kubelet extra args?

Edited: It works! Holly sh***t!

[Service]
Environment='HTTP_PROXY=http://your.corporate.proxy.host:8080'
Environment='HTTPS_PROXY=http://your.corporate.proxy.host:8080'
Environment='NO_PROXY=localhost,127.0.0.1,instance-data,169.254.169.254,.your.corporate.domain.com,$AWS_VPC_CIDR_HERE,$THE_NODE_IP_HERE'

[fbdo]

Hi all,

Using the tips I found in this thread, I forked the official AWS AMI for EKS and I added the required configuration for docker and kubelets, see them here: https://github.com/fbdo/amazon-eks-ami/tree/proxy-feature

I am passing the proxy configurations using the userdata.sh script, I have a Terraform script executing the bootstrap as in:

/etc/eks/bootstrap.sh --b64-cluster-ca '${cluster_auth_base64}' --apiserver-endpoint '${endpoint}' --kubelet-extra-args '${kubelet_extra_args}' --http-proxy '${http_proxy}' --https-proxy '${https_proxy}' --no-proxy '${no_proxy}' '${cluster_name}'

My proxy config is:
http_proxy = ""
https_proxy = ""
no_proxy = "instance-data,localhost,127.0.0.1,169.254.169.254,10.0.0.0/8,192.168.0.0/16,172.16.0.0/12,.internal"

I have a EKS cluster almost working, I am stuck now with the pods in the state:

NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system aws-node-6r6pn 1/1 Running 11 2h
kube-system aws-node-pkmx6 1/1 Running 11 2h
kube-system kube-dns-68d76b8689-gxqww 0/3 ContainerCreating 0 2h
kube-system kube-dns-fcd468cb-vltrt 0/3 ContainerCreating 0 2h
kube-system kube-proxy-f78wc 1/1 Running 1 2h
kube-system kube-proxy-xm85r 1/1 Running 1 2h

In one worker node, I can see in the logs:
Dec 11 16:38:19 kubelet[4102]: E1211 16:38:19.822331 4102 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-dns-fcd468cb-vltrt_kube-system(5abf2f9c-fd50-11e8-a4aa-0a4e2bf81922)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-fcd468cb-vltrt_kube-system" network: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:50051: connect: connection refused"

I know, you guys already helped me a lot, but if someone has any clues it will be amazing! That issue was really make me even more hairless...

Thanks!

[kovalyukm]

Seems your aws-nodes are not really in Running state:
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system aws-node-6r6pn 1/1 Running 11 2h
they are restarting continuously.
I guess you didn't configure kube-proxy daemonset to use proxy to reach EKS cluster API, did you?

[fbdo]

I followed exactly the @sdavids13 recipe, doing changes for all 3:

kubectl edit daemonset kube-proxy --namespace=kube-system
kubectl edit daemonset aws-node --namespace=kube-system
kubectl edit deployment kube-dns --namespace=kube-system

No success.

[dresnick-sf]

I just got this working with k8s 1.11. Note that
kubectl edit deployment coredns --namespace=kube-system
replaces
kubectl edit deployment kube-dns --namespace=kube-system
Make sure you kill all previously running pods after applying the proxy settings to the daemonsets/deployment.

[dresnick-sf]

Patching the 3 entities with kubectl patch:

bash-5.0$ cat patch-aws-node-env-from.yaml
spec:
  template:
    spec:
      containers:
        - name: aws-node
          envFrom:
          - configMapRef:
              name: proxy-environment-variables
bash-5.0$ cat patch-coredns-env-from.yaml
spec:
  template:
    spec:
      containers:
        - name: coredns
          envFrom:
          - configMapRef:
              name: proxy-environment-variables
bash-5.0$ cat patch-kube-proxy-env-from.yaml
spec:
  template:
    spec:
      containers:
        - name: kube-proxy
          envFrom:
          - configMapRef:
              name: proxy-environment-variables
bash-5.0$ kubectl patch daemonset kube-proxy -n kube-system --patch "$(cat patch-kube-proxy-env-from.yaml)"
daemonset.extensions "kube-proxy" patched
bash-5.0$ kubectl patch daemonset aws-node -n kube-system --patch "$(cat patch-aws-node-env-from.yaml)"
daemonset.extensions "aws-node" patched
bash-5.0$ kubectl patch deployment coredns -n kube-system --patch "$(cat patch-coredns-env-from.yaml)"
deployment.extensions "coredns" patched

[pssayi]

I had the same problem on k8s 1.12 even after patching aws-node and kube-proxy, the pods were still trying to create and then getting into a Crash. I resolved the issue re-running to pull the config maps again.

kubectl --kubeconfig=/<path>/config set env daemonset/aws-node --namespace=kube-system --from=configmap/proxy-environment-variables --containers='*

kubectl --kubeconfig=/<path>/config set env daemonset/kube-proxy --namespace=kube-system --from=configmap/proxy-environment-variables --containers='*'

All the pods started running fine after that, and is leveraging proxy variables now.

[ausmith]

Turns out you should also add the control plane endpoint to the NOPROXY environment variable, else kubelet will try to go out through your HTTP proxy to get to the control plane. This impacted me as I was spinning up an EKS cluster behind a restrictive HTTP proxy but the kubelet could not start because it was getting a 403 Forbidden when trying to go out to the EKS endpoint through the HTTP Proxy. This resulted in my NOPROXY looking more like this with $CONTROL_PLANE_DOMAIN being the EKS endpoint domain name:

NOPROXY=localhost,127.0.0.1,169.254.169.254,.internal,$VPC_CIDR,$OTHER_INTERNAL_CIDR_RANGES,$CORP_INTERNAL_TLD,$CONTROL_PLANE_DOMAIN

If you are using Terraform like I am to create my EKS clusters, you may have noticed that the AWS provider gives an endpoint for the EKS cluster but that includes https:// on the front of it. My quick workaround for passing into a template for the user data was

data "template_file" "eks_worker_user_data" {
  template = file("${path.module}/templates/eks_workers-cloud-config.yaml")

  vars = {
    KubeDomainName = split("://", aws_eks_cluster.cluster.endpoint)[1]
  }
}

[mogren]

Using CNI v1.5.7 or v1.6.2 or later, using a proxy should work just fine. There is a blog post describing how to set it up here:

https://aws.amazon.com/premiumsupport/knowledge-center/eks-http-proxy-configuration-automation/

Will close this ticket once the instructions have been added to the official AWS documentation.

[pow-devops2020]

Following this tutorial to set up proxy for my cluster. However it seemed like the proxy does not work properly when I ssh into my node to check. Is it an issue?

[jayanthvn]

Hi @pow-devops2020

Can you please confirm if you followed the steps here - https://aws.amazon.com/premiumsupport/knowledge-center/eks-http-proxy-configuration-automation/

Also can you please explain more on the steps to repro and what issue are you seeing?

Thank you!

[achevuru]

@pow-devops2020 Closing the issue. Feel free to reopen it if the instructions in the above doc didn't help.