-
Notifications
You must be signed in to change notification settings - Fork 4.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kubectl logs POD broken when using amazon-vpc-cni-k8s (kubelet registers the wrong IP) #4218
Comments
Can you please file this issue with https://github.com/aws/amazon-vpc-cni-k8s? |
Hi @chrislovecnm, I will do as I cannot seem to reproduce the same issue when running a private kops cluster with flannel-vxlan by simply adding eni and IPs to the nodes |
Hi @dezmodue @chrislovecnm , the cni plugin back-end (L-IPAM) allocates IP addresses on the primary ENI interface right after it is initialized. So at the time when kubelet is reporting node address, the primary ENI interface already have multiple secondary IP addresses assigned. I like the idea of your poc that passing --node-ip=LOCAL-IPV4 to the kubelet. Can we use that to fix this issue? |
@chrislovecnm if that is a satisfactory solution I could try to work on it (with some guidance on how it should be implemented) |
@liwenwu-amazon what is the recommended solution? |
The recommended solution is
|
If someone wants to contribute this, I can provide more details on how to do this in nodeup. |
@chrislovecnm I would like to contribute |
@chrislovecnm I gave it a try - #4417 - let me know |
@chrislovecnm is it ok to close this issue since 4417 is released with 1.9.0 ? |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
Stale issues rot after 30d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
Rotten issues close after 30d of inactivity. Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
@fejta-bot: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
What
kops
version are you running? The commandkops version
, will displaythis information.
kops and nodeup built from master 2fdf834 - golang 1.8.5
What Kubernetes version are you running?
kubectl version
will print theversion if a cluster is running or provide the Kubernetes version specified as
a
kops
flag.Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:17:43Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
What cloud provider are you using?
AWS
What commands did you run? What is the simplest way to reproduce this issue?
kubectl logs POD -n NAMESPACE
What happened after the commands executed?
What did you expect to happen?
The container logs to be displayed
Please provide your cluster manifest. Execute
kops get --name my.example.com -oyaml
to display your cluster manifest.You may want to remove your cluster name and other sensitive information.
Please run the commands with most verbose logging by adding the
-v 10
flag.Paste the logs into this report, or in a gist and provide the gist link here.
Anything else do we need to know?
This error seems to be related to the fact that the kubelet registers the wrong IP, specifically it seems that the kubelet reports one of the secondary private IPs
In the example error reported:
10.103.20.110 is a secondary private IP and it is indeed the IP shown by kubectl describe node:
Locally curl works on both the primary IPs on eth0 and eth1
The problem has also been occurring on the master nodes and it manifests with new nodes unable to join the cluster because the kubelet is unable to contact the API (same reason, wrong IPs)
As a poc I built a modified nodeup that passes the flag --node-ip=LOCAL-IPV4 to the kubelet (where LOCAL-IPV4 is the result of
curl http://169.254.169.254/latest/meta-data/local-ipv4
)With that in place the master and nodes build correctly and
kubectl logs
works as expectedThe text was updated successfully, but these errors were encountered: