Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unstable service ip access from another pod #38802

Closed
rxwen opened this issue Dec 15, 2016 · 17 comments
Closed

Unstable service ip access from another pod #38802

rxwen opened this issue Dec 15, 2016 · 17 comments

Comments

@rxwen
Copy link

rxwen commented Dec 15, 2016 •
edited
Loading

Is this a request for help? :
Yes
What keywords did you search in Kubernetes issues before filing this one?:
service ip

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1", GitCommit:"82450d03cb057bab0950214ef122b67c83fb11df", GitTreeState:"clean", BuildDate:"2016-12-14T00:57:05Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.7", GitCommit:"92b4f971662de9d8770f8dcd2ee01ec226a6f6c0", GitTreeState:"clean", BuildDate:"2016-12-10T04:43:42Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

  • Cloud provider or hardware configuration: Cloud provider, aliyun
  • OS (e.g. from /etc/os-release):
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

What happened:
I deployed pods and a redis pod. The redis exposes 6379 port as a cluster service.
From one of my pod, I can't establish tcp connection to the redis service (netstat showed the tcp connection is in SYN_SENT state). And it's ok to use redis service from other pods. But it's fine if I tried to access the redis with its pod ip, instead of service ip.
I scaled the pod to larger numbers, then tried to access redis from other copies of the pod, the redis service was ok.

What you expected to happen:
The redis service can be used from any of pod in the cluster.

How to reproduce it (as minimally and precisely as possible):

  • setup a 1.4.7 k8s cluster
  • deploy a redis service
  • deploy my own application pod which uses redis

Anything else do we need to know:
The overlay network I'm using is flannel:0.6.2

# iptables-save | grep redis
-A KUBE-SEP-HVZ75DS3MAWHRBXB -s 10.1.81.3/32 -m comment --comment "default/redis:redis" -j KUBE-MARK-MASQ
-A KUBE-SEP-HVZ75DS3MAWHRBXB -p tcp -m comment --comment "default/redis:redis" -m tcp -j DNAT --to-destination 10.1.81.3:6379
-A KUBE-SERVICES -d 10.0.0.66/32 -p tcp -m comment --comment "default/redis:redis cluster IP" -m tcp --dport 6379 -j KUBE-SVC-GDFSTJAUHP7KZZTP
-A KUBE-SVC-GDFSTJAUHP7KZZTP -m comment --comment "default/redis:redis" -j KUBE-SEP-HVZ75DS3MAWHRBXB

access redis ok from a pod:

/ # nc 10.0.0.66 6379
help
-ERR unknown command 'help'

access redis failed from a pod: (exact copy of the successed pod created via kutectl scale)

tcp        0      1 10.1.81.5:45624         10.0.0.66:6379          SYN_SENT    35/nc

access redis ok with its pod ip:

# nc 10.1.81.3 6379
f
-ERR unknown command 'f'
@rxwen
Copy link
Author

rxwen commented Dec 15, 2016

I also reported a similar issue on stackoverflow. In which case, I used kubernetes 1.5.1.

@rxwen
Copy link
Author

rxwen commented Dec 15, 2016

Another issue I found that may be related is when I used dig tool to resolve a name within replicates of a pod. Two of them were resolved successfully. But on one pod, the dig reported that the response is from a different ip address.
In my setup, the dns service's ip address is 10.0.0.10. And all pods' /etc/resolv.conf are configured to use this ip as dns server. But the pod that reported error complained that the dns reply is from ip address 10.2.2.*, which is the ip address of the dns pod. Does it imply that the dns response packet isn't SNATed by the iptable rule, for this particular pod?
I'm not able to reproduce this issue after I tear down the cluster.

@rxwen
Copy link
Author

rxwen commented Dec 15, 2016 •
edited
Loading

I did some network sniffing on the cni0 interface on the actual node that runs the pod failed to connect redis. Found something interesting. The ip of the app pod is 10.1.85.5. And the pod ip of redis is 10.1.81.3, redis service ip is 10.0.0.66.
When I issued nc 10.0.0.66 6379 command, I saw two tcp SYN packets immediately.

10.1.81.5   --  SYN  --> 10.0.0.66
10.1.81.5   --  SYN  --> 10.1.81.3
10.1.81.5  <-- SYN,ACK -- 10.1.81.3
10.1.81.5  -- RST --> 10.1.81.3
10.1.81.5  --  retranmission of SYN --> 10.0.0.66

image

How come will 10.1.85.5 send SYN to 10.1.81.3? The expected result should be 10.1.81.5 talks to 10.0.0.66 only. And why 10.0.0.66 doesn't send back SYN,ACK.

Well, further investigation showed that the redis pod is scheduled on the same node. So the second SYN packet is the packet received by the redis.
So it seems that other replicates of the pod run on different node can talk to redis without problem. But the pod running on the same node can't access the service.

I believe this should be a basic scenario supported by k8s, especially considering minikube that runs everything on a single node. Is there any possible configuration that I didn't set right?

@thockin Execue me, I'm hoping it's ok to bring your attention to this.

@aoxn
Copy link
Contributor

aoxn commented Dec 16, 2016

@rxwen @du2016 What network modal do you use? Try k8s with flannel VPC on AliCloud. Link

@rxwen
Copy link
Author

rxwen commented Dec 16, 2016

@spacexnice Unfortunately, we're using classical network model.

@thockin
Copy link
Member

thockin commented Dec 16, 2016 via email

@rxwen
Copy link
Author

rxwen commented Dec 16, 2016

Yes, I used tcpdump. As you said, the SYNACK isn't un-NATed, that's why our app end send a RST.

Should there be a iptables rule to do unNAT explicitly? I can't seem to find a rule did this.

# conntrack -L
tcp      6 3069 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=59973 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=59973 [ASSURED] mark=0 use=1
tcp      6 1197 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=56533 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=56533 [ASSURED] mark=0 use=1
tcp      6 86388 ESTABLISHED src=10.28.65.157 dst=10.28.65.112 sport=56912 dport=10250 src=10.28.65.112 dst=10.28.65.157 sport=10250 dport=56912 [ASSURED] mark=0 use=1
tcp      6 1037 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=53907 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=53907 [ASSURED] mark=0 use=1
tcp      6 2141 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=57557 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=57557 [ASSURED] mark=0 use=1
tcp      6 86373 ESTABLISHED src=10.28.65.112 dst=10.28.65.157 sport=51402 dport=8080 src=10.28.65.157 dst=10.28.65.112 sport=8080 dport=51402 [ASSURED] mark=0 use=1
tcp      6 86399 ESTABLISHED src=10.10.28.7 dst=10.28.65.157 sport=59492 dport=2379 src=10.28.65.157 dst=10.28.65.112 sport=2379 dport=59492 [ASSURED] mark=0 use=1
tcp      6 86373 ESTABLISHED src=10.28.65.112 dst=10.28.65.157 sport=60802 dport=2379 src=10.28.65.157 dst=10.28.65.112 sport=2379 dport=60802 [ASSURED] mark=0 use=1
tcp      6 1092 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=54808 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=54808 [ASSURED] mark=0 use=1
tcp      6 86395 ESTABLISHED src=10.28.65.112 dst=10.28.65.157 sport=51383 dport=8080 src=10.28.65.157 dst=10.28.65.112 sport=8080 dport=51383 [ASSURED] mark=0 use=1
tcp      6 2713 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=59036 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=59036 [ASSURED] mark=0 use=1
tcp      6 2182 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=58314 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=58314 [ASSURED] mark=0 use=1
tcp      6 86293 ESTABLISHED src=10.28.65.157 dst=10.28.65.112 sport=56684 dport=10250 src=10.28.65.112 dst=10.28.65.157 sport=10250 dport=56684 [ASSURED] mark=0 use=1
tcp      6 3316 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=60865 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=60865 [ASSURED] mark=0 use=1
tcp      6 1182 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=56287 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=56287 [ASSURED] mark=0 use=1
tcp      6 3059 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=59804 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=59804 [ASSURED] mark=0 use=1
tcp      6 86373 ESTABLISHED src=10.28.65.112 dst=10.28.65.157 sport=60804 dport=2379 src=10.28.65.157 dst=10.28.65.112 sport=2379 dport=60804 [ASSURED] mark=0 use=1
tcp      6 1227 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=57034 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=57034 [ASSURED] mark=0 use=2
tcp      6 3054 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=59725 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=59725 [ASSURED] mark=0 use=1
tcp      6 1117 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=55229 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=55229 [ASSURED] mark=0 use=1
tcp      6 3064 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=59884 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=59884 [ASSURED] mark=0 use=1
tcp      6 2146 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=57718 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=57718 [ASSURED] mark=0 use=1
tcp      6 3276 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=60202 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=60202 [ASSURED] mark=0 use=1
tcp      6 86399 ESTABLISHED src=10.28.65.106 dst=10.28.65.112 sport=47056 dport=22 src=10.28.65.112 dst=10.28.65.106 sport=22 dport=47056 [ASSURED] mark=0 use=1
tcp      6 86382 ESTABLISHED src=10.28.65.112 dst=10.28.65.157 sport=51390 dport=8080 src=10.28.65.157 dst=10.28.65.112 sport=8080 dport=51390 [ASSURED] mark=0 use=1
tcp      6 1027 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=53741 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=53741 [ASSURED] mark=0 use=1
tcp      6 86399 ESTABLISHED src=10.28.65.112 dst=10.28.65.157 sport=51398 dport=8080 src=10.28.65.157 dst=10.28.65.112 sport=8080 dport=51398 [ASSURED] mark=0 use=1
tcp      6 3074 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=60050 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=60050 [ASSURED] mark=0 use=1
tcp      6 3291 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=60461 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=60461 [ASSURED] mark=0 use=1
tcp      6 86384 ESTABLISHED src=10.28.65.112 dst=10.28.65.157 sport=51394 dport=8080 src=10.28.65.157 dst=10.28.65.112 sport=8080 dport=51394 [ASSURED] mark=0 use=1
tcp      6 2142 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=57602 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=57602 [ASSURED] mark=0 use=1
tcp      6 1162 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=55960 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=55960 [ASSURED] mark=0 use=1
tcp      6 1187 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=56369 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=56369 [ASSURED] mark=0 use=1
tcp      6 1207 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=56699 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=56699 [ASSURED] mark=0 use=1
tcp      6 1152 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=55794 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=55794 [ASSURED] mark=0 use=1
tcp      6 3301 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=60622 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=60622 [ASSURED] mark=0 use=1
tcp      6 1147 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=55714 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=55714 [ASSURED] mark=0 use=1
tcp      6 1057 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=54235 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=54235 [ASSURED] mark=0 use=1
tcp      6 1047 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=54071 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=54071 [ASSURED] mark=0 use=1
tcp      6 1032 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=53823 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=53823 [ASSURED] mark=0 use=1
tcp      6 1247 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=57364 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=57364 [ASSURED] mark=0 use=1
tcp      6 1172 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=56122 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=56122 [ASSURED] mark=0 use=1
tcp      6 86392 ESTABLISHED src=10.28.65.112 dst=100.100.25.3 sport=54028 dport=80 src=100.100.25.3 dst=10.28.65.112 sport=80 dport=54028 [ASSURED] mark=0 use=1
tcp      6 1112 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=55150 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=55150 [ASSURED] mark=0 use=1
tcp      6 1202 CLOSE_WAIT src=10.10.12.0 dst=10.10.28.7 sport=56617 dport=5222 src=10.10.28.7 dst=10.10.12.0 sport=5222 dport=56617 [ASSURED] mark=0 use=1
tcp      6 86395 ESTABLISHED src=10.28.65.112 dst=10.28.65.157 sport=51948 dport=8080 src=10.28.65.157 dst=10.28.65.112 sport=8080 dport=51948 [ASSURED] mark=0 use=1
conntrack v1.4.3 (conntrack-tools): 46 flow entries have been shown.

@thockin
Copy link
Member

thockin commented Dec 16, 2016 via email

@chenchun
Copy link
Contributor

@rxwen can you check if cat /proc/sys/net/bridge/bridge-nf-call-iptables is set? Maybe related #38751

@rxwen
Copy link
Author

rxwen commented Dec 16, 2016

@chenchun No, it's 0.
On another cluster that is working fine, it's 1.

Is it a requirement?

@chenchun
Copy link
Contributor

Yes, if your are using bridge device. Can you try to set it and test again?

@rxwen
Copy link
Author

rxwen commented Dec 16, 2016

Seems to find relevant doc here. http://kubernetes.io/docs/admin/network-plugins/

@rxwen
Copy link
Author

rxwen commented Dec 16, 2016

@chenchun A quick test succeed! Thanks.

@thockin
Copy link
Member

thockin commented Dec 16, 2016

@freehan we should make sure the CNI bridge driver takes over this responsibility.

@thockin
Copy link
Member

thockin commented Dec 16, 2016

should we close this bug?

@freehan
Copy link
Contributor

freehan commented Dec 16, 2016 •
edited
Loading

Opened an issue for tracking #38890

closing this one

@freehan freehan closed this as completed Dec 16, 2016
@ikaven1024
Copy link

i meet the same issue. i am using k8s v1.6.4 with flannel v0.7.1. Do flannel need /proc/sys/net/bridge/bridge-nf-call-iptables? the file does not exist on my host system.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@rxwen @chenchun @freehan @aoxn @thockin @ikaven1024