POD to Service no response #1073
Labels
Comments
|
I'm not too familiar with OpenShift but in vanilla k8s, I'd expect to see kube-proxy running. kube-proxy is the pod that implements service VIP on top of Calico's pod networking. |
|
Initially I was facing different issue, that was resolved. To me its a calico issue. Let me link that also. openshift/openshift-ansible#5235 |
|
Calico does not configure services, it provides IP addresses for containers and policy if it is configured. kube-proxy is responsible for setting up services and the proper rules for redirecting traffic to the appropriate IP address. Since Pod-to-Pod traffic is working and you have no policy a likely candidate is kube-proxy or access to the service itself. kube-proxy does need to be configured with the Calico IP Pool CIDR, I'm not too familiar with OpenShift but I would assume that configuration is handled automatically. I had a couple thoughts from the issue you linked:
|
|
Though the issue I linked, I given system logs from where POD running. In firewall, I allow etcd port which resolve previous issue. |
|
Errors was |
|
The logs show the tear down having a problem and the root of the problem is |
|
You should try taking a look at the iptables rules (I'd suggest |
|
Hmmm, but in starting I mentioned all containers are running by executing command |
|
By default openshift support proxy with iptable based. openshift_node_proxy_mode=iptables |
|
Not sure but if it is related any --hairpin-mode ? |
|
May be it will help you. POD to Service no responseAble to connect from router pod to Kubernetes, router & docker registry (which is running in same host)But not able to connect from router pod to mongo or myemp (which is running in different host)IP route outputBasically POD to POD NO communication if pod in
|
|
@prasenforu are you deploying on public cloud? By default Calico for OpenShift enables IPIP encapsulation. You may need to either:
|
|
Yes, I am running in AWS & my AWS architecture as follows.
Initially I allowed 179 port, that is you can see Then tried with As per both document, I think EC2 hosts All EC2 hosts src/dest checks Disabled.Right now setup is with Additionally I am getting getting error in |
|
You should have your AWS security groups configured as listed here https://docs.projectcalico.org/v2.5/reference/public-cloud/aws#configure-security-groups For the following two reasons I believe you need the recommended security group settings and if you already have those please double check the settings:
|
|
I tried with both but no luck Sorry for last statement "disable ipip" basically it enabled if saw the output of I disabled EC2 src/dest checks. |
|
I think you should do both allow IPIP traffic and disable src/dest checks until you have a working system. Independent of those options though it also looks like your node-to-node mesh is not working and you need to figure out the problem. I suggest:
|
|
OK, let me try. Quick question do I need ipip enable? |
|
@prasenforu If you'd like, happy to help with some real time troubleshooting together with you via zoom or hangouts today - let me know. |
|
Thanks Karthik. Unfortunately I am in IST zone. Can you please setup web call (zoom) on Monday (11 sep). Also let me know the good time. |
|
Today I tried to install fresh by using git clone openshift-ansible with EC2
Noticed pod ip range change. Then tried with But no luck :(Looks PODs are communicating in same subnet (same AZ) not with POD which resides other subnet (different AZ). |
|
Could you check if you are using protocol 4 for IPIP or protocol 94? IIRC, AWS provides options for both in their security group configuration options |
|
When I entered IPIP in protocol, I noticed its automatically taken
|
|
Can you add IP Protocol 4 to the security group list as well? I think Linux (and therefore Calico) use standard rfc2003 type IP-in-IP, which is IP protocol type 4. Also, not sure if you're planning on using nodeports, or other kubernetes service types, if so, you might need to open up other ports as well in case of service redirects from one node to an endpoint in a different node. BTW, if you have time for a zoom/hangout session now, and still want to do some real time troubleshooting together, let me know. |
|
Sent you a zoom invite. |
|
[root@ose-master ~]# ip route show [root@ose-master ~]# oc get pod -o wide [root@ose-master ~]# calicoctl node status IPv4 BGP status IPv6 BGP status [root@ose-master ~]# calicoctl get ippool -o yaml
|
|
@ozdanborne @tmjd As you can see from the output shared by @prasenforu , the (bird) routes on the nodes are 10.0.129.64/26, 10.0.134.192/26 and 10.0.157.128/26 and don't match the pod's IP addresses which are 10.129.177.195, 10.130.134.195, etc. |
|
Is it possible there are 2 network configurations set up? And Calico is not being used? |
|
As discussed with Prabhakar, I have freshly install by adding following variables and my ansible host file as follows, Below output after installation (by default came, NOT executed by me)My PODs runningIP route of each hostBut no luckAbove I tried ping form router pod (running in HUB host) to mongo pod (running in NODE-1 host), result
|
|
[root@ose-master ~]# calicoctl get wep [root@ose-master ~]# calicoctl node status IPv4 BGP status IPv6 BGP status |
|
@prasenforu Your Calico ippool (10.128.0.0/14) still does not match the pool you configure in /etc/ansible/hosts, i.e. calico_ipv4pool_cidr="192.168.0.0/16". When you do a fresh install, are you simply rerunning ansible-playbook? As @tmjd points out, perhaps there is cruft left behind in etcd from previous installs on the same master node, and might need to be cleaned up. |
|
Everything was installed in a new fresh ec2 hosts. Not sure from where its pulling ipool (10 cidr) |
|
Though its did not |
|
After little search I found from where its pulling Because of recent change made on ansible file Due to above parameter its uses default configuration, the cluster network I tried to edit After changing above parameter I was facing below issue. That is why |
|
@prasenforu Indeed, I had suggested backing out of that commit (i.e., replacing openshift.master.sdn_cluster_network_cidr with calico_ipv4pool_cidr) prior to redeploying. Would suggest checking 'calicoctl get ippool -o yaml', 'oc get pods -o wide', 'ip addr show' and 'ip route show' to make sure that:
|
Ansible hostsOut put as requested ...below some testing but result as earlier before, in same host pods able connect but not with other hsot. |
|
From what I see it looks like everything is correct (except that traffic is not flowing). |
|
Finally able to resolve the issue, Issue was in iptables, but not sure who was cluprit, is it Openshift or Calico because both created rules after default setup. My default /etc/sysconfig/iptables After Finally what I did on on node1 & node2Issue got resolved but noticed that pods able to communicate cross-subnet but not in same subnet. then I edited (remove Though I am not expert in iptable, please help me to find out Thanks ALL for helping me to do proper analysis, also for your valuable time. Not yet finished, still pending testing calico policy :) 👍 |
|
Closing this issue & opening another issue in k8-policy. Not sure if new policy issue any relation with this solution taken to resolve. Same type of scenario I tested in kubernetes with calico and it woks but facing challenges in OpenShift. In that case difference between kubernetes and openshift is, In OpenShift use Router & in Kubernetes it uses Ingress controler. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Version
CentOS 3.7
OC 3.6
Ansible 2.3
docker 1.12.6
kubectl 1.6.1
NO policy setup as of now
Ansible hostfile
POD to Service no response
POD to POD response OK
The text was updated successfully, but these errors were encountered: