Support service type ClusterIP

[kshafiee]

No description provided.

[shettyg]

We do support ClusterIP. But there is a gap wherein if you create a nodeport, we do not create clusterIP. Is that what you meant?

[kshafiee]

I can see that ClusterIP is supported. But, even though PodIP (EndpointIP) is reachable, ClusterIP is not reachable:

@shettyg any ideas if the ClusterIPs have been tested to see if they're accessible?

[shettyg]

@kshafiee

They do work. I am trying to understand why it is not working in your case. What is the netmask for 10.100.65.247? One possibility is that In this case,there is no route to 10.100.65.247 from your host.

What is the minion or master subnet for this host? When we run the master-init/minion-init, we give a cluster wide subnet. Is your cluster IP provided in the same large subnet?

You can do a 'route -n' and that should give a hint.

[shettyg]

You can also try to reach the clusterIP from inside a pod. Does that work?

[kshafiee]

@shettyg
I got the above snapshot with these settings:
ovn-k8s-overlay master-init
--cluster-ip-subnet="10.244.0.0/16"
--master-switch-subnet="10.244.1.0/24"
--node-name="$HOSTNAME"

When I pass "10.100.0.0/16" as --cluster-ip-subnet:
ovn-k8s-overlay master-init
--cluster-ip-subnet="10.100.0.0/16"
--master-switch-subnet="10.244.0.0/16"
--node-name="$HOSTNAME"

ClusterIP works on the master node, but not on the slave node, nor from inside the pod:

[shettyg]

@kshafiee

Summary:
You need --master-switch-subnet to be a subnet of --cluster-ip-subnet. You also need the clusterIP provided by k8s to be also a subnet of --cluster-ip-subnet.

Details:

What we do is that, on each node, we create a OVS internal port (or a host interface) and assign it an IP address in the subnet assigned for that node. For e.g., if --master-switch-subnet is 192.168.1.0/24, then we assign 192.168.1.2/24 to that host interface. And 192.168.1.1/24 would be a OVN virtual router IP address. We Also add a route entries on that host - which say that 192.168.0.0/16 is reachable via 192.168.1.1/24. So if you do a curl 192.168.X.Y from host, it enters the OVN logical pipeline.

Now, if the clusterIP for k8s is also a subnet of 192.168.0.0/16, the route entry that we added is automatically valid for a VIPs too. i.e you can access a VIP from a host. But irrespective of this, a VIP should always be accessible from inside the POD.

You can either create your clusterIP to be one of the subnets of 192.168.0.0/16 or you can add a manual route that says that your cluster subnet is reachable from 192.168.1.1/24 from master and 192.168.2.1/24 from minion1, 192.168.3.1/24 from minion2 etc

Once you get the above correctly, let me know how it goes. Also, let me know whether your pod from which you are trying to access clusterIP is in a windows node or a linux node.

[shettyg]

I would also suggest to create a services backing pods as endpoints for your testing to remove some other possible issues.

[kshafiee]

Thanks @shettyg

on master I have:
ovn-k8s-overlay master-init --cluster-ip-subnet="10.0.0.0/8" --master-switch-subnet="10.244.1.0/24" --node-name="$HOSTNAME"

on slave I have:
windows-init.exe windows-init --node-name $HOSTNAME --minion-switch-subnet "10.244.9.0/24" --cluster-ip-subnet "10.0.0.0/8"

On master I can access both PodIP & ServiceIP:

But, inside the Pod ServiceIP is not reachable (I can only access PodIP):

The routing table inside the pod is (no entry for 10.0.0.0/8):

However, the serviceIP is not even reachable on the slave node, even though there's an entry for 10.0.0.0/8. The routing table on the slave node is:

[kshafiee]

BTW, EndpointIP (PodIP) and ServiceIP for the nginx are:

[shettyg]

@alinbalutoiu @aserdean

Any tips on how to debug this? Looks like NAT is not taking place on Windows host.

[aserdean]

We added some stateless NAT-ing when we implemented the python agent that emulates the CNI plugin.

@alinbalutoiu is working on removing the agent and implementing a CNI plugin.

We will be back with updates when we address this issue.

[aserdean]

@shettyg this should be closed by PR: ace1c6b . Sorry it took so much to send the PR.