Support Calico CNI for ambient #40973

irisdingbj · 2022-09-14T03:16:16Z

Bug Description

Have ambient profile installed on a single node k8s cluster as below:

NAME                                    READY   STATUS    RESTARTS   AGE
istio-cni-node-7h47k                    1/1     Running   0          5d18h
istio-ingressgateway-77ff84cdf6-2mdbp   1/1     Running   0          5d18h
istiod-576df488d5-j857h                 1/1     Running   0          5d18h
ztunnel-hd4wj                           1/1     Running   0          5d18h

Have deployed applications following the blog post in default ns

$ kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml
$ kubectl apply -f https://raw.githubusercontent.com/linsun/sample-apps/main/sleep/sleep.yaml
$ kubectl apply -f https://raw.githubusercontent.com/linsun/sample-apps/main/sleep/notsleep.yaml

sleep pod can communicate with product page without issue before joined ambient mesh:

kubectl exec deploy/sleep -- curl -s http://productpage:9080/ | head -n1
<!DOCTYPE html>

Join default ns into ambient mesh:

kubectl label namespace default istio.io/dataplane-mode=ambient
namespace/default labeled

sleep pod CAN NOT communicate with product page :

kubectl exec deploy/sleep -- curl -s -vvv http://productpage:9080/ | head -n1
* Could not resolve host: productpage
* Closing connection 0
command terminated with exit code 6

sleep pod CAN NOT communicate with product page via pod ip:

kubectl exec deploy/sleep -- curl -s -vvv http://172.16.138.12:9080/
*   Trying 172.16.138.12:9080...
* connect to 172.16.138.12 port 9080 failed: Operation timed out
* Failed to connect to 172.16.138.12 port 9080: Operation timed out
* Closing connection 0
command terminated with exit code 28

Version

istioctl version
client version: 0.0.0-ambient.191fe680b52c1754ee72a06b3e0d3f9d116f2e82
control plane version: 0.0.0
data plane version: 0.0.0-ambient.191fe680b52c1754ee72a06b3e0d3f9d116f2e82 (2 proxies)

Additional Information

No response

The text was updated successfully, but these errors were encountered:

irisdingbj · 2022-09-14T06:19:56Z

Thanks @howardjohn who pointed to me this is due to Calico CNI will revert IPtable rules.

PlatformLC · 2022-10-17T15:03:36Z

@dhawton, is there any detail or status update for supporting either calico or other bridging CNI plugin rather than kind CNI? Any materials or information could be shared will be appreciated.

dhawton · 2022-10-17T15:51:30Z

@dhawton, is there any detail or status update for supporting either calico or other bridging CNI plugin rather than kind CNI? Any materials or information could be shared will be appreciated.

Not at this point. We do support the CNIs in GKE (non-Calico and non-DPv2) and EKS... but I do not have anything more I can share at this moment.

dcw329 · 2022-11-25T16:27:27Z

I saw note that GKE with Calico does not work. I believe the default is deployment is with calico. Is there an alternative that should be used in Dev to test ambient with?

howardjohn · 2022-11-25T16:50:56Z

The default isn't calico unless you enable network policy when creating the cluster. If you just use the defaults in the UI it will work

…

On Fri, Nov 25, 2022, 8:27 AM Daniel Wayne ***@***.***> wrote: I saw note that GKE with Calico does not work. I believe the default is deployment is with calico. Is there an alternative that should be used in Dev to test ambient with? — Reply to this email directly, view it on GitHub <#40973 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXNXNYY3OULM6XMQEYTWKDSHTANCNFSM6AAAAAAQL7ZUH4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

DANic-git · 2023-09-14T14:11:45Z

Hello,

Is there any way to use ambient with calico?

chenwng · 2023-10-17T03:25:08Z

Hello,

Is there any way to use ambient with calico?

The following may help.

Upgrade the calico version to the latest.
Modify chainInsertMode to Append and workloadSourceSpoofing to Any in Calico FelixConfiguration.
Add the annotation cni.projectcalico.org/allowedSourcePrefixes: '["0.0.0.0/0"]' to your workload.

WoodyWoodsta · 2023-10-20T15:40:10Z

I'm attempting to lay istio over calico in my clusters. Calico unfortunately only supports 1.15 - and it appears as though Ambient would solve a lot of the knitting that has to be done to integrate the two technologies.

Would be great if ambient supported the current calico project!

KfreeZ · 2023-10-24T01:31:40Z

Hello,
Is there any way to use ambient with calico?

The following may help.

Upgrade the calico version to the latest.

Modify chainInsertMode to Append and workloadSourceSpoofing to Any in Calico FelixConfiguration.

Add the annotation cni.projectcalico.org/allowedSourcePrefixes: '["0.0.0.0/0"]' to your workload.

It works for me,
I see the annotation is included in https://github.com/istio/istio/blob/master/manifests/charts/ztunnel/templates/daemonset.yaml, so this step is not mandatory for the latest release.

chenwng · 2023-10-24T01:56:37Z

Hello,
Is there any way to use ambient with calico?

The following may help.

Upgrade the calico version to the latest.

Modify chainInsertMode to Append and workloadSourceSpoofing to Any in Calico FelixConfiguration.

Add the annotation cni.projectcalico.org/allowedSourcePrefixes: '["0.0.0.0/0"]' to your workload.

It works for me, I see the annotation is included in https://github.com/istio/istio/blob/master/manifests/charts/ztunnel/templates/daemonset.yaml, so this step is not mandatory for the latest release.

Do you mean the 3rd step is not necessary?

The issue I encountered was not with ztunnel pods but with my workloads. Without the annotation added to my workloads, egress traffic of my workload pods were dropped due to RPF. I believe 103 ip rule and routes in table 100 were causing this RPF failure. So I had to add the annotation to my workloads to bypass the RPF check by calico.

ip rule 
0:      from all lookup local
100:    from all fwmark 0x200/0x200 goto 32766
101:    from all fwmark 0x100/0x100 lookup 101
102:    from all fwmark 0x40/0x40 lookup 102
103:    from all lookup 100
...

ip route sh table 100 
192.168.1.69 dev cali30566341ea4 scope link 
192.168.1.76 via 192.168.126.2 dev istioin src 172.18.0.5

I suppose this 103 ip rule should be modified as it makes all egress traffic of pods served in ambient mode fail the RPF check.

KfreeZ · 2023-10-24T01:59:48Z

Hello,
Is there any way to use ambient with calico?

The following may help.

Upgrade the calico version to the latest.

Modify chainInsertMode to Append and workloadSourceSpoofing to Any in Calico FelixConfiguration.

Add the annotation cni.projectcalico.org/allowedSourcePrefixes: '["0.0.0.0/0"]' to your workload.

It works for me, I see the annotation is included in https://github.com/istio/istio/blob/master/manifests/charts/ztunnel/templates/daemonset.yaml, so this step is not mandatory for the latest release.

Do you mean the 3rd step is not necessary?

The issue I encountered was not with ztunnel pods but with my workloads. Without the annotation added to my workloads, egress traffic of my workload pods were dropped due to RPF. I believe 103 ip rule and routes in table 100 were causing this RPF failure. So I had to add the annotation to my workloads to bypass the RPF check by calico.
ip rule 
0:      from all lookup local
100:    from all fwmark 0x200/0x200 goto 32766
101:    from all fwmark 0x100/0x100 lookup 101
102:    from all fwmark 0x40/0x40 lookup 102
103:    from all lookup 100
...
ip route sh table 100 
192.168.1.69 dev cali30566341ea4 scope link 
192.168.1.76 via 192.168.126.2 dev istioin src 172.18.0.5
I suppose this 103 ip rule should be modified as it makes all egress traffic of pods served in ambient mode fail the RPF check.

Yes, I did not apply step 3 because I am using the ebpf redirection mode, so I think the ip rules is not a problem for me, maybe you could give it a try :)

chenwng · 2023-10-24T02:02:16Z

Hello,
Is there any way to use ambient with calico?

The following may help.

Upgrade the calico version to the latest.

Modify chainInsertMode to Append and workloadSourceSpoofing to Any in Calico FelixConfiguration.

Add the annotation cni.projectcalico.org/allowedSourcePrefixes: '["0.0.0.0/0"]' to your workload.

It works for me, I see the annotation is included in https://github.com/istio/istio/blob/master/manifests/charts/ztunnel/templates/daemonset.yaml, so this step is not mandatory for the latest release.

Do you mean the 3rd step is not necessary?
The issue I encountered was not with ztunnel pods but with my workloads. Without the annotation added to my workloads, egress traffic of my workload pods were dropped due to RPF. I believe 103 ip rule and routes in table 100 were causing this RPF failure. So I had to add the annotation to my workloads to bypass the RPF check by calico.
ip rule 
0:      from all lookup local
100:    from all fwmark 0x200/0x200 goto 32766
101:    from all fwmark 0x100/0x100 lookup 101
102:    from all fwmark 0x40/0x40 lookup 102
103:    from all lookup 100
...
ip route sh table 100 
192.168.1.69 dev cali30566341ea4 scope link 
192.168.1.76 via 192.168.126.2 dev istioin src 172.18.0.5
I suppose this 103 ip rule should be modified as it makes all egress traffic of pods served in ambient mode fail the RPF check.
Yes, I did not apply step 3 because I am using the ebpf redirection mode, so I think the ip rules is not a problem for me, maybe you could give it a try :)

Thanks. I was using the default iptables mode and haven't tried ebpf mode.

craigbox · 2023-12-21T01:14:02Z

Will #48212 address this?

dhawton · 2023-12-21T01:56:48Z

Will #48212 address this?

Yes

linsun · 2024-01-29T00:14:03Z

This is resolved, please try latest master or release 1.21 build, or wait for the official 1.21 release. see #48212

linsun · 2024-01-29T15:08:01Z

Sorry, just got confirmation from Eric that the change is not in beta 0 yet, but should be in -beta.1 since it was merged after beta.0 was released. I expect that the new build will be available in a day or 2.

istio-policy-bot added the area/ambient Issues related to ambient mesh label Sep 14, 2022

zirain mentioned this issue Sep 14, 2022

Upstream experimental-ambient branches to master #40879

Closed

9 tasks

howardjohn changed the title ~~Pods on the same node CAN NOT communicate when joined ambient mesh~~ Support Calico CNI for ambient Sep 15, 2022

PlatformLC mentioned this issue Sep 26, 2022

CNI encountered an error: failed to wait for resource - Calico CNI? #41153

Closed

linsun mentioned this issue Nov 30, 2022

Install Istio ambient failed - RKE + Calico #41191

Closed

SkyfireFrancisZ added the P1 label Jan 5, 2023

istio-policy-bot added the lifecycle/stale Indicates a PR or issue hasn't been manipulated by an Istio team member for a while label Feb 24, 2023

dhawton removed the lifecycle/stale Indicates a PR or issue hasn't been manipulated by an Istio team member for a while label Feb 24, 2023

istio-policy-bot added the lifecycle/stale Indicates a PR or issue hasn't been manipulated by an Istio team member for a while label Feb 24, 2023

dhawton added the lifecycle/staleproof Indicates a PR or issue has been deemed to be immune from becoming stale and/or automatically closed label Feb 24, 2023

istio-policy-bot removed the lifecycle/stale Indicates a PR or issue hasn't been manipulated by an Istio team member for a while label Feb 24, 2023

howardjohn removed the P1 label Jul 18, 2023

hzxuzhonghu added Ambient Beta Must have for Beta of Ambient Mesh and removed lifecycle/staleproof Indicates a PR or issue has been deemed to be immune from becoming stale and/or automatically closed labels Sep 12, 2023

istio-policy-bot closed this as completed Sep 12, 2023

istio-policy-bot added the lifecycle/automatically-closed Indicates a PR or issue that has been closed automatically. label Sep 12, 2023

dhawton removed the lifecycle/automatically-closed Indicates a PR or issue that has been closed automatically. label Sep 12, 2023

dhawton reopened this Sep 12, 2023

linsun closed this as completed Jan 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Calico CNI for ambient #40973

Support Calico CNI for ambient #40973

irisdingbj commented Sep 14, 2022 •

edited

irisdingbj commented Sep 14, 2022

PlatformLC commented Oct 17, 2022

dhawton commented Oct 17, 2022

dcw329 commented Nov 25, 2022

howardjohn commented Nov 25, 2022 via email

DANic-git commented Sep 14, 2023

chenwng commented Oct 17, 2023 •

edited

WoodyWoodsta commented Oct 20, 2023

KfreeZ commented Oct 24, 2023

chenwng commented Oct 24, 2023

KfreeZ commented Oct 24, 2023

chenwng commented Oct 24, 2023

craigbox commented Dec 21, 2023

dhawton commented Dec 21, 2023

linsun commented Jan 29, 2024

linsun commented Jan 29, 2024

Support Calico CNI for ambient #40973

Support Calico CNI for ambient #40973

Comments

irisdingbj commented Sep 14, 2022 • edited

Bug Description

Version

Additional Information

irisdingbj commented Sep 14, 2022

PlatformLC commented Oct 17, 2022

dhawton commented Oct 17, 2022

dcw329 commented Nov 25, 2022

howardjohn commented Nov 25, 2022 via email

DANic-git commented Sep 14, 2023

chenwng commented Oct 17, 2023 • edited

WoodyWoodsta commented Oct 20, 2023

KfreeZ commented Oct 24, 2023

chenwng commented Oct 24, 2023

KfreeZ commented Oct 24, 2023

chenwng commented Oct 24, 2023

craigbox commented Dec 21, 2023

dhawton commented Dec 21, 2023

linsun commented Jan 29, 2024

linsun commented Jan 29, 2024

irisdingbj commented Sep 14, 2022 •

edited

chenwng commented Oct 17, 2023 •

edited