Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cilium does not recognise identity thus got policy denied for a within namespace call on the same POD. #28320

Closed
2 tasks done
peterbosalliandercom opened this issue Sep 28, 2023 · 4 comments
Labels
kind/bug This is a bug in the Cilium logic. kind/community-report This was reported by a user in the Cilium community, eg via Slack. need-more-info More information is required to further debug or fix the issue. sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages. sig/policy Impacts whether traffic is allowed or denied based on user-defined policies. stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale.

Comments

@peterbosalliandercom
Copy link

peterbosalliandercom commented Sep 28, 2023

Is there an existing issue for this?

  • I have searched the existing issues

What happened?

When deploying the following cnp it appears that Cilium incorrectly identifies the calls within namespace traffic as (world) entity. We expect it to accept calls within the same namespace.

We get the following drops:

Sep 28 07:35:18.643: argocd/argocd-redis-ha-announce-2:6379 (world) <> argocd/argocd-redis-ha-server-2:43596 (ID:36192) Policy denied DROPPED (TCP Flags: ACK)
Sep 28 07:35:18.884: argocd/argocd-redis-ha-announce-2:6379 (world) <> argocd/argocd-redis-ha-server-2:43596 (ID:36192) Policy denied DROPPED (TCP Flags: ACK)
Sep 28 07:35:19.632: argocd/argocd-redis-ha-announce-2:6379 (world) <> argocd/argocd-redis-ha-server-2:43604 (ID:36192) Policy denied DROPPED (TCP Flags: ACK, PSH)
Sep 28 07:35:19.645: argocd/argocd-redis-ha-announce-0:6379 (world) <> argocd/argocd-redis-ha-server-0:53890 (ID:30596) Policy denied DROPPED (TCP Flags: ACK, PSH)
Sep 28 07:35:19.709: argocd/argocd-redis-ha-announce-0:6379 (world) <> argocd/argocd-redis-ha-server-0:53888 (ID:30596) Policy denied DROPPED (TCP Flags: ACK, PSH)
Sep 28 07:35:19.709: argocd/argocd-redis-ha-announce-0:6379 (world) <> argocd/argocd-redis-ha-server-0:53888 (ID:30596) Policy denied DROPPED (TCP Flags: ACK)
Sep 28 07:35:19.658: argocd/argocd-redis-ha-announce-1:6379 (world) <> argocd/argocd-redis-ha-server-1:57626 (ID:30111) Policy denied DROPPED (TCP Flags: ACK, PSH)
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: argocd-policy
  namespace: argocd
specs:
- egress:
  - toEntities:
    - all
  endpointSelector:
    matchLabels: {}
  ingress:
  - fromEndpoints:
    - matchLabels:
        k8s:io.kubernetes.pod.namespace: argocd
  - fromEntities:
    - remote-node
    - health
    - kube-apiserver
    - cluster

The weird thing is that the drop also occurs when deploying this netpol instead of cnp and using the same pod.

- apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata:
    labels:
      capsule.clastix.io/network-policy: "0"
      capsule.clastix.io/tenant: itest-28022
    name: capsule-itest-28022-0
    namespace: itest-28022-28022
  spec:
    ingress:
    - from:
      - namespaceSelector:
          matchLabels:
            tenant: itest-28022
    podSelector: {}
    policyTypes:
    - Ingress
  status: {}

So if you do a call to the same pod in the same namespace it fails with a drop on (world).

It appears to be related to one of these:
#21083
#14284
#23911

Cilium Version

Client: 1.13.4 4061cdf 2023-06-14T03:37:39+00:00 go version go1.19.10 linux/amd64
Daemon: 1.13.4 4061cdf 2023-06-14T03:37:39+00:00 go version go1.19.10 linux/amd64

Kernel Version

Linux ip-10-116-83-84.eu-central-1.compute.internal 5.10.186-179.751.amzn2.x86_64 #1 SMP Tue Aug 1 20:51:38 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Kubernetes Version

Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-14T09:47:38Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"25+", GitVersion:"v1.25.12-eks-2d98532", GitCommit:"0aa16cf4fac4da27b9e9e9ba570b990867f6a3d8", GitTreeState:"clean", BuildDate:"2023-07-28T16:52:04Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.27) and server (1.25) exceeds the supported minor version skew of +/-1

Sysdump

cilium-sysdump-20230928-095303.zip

Relevant log output

No response

Anything else?

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct
@peterbosalliandercom peterbosalliandercom added kind/bug This is a bug in the Cilium logic. kind/community-report This was reported by a user in the Cilium community, eg via Slack. needs/triage This issue requires triaging to establish severity and next steps. labels Sep 28, 2023
@peterbosalliandercom peterbosalliandercom changed the title Cilium does not recognise identity thus got policy denied argocd redis Cilium does not recognise identity thus got policy denied for a within namespace call. Sep 28, 2023
@peterbosalliandercom peterbosalliandercom changed the title Cilium does not recognise identity thus got policy denied for a within namespace call. Cilium does not recognise identity thus got policy denied for a within namespace call on the same POD. Sep 28, 2023
@peterbosalliandercom
Copy link
Author

This seems to be fixed in this issue? #27798
Stil needs to be confirmed

@ti-mo
Copy link
Contributor

ti-mo commented Oct 4, 2023

@peterbosalliandercom #27798 fixes Pods calling themselves through a k8s Service. Is that the case here as well? The logs show argocd-redis-ha-announce-2 -> argocd-redis-ha-server-2, I assume those are different Pods?

@ti-mo ti-mo added sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages. sig/policy Impacts whether traffic is allowed or denied based on user-defined policies. and removed needs/triage This issue requires triaging to establish severity and next steps. labels Oct 4, 2023
@julianwiedmann julianwiedmann added the need-more-info More information is required to further debug or fix the issue. label Nov 24, 2023
Copy link

This issue has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.

@github-actions github-actions bot added the stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. label Jan 24, 2024
Copy link

github-actions bot commented Feb 8, 2024

This issue has not seen any activity since it was marked stale.
Closing.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Feb 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug This is a bug in the Cilium logic. kind/community-report This was reported by a user in the Cilium community, eg via Slack. need-more-info More information is required to further debug or fix the issue. sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages. sig/policy Impacts whether traffic is allowed or denied based on user-defined policies. stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale.
Projects
None yet
Development

No branches or pull requests

3 participants