datum-cloud-viewer 403s on trafficprotectionpolicies while httpproxies succeeds (stale deployed gateway-viewer)

[yahyafakhroji]

Summary

Users bound to the org datum-cloud-viewer role get 403 on trafficprotectionpolicies.networking.datumapis.com but 200 on httpproxies when loading the AI Edge listing in cloud-portal. Both permissions originate from the same gateway-viewer role via the same inheritance chain, so this is a live-state/deployment drift issue, not a source-config bug.

Investigation (source is correct)

Inheritance chain, all present in source:

datum-cloud-viewer  (datum: config/assignable-organization-roles/roles/datum-cloud-viewer.yaml)
  └─ networking.datumapis.com-viewer        (NSO: config/iam/roles/networking-viewer.yaml)
       └─ networking.datumapis.com-gateway-viewer  (NSO: config/iam/roles/gateway-viewer.yaml)
            ├─ httpproxies.{list,get,watch}                 ← since v0.3.0 (commit 3163220)
            └─ trafficprotectionpolicies.{list,get,watch}   ← since v0.9.0 (commit b19af94)

• Deployed to milo-system via Flux core-control-plane-resources (infra: apps/network-services-operator/control-plane/base/core-control-plane-resources.yaml), sourced from OCIRepository network-services-operator-kustomize, component ../iam.
• That OCIRepository tracks latest in base/staging/production (no old tag: pin anywhere in infra).

The httpproxies-✓ / WAF-✗ window maps exactly to a deployed gateway-viewer revision in [v0.3.0, v0.9.0).

Likely cause

Live gateway-viewer Role in milo-system is stale vs. the tracked bundle — Flux hasn't reconciled, is failing, or resolved a stale OCI artifact.

Verification / remediation

kubectl get role networking.datumapis.com-gateway-viewer -n milo-system -o yaml | grep trafficprotection
# if missing:
flux reconcile source oci network-services-operator-kustomize -n <flux-ns>
flux reconcile kustomization core-control-plane-resources -n <flux-ns>
# also inspect: flux get kustomization core-control-plane-resources  (apply errors + resolved digest)

Affected org observed: organization-asdasd-2zxpqq (identify env: prod/staging/preview).

Related work (cloud-portal, already fixed on feat/rbac-ai-edge)

• http-proxy.service.ts fetchList: WAF policy list wrapped in .catch so a trafficprotectionpolicies 403 no longer hard-fails the whole AI Edge list.
• edge/index.tsx: added trafficprotectionpolicies:list check → Protection column shows "—" + tooltip instead of a misleading "Disabled".

Optional follow-up (robustness, not a fix)

Consider having datum-cloud-viewer inherit gateway-viewer directly (shorter chain). Won't resolve the 403 if the deployed role is stale.

[yahyafakhroji]

@scotwells @JoseSzycho Can you give me a clue as to the repo that relates to trafficprotectionpolicies permissions for this issue? Because I found that some permissions are setup in the datum repo, but the networking modules use the network-services-operator repo.

[JoseSzycho]

@yahyafakhroji could you give me the CURL that gives you the authorization error?

[yahyafakhroji]

@JoseSzycho this is the CURL

curl 'https://api.staging.env.datum.net/apis/resourcemanager.miloapis.com/v1alpha1/projects/{project_name}/control-plane/apis/networking.datumapis.com/v1alpha/namespaces/default/trafficprotectionpolicies' \
  -H 'Accept: application/json, text/plain, */*' \
  -H 'Authorization: Bearer {access_token}' \
  -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36'

[JoseSzycho]

• trafficprotectionpolicies.networking.datumapis.com CRD is installed
• networking.datumapis.com-viewer is installed in the datum-cloud namespace and status type is ready
• networking.datumapis.com-viewer inherits the networking.datumapis.com/trafficprotectionpolicies.get permission from the networking.datumapis.com-gateway-viewer role

The networking.datumapis.com-gateway-viewer role in the datum-cloud namespace, it is outdated. When you compare the stored role, with the GitOps declared one:

• https://github.com/datum-cloud/network-services-operator/blob/main/config/iam/roles/gateway-viewer.yaml
  The permissions related to networking.datumapis.com/trafficprotectionpolicies are missing in the staging cluster

# networking.datumapis.com-gateway-viewer installed in the staging datum-cloud namespace
 ~ kubectl -n datum-system exec deployment/kubectl -it -- kubectl --kubeconfig /etc/kubernetes/auth/kubeconfig get role.iam.miloapis.com networking.datumapis.com-gateway-viewer -n datum-cloud -o yaml
apiVersion: iam.miloapis.com/v1alpha1
kind: Role
metadata:
  annotations:
    kubernetes.io/description: View access to gateway resources
    kubernetes.io/display-name: Gateway Viewer
  creationTimestamp: "2025-11-07T01:57:35Z"
  finalizers:
  - iam.miloapis.com/openfga-role
  generation: 1
  labels:
    kustomize.toolkit.fluxcd.io/name: assignable-organization-roles
    kustomize.toolkit.fluxcd.io/namespace: preview-pr-746
  name: networking.datumapis.com-gateway-viewer
  namespace: datum-cloud
  resourceVersion: "978687947"
  uid: 25e8c332-89d3-47c7-94b4-206f37f6de20
spec:
  includedPermissions:
  - gateway.networking.k8s.io/gatewayclasses.list
  - gateway.networking.k8s.io/gatewayclasses.get
  - gateway.networking.k8s.io/gatewayclasses.watch
  - gateway.networking.k8s.io/gateways.list
  - gateway.networking.k8s.io/gateways.get
  - gateway.networking.k8s.io/gateways.watch
  - gateway.networking.k8s.io/httproutes.list
  - gateway.networking.k8s.io/httproutes.get
  - gateway.networking.k8s.io/httproutes.watch
  - gateway.networking.k8s.io/backendtlspolicies.list
  - gateway.networking.k8s.io/backendtlspolicies.get
  - gateway.networking.k8s.io/backendtlspolicies.watch
  - gateway.envoyproxy.io/backends.list
  - gateway.envoyproxy.io/backends.get
  - gateway.envoyproxy.io/backends.watch
  - gateway.envoyproxy.io/backendtrafficpolicies.list
  - gateway.envoyproxy.io/backendtrafficpolicies.get
  - gateway.envoyproxy.io/backendtrafficpolicies.watch
  - gateway.envoyproxy.io/httproutefilters.list
  - gateway.envoyproxy.io/httproutefilters.get
  - gateway.envoyproxy.io/httproutefilters.watch
  - gateway.envoyproxy.io/securitypolicies.list
  - gateway.envoyproxy.io/securitypolicies.get
  - gateway.envoyproxy.io/securitypolicies.watch
  - discovery.k8s.io/endpointslices.list
  - discovery.k8s.io/endpointslices.get
  - discovery.k8s.io/endpointslices.watch
  - networking.datumapis.com/httpproxies.list
  - networking.datumapis.com/httpproxies.get
  - networking.datumapis.com/httpproxies.watch
  launchStage: Beta
status:
  conditions:
  - lastTransitionTime: "2026-03-19T21:37:48Z"
    message: Role reconciled successfully with OpenFGA.
    observedGeneration: 1
    reason: ReconciliationSuccessful
    status: "True"
    type: Ready
  - lastTransitionTime: "2026-03-19T21:37:48Z"
    message: All permissions validated successfully.
    observedGeneration: 1
    reason: ValidationSuccessful
    status: "True"
    type: PermissionsValid
  effectivePermissions:
  - discovery.k8s.io/endpointslices.get
  - discovery.k8s.io/endpointslices.list
  - discovery.k8s.io/endpointslices.watch
  - gateway.envoyproxy.io/backends.get
  - gateway.envoyproxy.io/backends.list
  - gateway.envoyproxy.io/backends.watch
  - gateway.envoyproxy.io/backendtrafficpolicies.get
  - gateway.envoyproxy.io/backendtrafficpolicies.list
  - gateway.envoyproxy.io/backendtrafficpolicies.watch
  - gateway.envoyproxy.io/httproutefilters.get
  - gateway.envoyproxy.io/httproutefilters.list
  - gateway.envoyproxy.io/httproutefilters.watch
  - gateway.envoyproxy.io/securitypolicies.get
  - gateway.envoyproxy.io/securitypolicies.list
  - gateway.envoyproxy.io/securitypolicies.watch
  - gateway.networking.k8s.io/backendtlspolicies.get
  - gateway.networking.k8s.io/backendtlspolicies.list
  - gateway.networking.k8s.io/backendtlspolicies.watch
  - gateway.networking.k8s.io/gatewayclasses.get
  - gateway.networking.k8s.io/gatewayclasses.list
  - gateway.networking.k8s.io/gatewayclasses.watch
  - gateway.networking.k8s.io/gateways.get
  - gateway.networking.k8s.io/gateways.list
  - gateway.networking.k8s.io/gateways.watch
  - gateway.networking.k8s.io/httproutes.get
  - gateway.networking.k8s.io/httproutes.list
  - gateway.networking.k8s.io/httproutes.watch
  - networking.datumapis.com/httpproxies.get
  - networking.datumapis.com/httpproxies.list
  - networking.datumapis.com/httpproxies.watch
  observedGeneration: 1

[JoseSzycho]

@yahyafakhroji based on the previous investigation, I can see that the networking.datumapis.com-gateway-viewer is stored in the milo-system namespace and does have the missing permissions.

 ~ kubectl -n datum-system exec deployment/kubectl -it -- kubectl --kubeconfig /etc/kubernetes/auth/kubeconfig get role.iam.miloapis.com networking.datumapis.com-gateway-viewer -n milo-system -o yaml
apiVersion: iam.miloapis.com/v1alpha1
kind: Role
metadata:
  annotations:
    kubernetes.io/description: View access to gateway resources
    kubernetes.io/display-name: Gateway Viewer
  creationTimestamp: "2026-03-19T21:30:21Z"
  finalizers:
  - iam.miloapis.com/openfga-role
  generation: 1
  labels:
    kustomize.toolkit.fluxcd.io/name: network-services-operator-core-control-plane-resources
    kustomize.toolkit.fluxcd.io/namespace: datum-system
  name: networking.datumapis.com-gateway-viewer
  namespace: milo-system
  resourceVersion: "978687977"
  uid: 1dcef889-ebd5-48ce-bcaa-16cef786fc6c
spec:
  includedPermissions:
  ...
  ...
  ...
  - networking.datumapis.com/trafficprotectionpolicies.list
  - networking.datumapis.com/trafficprotectionpolicies.get
  - networking.datumapis.com/trafficprotectionpolicies.watch
  launchStage: Beta
status:
  conditions:
  - lastTransitionTime: "2026-03-19T21:37:49Z"
    message: Role reconciled successfully with OpenFGA.
    observedGeneration: 1
    reason: ReconciliationSuccessful
    status: "True"
    type: Ready
  - lastTransitionTime: "2026-03-19T21:37:49Z"
    message: All permissions validated successfully.
    observedGeneration: 1
    reason: ValidationSuccessful
    status: "True"
    type: PermissionsValid
  effectivePermissions:
  ...
  ...
  - networking.datumapis.com/trafficprotectionpolicies.get
  - networking.datumapis.com/trafficprotectionpolicies.list
  - networking.datumapis.com/trafficprotectionpolicies.watch
  observedGeneration: 1

The missing permissions where added in this commit

• datum-cloud/network-services-operator@b19af94

But for some reason, the installed role datum-cloud-viewer never receive the updates when the networking.datumapis.com-gateway-viewer role got updated with the new trasfficprotectionpolicies permissions.

Seems like a bug in our authorization system, I'm carting a issue for tracking that.

[JoseSzycho]

Root cause confirmed and tracked in the auth provider: milo-os/openfga-provider#111.

Working in the fix right now

[JoseSzycho]

@yahyafakhroji the fix is in staging. Now the viewer role has all of the inherited roles.

However, I'm confused, why the only users that use the datum-cloud-viewer are you and @mattdjenkinson ? Is that a custom role that you crated for testing? The datum-cloud-viewer have missing namespaces for each inheritedRoles, which causes inheritance to not work correctly.

The correct role is the viewer which lives in the datum-cloud namespace

[yahyafakhroji]

@JoseSzycho thanks, but just checking with this curl. my account still denied from trafficprotectionpolicies

curl \
  -X POST \
  'https://api.staging.env.datum.net/apis/resourcemanager.miloapis.com/v1alpha1/projects/test-rbac-dpfpw7/control-plane/apis/authorization.k8s.io/v1/selfsubjectaccessreviews' \
  -H 'Accept: application/json, text/plain, */*' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer {acccess_token}' \
  -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36' \
  -d $'{
  "apiVersion": "authorization.k8s.io/v1",
  "kind": "SelfSubjectAccessReview",
  "spec": {
    "resourceAttributes": {
      "namespace": "default",
      "verb": "list",
      "group": "networking.datumapis.com",
      "resource": "trafficprotectionpolicies"
    }
  }
}'

[JoseSzycho]

@yahyafakhroji I think that you got this datum-viewer role when someone invited you to an organization. The datum-viewer role is a stale role, which is incorrect, and don't have all of the permissions.

You and @mattdjenkinson are experiencing the same issue. In staging, when you invite someone to join an org, only use the viewer, editor, or owner role.

I'm working into deleting the stales roles.