Review casbin token policies synchronization #2486
Description
We have noticed a random issue with API Tokens in the casbin synchronization of pods after a deployment. In some cases, token policies are not synced and some pods deny access to some resources, whereas others do. For example, when creating a workflow from the CLI, sometimes we can see this in our logs, followed by a operation not allowed error in the console:
2025-10-28 16:31:00 {"level":"info","ts":1761669060.986087,"msg":"[authZ] checking authorization","sub":"api-token:d9ed4f84-1d30-457e-b868-9fb3bb489686","operation":"/controlplane.v1.WorkflowService/Create"}
2025-10-28 16:31:00 {"level":"info","ts":1761669060.9901664,"msg":"[authZ] policy not found","sub":"api-token:d9ed4f84-1d30-457e-b868-9fb3bb489686","operation":"/controlplane.v1.WorkflowService/Create","resource":"workflow","action":"create"}
In these cases, a pod restart (k8s delete pod) fixes the issue.
The initialization code might be running into some race condition.