OPA deployment split brain

[GalHalupSeaLights]

We moved our deployment from a single instance deployment on an EC2 instance which had a single OPA service on each EC2 instance to k8s.
The current architecture is that we have some micro-service that pushes the rego policy and data assignments using HTTP calls to the OPA service on each EC2 instance. meaning every-time there is any change in the data the "auth service" on the ec2 instance is incharge to update the OPA services on his EC2 instances.

In the new architecture we move to k8s and we would like to have at least 2 instances of OPA service on the cluster.
The issue we have is that we get a split brain situation meaning our "auth service" that is in charge to push the data and policy to the OPA service may push to only one of the OPA services and the 2 services may not be updated with latest assignment.

What is the right solution for this kind of architecture?

[orweis]

Maybe try OPAL (https://github.com/permitio/opal) ?
You can use it's event-driven nature to update both OPA instances from a single endpoint, and it provides health-checks, readiness policy, and callbacks to track the state of each of the OPA instances

[charlieegan3]

Since you already have a custom service to update OPA instances, you might also be interested to look into the Delta Bundle functionality: https://www.openpolicyagent.org/docs/latest/management-bundles/#delta-bundles. It wouldn't be a REST call, but would allow fast updates many OPA instances.

[GalHalupSeaLights]

I was thinking that since, I already have the custom service I can use it as the "bundle service API" and expose a new bundle API.
this was instead of pushing each opa service will pull the data from the auth service.

[charlieegan3]

👍 depending on how quickly you need the updates, you might not even need delta bundles.

Here's an example implementation that might be useful: https://github.com/ashutosh-narkar/example-delta-bundle-server

[chendrix]

Styra -- the creators and maintainers of OPA -- Also have a commercial product called Enterprise OPA with a bunch of pre-written data source integrations that will be able to fetch data out of the box.

Here's a tutorial for using Kafka https://docs.styra.com/enterprise-opa/tutorials/kafka