Can we have an API to push data to the server?

[rbanks-willow]

If I misunderstand anything here, or the functionality already exists, I'll let you correct my misunderstanding...

At the moment, the OPAL server appears to be concerned with only fetching data. The data/config endpoint takes a request that defines a set of callback endpoints, and the server then hits those endpoints to fetch data and patch the data in OPA.

This means the flow is

[system] -> [post /data/config] -> [server calls external url] -> [waits for response] -> [patches data]

If I already have an event based system, I could easily add a topic listener that then pushes relevant updates directly to the server, removing the need for the callback approach. The flow would then be

[system] -> [my event subscriber] -> [post /data/config] -> [server patches data]

If the Entries schema was extended to have an option to supply JSON directly, rather than just a URL callback, that would be helpful for integration and reduce the need to build specific data extraction endpoints for my various data sources.

[asafc]

Hi @rbanks-willow :)
Actually, you did misunderstand a little how the OPAL architecture works.

• The OPAL server is a coordinator - it is concerned with letting OPAL clients know what data and policy they should fetch, but it does not fetch data by itself.
• The OPAL client is an aggregator of state. It sits next to the OPA agent and manages the OPA state, keeping it in sync by subscribing to realtime updates, and aggregating state from multiple sources.

The main reasons for this architecture are:

• Compliance and data privacy - it is not always the case that all nodes in the network should be able to access all the data. Sometimes we want the OPA agent to contain sensitive billing data, because that is what the authorization policy requires, but we do not want the OPAL server (or other OPAL clients who don't require this billing information) to access this data. Furthermore, if the opal server and client sits in separate clouds/vpcs, since the client typically sits close to your services (e.g: as a sidecar) - a lot of the time you can keep sensitive data in your local network and it's better for compliance (e.g: GDPR, etc).
  Performance/reliability: websockets should be used with caution - they are great for lightweight message passing - but for big data documents it is more reliable to use HTTP/S. So we only pass the update metadata over websockets.
• Client can be stateless: only state the client needs is declarative - what are the topics for policy and dynamic data i need to subscribe to. Or in other words, what is the relevant state this client needs to keep in sync in OPA's cache.

Does this help? I can elaborate further if you need me to :)

[rbanks-willow]

Ah, got it. That helps, thanks.