How to read data from API, DB, external services, etc.?

[hongbo-miao]

Based on OPAL architecture, opal-client is able to read data through API, DB, external services, etc., while opal-server is reading policy from a git repo.

However, the docker compose demo seems opal-server read both data and policy from same repo opal-example-policy-repo.

I found a OPAL_DATA_CONFIG_SOURCES in the opal-server, but based on its description

OPAL_DATA_CONFIG_SOURCES configures from where the opal client should initially fetch data (when it first goes up, after disconnection, etc).
the data sources represents from where the opal clients should get a "complete picture" of the data they need.
after the initial sources are fetched, the client will subscribe only to update notifications sent by the server.

this is only for initial fetching data or after long disconnection.

So my question is how to read data from API, DB, external services, etc.? Thanks

Question 1

In the demo, it is

OPAL_POLICY_REPO_URL=https://github.com/authorizon/opal-example-policy-repo

But how does it know the policy filename is rbac.rego and data file is data.json? Does it read all rego files as policy, and all JSON files as data by default?

Question 2

What if we want to read data from different places like API, DB, external services, etc. that are different with policy in git repo?
I found OPAL_DATA_CONFIG_SOURCES, but in the demo, it is pointing to OPAL server, instead of an external source.

OPAL_DATA_CONFIG_SOURCES={"config":{"entries":[{"url":"http://host.docker.internal:7002/policy-data","topics":["policy_data"]}]}}

[orweis]

Hi :)

"seems opal-server read both data and policy from same repo"
Just like with OPA bundles you can include data (aka static data)

how to read data from API, DB, external services, etc.?
By providing a DataSourceEntry as part of a DataUpdate (aka an update event) you can instruct OPAL-clients to use their data-fetchers to retrieve data from multiple sources.
You can trigger such an event using the CLI as shown here
And you can write your own code to create other DataUpdates of your choice.

OPAL currently comes with an HTTP based data fetcher (e.g. good for http API calls) To add more types of data fetchers you can implement them as detailed in this guide

Question 1:
The opal-server searches the repo for files according to the config given to it
The defaults re indicated in as OPA_FILE_EXTENSIONS = ('.rego', '.json') [when used in the command-line prefix as OPAL_OPA_FILE_EXTENSIONS]

In the updates themselves The clients will receive new sources to fetch data as part of the update - these don't (and probably shouldn't) have to be the same ones as in the static data provided in the repo.

Question 2:
As I wrote above you can point the OPAL-clients to whichever URL (or urls) you'd like as part of the update event.
If there's an http server behind the URL it would work out of the box, if there's a different server - you can implement a new type of fetcher to support it.

[orweis]

There's no such ENV var for the opal-client

You can either set it as an ENV-VAR for OPAL-server - i.e. 'OPAL_DATA_CONFIG_SOURCES'
for example OPAL_DATA_CONFIG_SOURCES={"config":{"entries":[{"url":"http://YOUR-DOMAIN-OF-CHOICE.COM","topics":["policy_data"]}]}} which acts as a default

Or you can have it as part of the event you send to the clients as shown here via CLI or your own Python code.

With the CLI it would look something like this - where --src-url indicates the URL to be sent as part fo the update.
And -t indicates topics

opal-client publish-data-update $token --src-url http://mybillingserver.com/users -t users -t billing --src-config '{"headers":{"authorization":"bearer secret-token"}}'

[hongbo-miao]

Here is the best try based on my understanding:

1. Deleted the data.json from a forked version of opal-example-policy-repo.
2. Hosted a server which returns that json I just deleted as an object at https://e5d4bcbf32aa.ngrok.io/data
3. Updated my OPAL_DATA_CONFIG_SOURCES to '{"config":{"entries":[{"url":"https://e5d4bcbf32aa.ngrok.io/data","topics":["policy_data"]}]}}'
4. Removed everything and did a fresh restart opal-server and opal-client.

But seems they are not reading data from this new endpoint, because I expect the result decision is same as true before when the data is in the git repo, but I got false.

[asafc]

@hongbo-miao regarding this specific issue: it should work - because the data from the example rego is written to the root data document /, and that is where you are putting the dynamic data.

can you please attach your OPAL client/server logs?
Also - can you attach your OPA logs?

If you would run inline OPA it would be a matter of adding OPAL_INLINE_OPA_LOG_FORMAT=http, however with standalone OPA you'll need to configure logs differently - maybe try this?

[hongbo-miao]

Tried again, still same 😢
Here are the logs:

opal-server log

[2021-07-02 19:50:18 +0000] [1] [INFO] Listening at: http://0.0.0.0:7002 (1)
36
[2021-07-02 19:50:18 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
35
[2021-07-02 19:50:18 +0000] [8] [INFO] Booting worker with pid: 8
34
[2021-07-02 19:50:18 +0000] [9] [INFO] Booting worker with pid: 9
33
[2021-07-02 19:50:18 +0000] [10] [INFO] Booting worker with pid: 10
32
[2021-07-02 19:50:19 +0000] [11] [INFO] Booting worker with pid: 11
31
�[32m2021-07-02T19:50:21.190919+0000�[0m |�[34mopal_common.authentication.signer       �[0m|�[1m INFO  | OPAL was not provided with JWT encryption keys, cannot verify api requests!�[0m
30
�[32m2021-07-02T19:50:21.226802+0000�[0m |�[34mopal_common.authentication.signer       �[0m|�[1m INFO  | OPAL was not provided with JWT encryption keys, cannot verify api requests!�[0m
29
�[32m2021-07-02T19:50:21.311103+0000�[0m |�[34mopal_common.authentication.signer       �[0m|�[1m INFO  | OPAL was not provided with JWT encryption keys, cannot verify api requests!�[0m
28
�[32m2021-07-02T19:50:21.391912+0000�[0m |�[34mopal_server.server                      �[0m|�[1m INFO  | triggered startup event�[0m
27
�[32m2021-07-02T19:50:21.392111+0000�[0m |�[34mopal_common.topics.publisher            �[0m|�[1m INFO  | started topic publisher�[0m
26
�[32m2021-07-02T19:50:21.392473+0000�[0m |�[34mopal_server.server                      �[0m|�[1m INFO  | leadership lock acquired, leader pid: 9�[0m
25
�[32m2021-07-02T19:50:21.392578+0000�[0m |�[34mopal_server.server                      �[0m|�[1m INFO  | listening on webhook topic: 'webhook'�[0m
24
�[32m2021-07-02T19:50:21.393104+0000�[0m |�[34mfastapi_websocket_pubsub.event_notifier �[0m|�[1m INFO  | New subscription {'id': '5392fa46240c446bb2d1e2732aca5639', 'subscriber_id': 'e0d9f893f4dd45398349af18ea7185df', 'topic': 'webhook', 'callback': functools.partial(<function trigger_repo_watcher_pull at 0x7fd265e34ee0>, <opal_server.policy.watcher.task.RepoWatcherTask object at 0x7fd265e3afa0>), 'notifier_id': None}�[0m
23
�[32m2021-07-02T19:50:21.393235+0000�[0m |�[34mopal_server.policy.watcher.task         �[0m|�[1m INFO  | Launching repo watcher�[0m
22
�[32m2021-07-02T19:50:21.393811+0000�[0m |�[34mopal_common.git.repo_cloner             �[0m|�[1m INFO  | Cloning repo from 'https://github.com/Hongbo-Miao/opal-example-policy-repo' to '/regoclone'�[0m
21
�[32m2021-07-02T19:50:21.432451+0000�[0m |�[34mopal_server.server                      �[0m|�[1m INFO  | triggered startup event�[0m
20
�[32m2021-07-02T19:50:21.432757+0000�[0m |�[34mopal_common.topics.publisher            �[0m|�[1m INFO  | started topic publisher�[0m
19
�[32m2021-07-02T19:50:21.581670+0000�[0m |�[34mopal_server.server                      �[0m|�[1m INFO  | triggered startup event�[0m
18
�[32m2021-07-02T19:50:21.583396+0000�[0m |�[34mopal_common.topics.publisher            �[0m|�[1m INFO  | started topic publisher�[0m
17
�[32m2021-07-02T19:50:22.005727+0000�[0m |�[34mopal_common.authentication.signer       �[0m|�[1m INFO  | OPAL was not provided with JWT encryption keys, cannot verify api requests!�[0m
16
�[32m2021-07-02T19:50:22.145396+0000�[0m |�[34mopal_server.server                      �[0m|�[1m INFO  | triggered startup event�[0m
15
�[32m2021-07-02T19:50:22.145709+0000�[0m |�[34mopal_common.topics.publisher            �[0m|�[1m INFO  | started topic publisher�[0m
14
�[32m2021-07-02T19:50:24.186247+0000�[0m |�[34mopal_common.git.repo_cloner             �[0m|�[1m INFO  | Clone succeeded�[0m
13
�[32m2021-07-02T19:50:24.195650+0000�[0m |�[34mopal_common.git.repo_watcher            �[0m|�[1m INFO  | Launching polling task, interval: 30 seconds�[0m
12
�[32m2021-07-02T19:50:30.095729+0000�[0m |�[34mfastapi_websocket_pubsub.event_broadc...�[0m|�[1m INFO  | Listening for incoming events from broadcast channel (first listener started)�[0m
11
�[32m2021-07-02T19:50:30.095913+0000�[0m |�[34mfastapi_websocket_pubsub.event_broadc...�[0m|�[1m INFO  | Spawning broadcast listen task�[0m
10
�[32m2021-07-02T19:50:30.123731+0000�[0m |�[34mfastapi_websocket_pubsub.event_broadc...�[0m|�[1m INFO  | Subscribing to ALL TOPICS, and sharing messages with broadcast channel�[0m
9
�[32m2021-07-02T19:50:30.124595+0000�[0m |�[34mfastapi_websocket_pubsub.event_notifier �[0m|�[1m INFO  | New subscription {'id': '8044b78c4ca94a6a8a449557407f2ab3', 'subscriber_id': '752149bcbcdd44caab5687f893a864f4', 'topic': '__EventNotifier_ALL_TOPICS__', 'callback': <bound method EventBroadcaster.__broadcast_notifications__ of <fastapi_websocket_pubsub.event_broadcaster.EventBroadcaster object at 0x7fd265dbd7f0>>, 'notifier_id': None}�[0m
8
�[32m2021-07-02T19:50:30.125039+0000�[0m |�[34mfastapi_websocket_rpc.websocket_rpc_e...�[0m|�[1m INFO  | Client connected�[0m
7
�[32m2021-07-02T19:50:30.125617+0000�[0m |�[34mfastapi_websocket_pubsub.event_broadc...�[0m|�[1m INFO  | Starting broadcaster listener�[0m
6
�[32m2021-07-02T19:50:30.143651+0000�[0m |�[34mfastapi_websocket_rpc.websocket_rpc_e...�[0m|�[1m INFO  | Client connected�[0m
5
�[32m2021-07-02T19:50:30.158254+0000�[0m |�[34mfastapi_websocket_pubsub.event_notifier �[0m|�[1m INFO  | New subscription {'id': '97e2f81dc8834242909779a5182eb3db', 'subscriber_id': 'f0098e871bf34d90b7ed4124e933780d', 'topic': 'policy_data', 'callback': <function RpcEventServerMethods.subscribe.<locals>.callback at 0x7fd265c1eca0>, 'notifier_id': None}�[0m
4
�[32m2021-07-02T19:50:30.163055+0000�[0m |�[34mfastapi_websocket_pubsub.event_notifier �[0m|�[1m INFO  | New subscription {'id': '87a0928aae7a4c5193c8f1e5b60ca837', 'subscriber_id': 'bc64f1a223cd45918d494f959681195c', 'topic': 'policy:.', 'callback': <function RpcEventServerMethods.subscribe.<locals>.callback at 0x7fd265c1edc0>, 'notifier_id': None}�[0m
3
�[32m2021-07-02T19:50:54.176647+0000�[0m |�[34mopal_common.git.repo_watcher            �[0m|�[1m INFO  | Pulling changes from remote: 'origin'�[0m
2
�[32m2021-07-02T19:50:56.022125+0000�[0m |�[34mopal_common.git.repo_watcher            �[0m|�[1m INFO  | No new commits: HEAD is at 'ec42a83a1e1a8fa7c795a8ecfd26a8dbcb63d20d'�[0m
1
�[32m2021-07-02T19:51:26.001562+0000�[0m |�[34mopal_common.git.repo_watcher            �[0m|�[1m INFO  | Pulling changes from remote: 'origin'�[0m

opal-client log

[2021-07-02 19:50:28 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2021-07-02 19:50:28 +0000] [1] [INFO] Listening at: http://0.0.0.0:7000 (1)
[2021-07-02 19:50:28 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-07-02 19:50:28 +0000] [9] [INFO] Booting worker with pid: 9
�[32m2021-07-02T19:50:28.977886+0000�[0m |�[34mopal_common.fetcher.fetcher_register    �[0m|�[1m INFO  | Fetcher Register loaded�[0m
�[32m2021-07-02T19:50:29.011354+0000�[0m |�[34mopal_client.opa.runner                  �[0m|�[1m INFO  | Launching opa runner�[0m
�[32m2021-07-02T19:50:29.013804+0000�[0m |�[34mopal_client.opa.runner                  �[0m|�[1m INFO  | Running OPA inline: opa run --server --addr=:8181 --authentication=off --authorization=off --log-level=info�[0m
�[32m2021-07-02T19:50:30.026374+0000�[0m |�[34mopal_client.opa.runner                  �[0m|�[1m INFO  | Running OPA initial start callbacks�[0m
�[32m2021-07-02T19:50:30.027003+0000�[0m |�[34mopal_client.policy.updater              �[0m|�[1m INFO  | Launching policy updater�[0m
�[32m2021-07-02T19:50:30.027207+0000�[0m |�[34mopal_client.data.updater                �[0m|�[1m INFO  | Launching data updater�[0m
�[32m2021-07-02T19:50:30.027391+0000�[0m |�[34mopal_client.policy.updater              �[0m|�[1m INFO  | Subscribing to topics: ['policy:.']�[0m
�[32m2021-07-02T19:50:30.027636+0000�[0m |�[34mopal_client.data.updater                �[0m|�[1m INFO  | Subscribing to topics: ['policy_data']�[0m
�[32m2021-07-02T19:50:30.028151+0000�[0m |�[34mfastapi_websocket_pubsub.pub_sub_client �[0m|�[1m INFO  | Trying to connect to Pub/Sub server - ws://opal-server.hm:7002/ws�[0m
�[32m2021-07-02T19:50:30.029166+0000�[0m |�[34mfastapi_websocket_rpc.websocket_rpc_c...�[0m|�[1m INFO  | Trying server - ws://opal-server.hm:7002/ws�[0m
�[32m2021-07-02T19:50:30.037420+0000�[0m |�[34mfastapi_websocket_pubsub.pub_sub_client �[0m|�[1m INFO  | Trying to connect to Pub/Sub server - ws://opal-server.hm:7002/ws�[0m
�[32m2021-07-02T19:50:30.038165+0000�[0m |�[34mfastapi_websocket_rpc.websocket_rpc_c...�[0m|�[1m INFO  | Trying server - ws://opal-server.hm:7002/ws�[0m
�[32m2021-07-02T19:50:30.164878+0000�[0m |�[34mopal_client.data.updater                �[0m|�[1m INFO  | Connected to server�[0m
�[32m2021-07-02T19:50:30.165119+0000�[0m |�[34mopal_client.data.updater                �[0m|�[1m INFO  | Performing data configuration, reason: Initial load�[0m
�[32m2021-07-02T19:50:30.165502+0000�[0m |�[34mopal_client.data.updater                �[0m|�[1m INFO  | Getting data-sources configuration from 'http://opal-server.hm:7002/data/config'�[0m
�[32m2021-07-02T19:50:30.169007+0000�[0m |�[34mopal_client.policy.updater              �[0m|�[1m INFO  | Connected to server�[0m
�[32m2021-07-02T19:50:30.169786+0000�[0m |�[34mopal_client.policy.updater              �[0m|�[1m INFO  | Refetching policy code (full bundle)�[0m
�[32m2021-07-02T19:50:30.192195+0000�[0m |�[34mopal_client.data.updater                �[0m|�[1m INFO  | Triggering data update with id: 554b8db986824c4097353b50571751cf�[0m
�[32m2021-07-02T19:50:30.192606+0000�[0m |�[34mopal_client.data.updater                �[0m|�[1m INFO  | Fetching policy data�[0m
�[32m2021-07-02T19:50:30.193302+0000�[0m |�[34mopal_client.data.fetcher                �[0m|�[1m INFO  | Fetching data from url: https://e5d4bcbf32aa.ngrok.io/data�[0m
�[32m2021-07-02T19:50:30.230687+0000�[0m |�[34mfastapi_websocket_pubsub.pub_sub_client �[0m|�[1m INFO  | Connected to PubSub server ws://opal-server.hm:7002/ws�[0m
�[32m2021-07-02T19:50:30.232767+0000�[0m |�[34mopal_client.policy.updater              �[0m|�[1m INFO  | got policy bundle, commit hash: 'ec42a83a1e1a8fa7c795a8ecfd26a8dbcb63d20d'�[0m
�[32m2021-07-02T19:50:30.257152+0000�[0m |�[34mfastapi_websocket_pubsub.pub_sub_client �[0m|�[1m INFO  | Connected to PubSub server ws://opal-server.hm:7002/ws�[0m
�[32m2021-07-02T19:50:31.800866+0000�[0m |�[34mopal_client.data.updater                �[0m|�[1m INFO  | Saving fetched data to policy-store: source url='https://e5d4bcbf32aa.ngrok.io/data', destination path='/'�[0m

[hongbo-miao]

Followed up offline, after all steps above, I need another step which is notifying opal-client through opal-server (listen at http://localhost:7002 in this case) by this POST call

curl --location --request POST 'http://localhost:7002/data/config' \
--header 'Content-Type: application/json' \
--data-raw '{
  "entries": [
    {
      "url": "https://e5d4bcbf32aa.ngrok.io/data",
      "topics": [
        "policy_data"
      ]
    }
  ],
  "reason": "update data"
}
'

The doc for this POST call is at https://opal.authorizon.com/redoc#operation/get_data_sources_config_data_config_get

Then opal-client will fetch again based on the URL (https://e5d4bcbf32aa.ngrok.io/data in this case) it just got from opal-server.

Thanks for the help!

[asafc]

Hey @hongbo-miao, you asked a lot of important questions - i'll try to be detailed so that others reading this may also benefit.

OPAL clients have two distinct types of data sources:

Data Source Type	Format of data	Function	Source of data
Policies (and static data)	Policy bundle - data format is very similar to OPA native bundles.	Each bundle contains the policies (rego) and static data that is usually included as part of the policy. By "static", we mean the data rarely changes, same as declaring the data inside the rego file.	Git repository (we are thinking about expanding this to more sources in the near future).
Dynamic Data	DataSourceEntry - think about it as a "directive" how to fetch the data and where to put it in OPA document tree - i will give an example below.	Realtime updates about data that changes in the rate of the application (i.e: as a result of user action), for example - we invited a user to a document in google drive. If we implement this internally as the user now having a Viewer role on the document, we can send a data update containing the new role, with instructions to put it inside the roles document in OPA cache.	Can be anything - a database, a 3rd party API, your APIs, etc - extensible by fetch providers. We support http apis out of the box.

Q: What are OPAL_DATA_CONFIG_SOURCES?

• They are dynamic data sources, so they are directives how to fetch the data and where to put it in OPA.
• OPAL_DATA_CONFIG_SOURCES are statically configured because they contain the list of "complete picture" data sources - which are fetched after OPAL client loads, and after every disconnect between OPAL client and OPAL server. Think of it as directives how to get the up-to-date "clean slate" data.

Q: for policy bundles - how do we know to bundle .rego and data.json files?

As @orweis mentioned - this is configured in code (by OPAL_OPA_FILE_EXTENSIONS).
As to why - because this is the format of native OPA bundles.

Q: how to get data from external services that are different than policy in git repo?

With dynamic data sources.

When using dynamic data sources, you should do two things:

1. Declare OPAL_DATA_CONFIG_SOURCES with where to fetch the complete data.

   For example let's say the app allows you to defines new roles on the fly. So the list of roles is dynamic and changing as a result of user actions. Your OPAL_DATA_CONFIG_SOURCES should be something like this (return all roles and override the entire roles document):

{"url":"https://api.yourservice.com/v1/roles","topics":["policy_data"], "dst_path": "roles"}

2. Realtime updates to the data can be passed with the updates api (and are also formatted as DataSourceEntries). Note that for updates, you don't need to pass an entry for the entire data - only for the data that changed.

   For example, if we added a new editor role (let's say the id for this role is editor, you can pass the following update directive:

{"url":"https://api.yourservice.com/v1/roles/editor","topics":["policy_data"], "dst_path": "roles/editor"}

The OPA cache will now have a new document: /roles/editor which can be accessed with the OPA data API at /v1/data/roles/editor.

Q: Can i replace a data.json static data config with a dynamic data source?

Yes and no. In essence you are right - there is no difference between data loaded statically via the bundle and data loaded dynamically via data source entries - they are both being written using the OPA Data API.

However, Your rego code still needs to be valid. I'll explain:

When OPA runs a query, your policy (rego code) is evaluated against the base data documents in cache (i.e: /v1/data). But if a document does not exist, OPA might evaluate a rule to undefined, and when that happens - the undefined value may cascade and make the entire query evaluate to undefined.

Your rego must evaluate correctly before (and after) the DataSourceEntries are fetched. This should be easy - simply make sure your code handles the case where the path to the "dynamic" documents points to a non-existing doc (or undefined key).

Q: Why use data source entries / directives and not simply pass the data inside the update message?

1. Central configuration management: It is easier to manage the configuration of authorization data sources centrally.
2. Client can be stateless: only state the client needs is declarative - what are the topics for policy and dynamic data i need to subscribe to. Or in other words, what is the relevant state this client needs to keep in sync in OPA's cache.
3. Performance/reliability: websockets should be used with caution - they are great for lightweight message passing - but for big data documents it is more reliable to use HTTP/S. So we only pass the update metadata over websockets.
4. Compliance: the server should only act as coordinator, but the client (which sits close to your services) should be exposed to the actual data - a lot of the time it keeps sensitive data in your local network and is better for compliance, e.g: GDPR, etc.