Demo/docs/snippets of external_source_url in OPAL_DATA_CONFIG_SOURCES

[hongbo-miao]

It would be great to have some demo/docs/snippets of external_source_url in OPAL_DATA_CONFIG_SOURCES so that a pool of opal-servers can serve different opal-clients.

For example, what would be expected to return from the config server based on OPAL_DATA_CONFIG_SOURCES={"external_source_url":"https://myBaseConfigServer.com"}?

Thanks! 😃

[asafc]

Hi @hongbo-miao, let's break down how an external data sources setup looks like.

General Architecture

When do you need to configure an external data source?

If each of your OPAL clients needs a slightly different configuration, you cannot use a static config var like OPAL_DATA_CONFIG_SOURCES to store your configuration. In such case, you want OPAL server to return a different DataSourceEntry configuration based on the identity of the OPAL client asking for the config.

Can different OPAL client have different "identities"?

Yes!! And they should!!

Each of your opal clients should use a unique JWT token issued to it by the OPAL server.
You can learn how to generate your OPAL client token here.

When issued via the API, each OPAL client token (JWT) supports custom claims (the claims attribute).

How to configure an external data source?

OPAL server supports redirecting to an external API server that can serve different DataSourceEntries based on the JWT token identifying the OPAL client. This external API server should have access to OPAL's public key so that it can validate OPAL JWTs correctly and make sure the OPAL identity passed in a Bearer token is valid.

This is how a typical configuration of OPAL_DATA_CONFIG_SOURCES looks like: (a breakdown of this config can be found here)

{
    "config": {
        "entries": [
            {
                "url": "https://api.authorizon.com/v1/policy-config",
                "topics": [
                    "policy_data"
                ],
                "config": {
                    "headers": {
                        "Authorization": "Bearer FAKE-SECRET"
                    }
                }
            }
        ]
    }
}

A OPAL_DATA_CONFIG_SOURCES configuration that redirects to an external data source looks slightly different:

{
    "external_source_url": "https://your-api.com/path/to/api/endpoint"
}

As you see, the config is now much simpler. OPAL server will simply redirect the query.

How the redirect actually works?

Upon being asked for data source entries, OPAL server will simply redirect to the external_source_url, but will also concat a url query param containing the OPAL CLIENT JWT.

Request by OPAL client:

GET https://opal-server.com/data/config

Response by OPAL server:

HTTP 307 Temporary Redirect
https://your-api.com/path/to/api/endpoint?token=<OPAL_CLIENT_JWT>

This is why you should never use an external_source_url that is not HTTPS.
OPAL relies on the TLS/SSL encryption so it can pass the JWT as a query param, knowing it will be encrypted.

Actions that should be taken by the external API

The external API should do the following:

• Expose an http GET endpoint with same path as in OPAL config: i.e: /path/to/api/endpoint
• When hit, this endpoint should extract the OPAL client JWT from the token query param.
• The endpoint should validate that the token is a valid OPAL JWT (by using the OPAL public key to verify) and use the custom claims in the JWT to determine what data sources to return.
• The endpoint should return a JSON, containing the unique value (to that OPAL client) of OPAL_DATA_CONFIG_SOURCES (i.e: the actual Data Source Entries).

Real code samples

You can use the following code samples in your external API server that serves data sources.
These are taken from our fastapi python service.

Step 0: when issuing OPAL client token use custom claims

These custom claims will identify one OPAL client from another.

For example, this is how we assign OPAL client tokens to our customers' OPAL clients:

from opal_common.schemas.security import AccessTokenRequest, PeerType
from acalla.config import OPAL_SERVER_URL, OPAL_MASTER_TOKEN

...

async with aiohttp.ClientSession(headers={"Authorization": f"bearer {OPAL_MASTER_TOKEN}", **self._headers}) as session:
    token_params = AccessTokenRequest(
         type=PeerType.client,
         ttl=timedelta(days=CLIENT_TOKEN_TTL_IN_DAYS),
         claims={'authorizon_client_id': pdp.client_id},
    ).json()
    async with session.post(f"{OPAL_SERVER_URL}/token", data=token_params) as response:
         data: dict = await response.json()
         token = data.get("token", None)

This function hits opal server's /token endpoint (see: API reference), generates a client token (peer type == client) and adds a custom claim that identifies the authorizon_client_id - which is unique for each of our SaaS clients.

Since OPAL JWTs are cryptographically signed - this cannot be forged.

Step 1: in your external api server, be able to parse JWTs from http query param

First, we have a custom class to verify OPAL JWT from HTTP query param.
Notice we use the opal-common pip package as a library in our python server.
You can of course write this code in any language you want - according to API spec we detailed above.

from typing import Optional

from fastapi import APIRouter, Depends, status, HTTPException, Query
from opal_common.authentication.types import JWTClaims
from opal_common.authentication.deps import _JWTAuthenticator, verify_logged_in
from opal_common.authentication.verifier import Unauthorized

class RedirectJWTAuthenticator(_JWTAuthenticator):
    """
    OPAL JWT authentication via HTTP query params.
    throws 401 if a valid jwt is not provided via the `token` query param.

    (same as JWTAuthenticator, but tries to get the JWT from a query param).
    """
    def __call__(self, token: Optional[str] = Query(None)) -> JWTClaims:
        """
        called when using the verifier as a dependency with Depends()
        """
        if token is None:
            raise Unauthorized(description="Access token was not provided!")
        return verify_logged_in(self.verifier, token)

Step 2: in your external api server, expose the endpoint to serve dynamic data sources

You dynamic data sources endpoints should read your custom claim and understand how to fetch the correct config based on these claims.

from typing import Optional
from fastapi import APIRouter, Depends, status, HTTPException

from opal_common.authentication.types import JWTClaims
from opal_common.schemas.data import DataSourceConfig, DataSourceEntry
from opal_common.fetcher.providers.http_fetch_provider import HttpFetcherConfig
from opal_common.authentication.verifier import JWTVerifier, Unauthorized
from opal_common.config import opal_common_config
...

def init_dynamic_data_sources_router():
    """
    inits router for opal data sources.

    this router will output different data sources depending on the jwt claims
    in the provided bearer token (claim will include pdp client id).
    """
    router = APIRouter()

    verifier = JWTVerifier(
        public_key=opal_common_config.AUTH_PUBLIC_KEY,
        algorithm=opal_common_config.AUTH_JWT_ALGORITHM,
        audience=opal_common_config.AUTH_JWT_AUDIENCE,
        issuer=opal_common_config.AUTH_JWT_ISSUER,
    )
    # notice - we defined RedirectJWTAuthenticator in our previous code sample (step 1)
    authenticator = RedirectJWTAuthenticator(verifier)

    async def extract_pdp_from_jwt_claims_or_throw(db: Session, claims: JWTClaims) -> PDP:
        claim_client_id = claims.get('authorizon_client_id', None)
        if claim_client_id is None:
            raise Unauthorized(description="provided JWT does not have an authorizon_client_id claim!")

        return await PDP.from_id_or_throw_401(db, claim_client_id)
    
    @router.get('/data/config', response_model=DataSourceConfig)
    async def get_opal_data_sources(db: Session = Depends(get_db), claims: JWTClaims = Depends(authenticator)):
        # at this point the jwt is valid, but we still have to extract a valid project from its claims
        pdp: PDP = await extract_pdp_from_jwt_claims_or_throw(db, claims)
        topics = [f"policy_data/{pdp.client_id}"]
        return DataSourceConfig(
            entries=[
                DataSourceEntry(
                    url=f"{BACKEND_PUBLIC_URL}/v1/policy/config",
                    topics=topics,
                    config=HttpFetcherConfig(
                        headers={'Authorization': f"Bearer {pdp.client_secret}"}
                    )
                )
            ],
        )

    return router

notice how our endpoint simply returns opal_common.schemas.data.DataSourceConfig, specific to each client. In our case all data is currently piped from our API - but each client has a unique client secret to identify it in our systems.

[hongbo-miao]

Hi @asafc

Usually we verify a JWT token in server by using a secret key.

I have these now

• OPAL_AUTH_MASTER_TOKEN
• OPAL_AUTH_PUBLIC_KEY (id_rsa.pub)
• OPAL_AUTH_PRIVATE_KEY (id_rsa)
• JWT token for opal-client generated by opal-server

In own config server, for

The endpoint should validate that the token is a valid OPAL JWT (by using the OPAL public key to verify) and use the custom claims in the JWT to determine what data sources to return.

What does OPAL public key here refer to? Thanks!

[asafc]

the public key is whatever you pass to OPAL_AUTH_PUBLIC_KEY, probably what's in id_rsa.pub.

[hongbo-miao]

Oh yeah, that is it!

[asafc]

FYI: also available under docs.