Skip to content

Document Authorization

Jump to bottom
Piergiorgio Lucidi edited this page Nov 3, 2026 · 1 revision

Document Authorization & OpenFGA (Relationship-Based Access Control)

OIS Compliance OpenFGA Zero-Trust Java

Module Reference: oc-core (DocumentAccessGate, AuthorizationProvider, EffectiveAcl), oc-authorization-openfga, oc-mcp-server

📌 1. Executive Summary

Every document OpenCrawling retrieves for an AI agent, Large Language Model (LLM), or API client through the Model Context Protocol (MCP) tools is authoritatively evaluated against that document's access permissions on the server, before any text, metadata, or chunk enters the model's context window.

OpenCrawling supports pluggable document authorization:

  1. Metadata Stored ACLs (metadata mode, default): Evaluates the Access Control Lists stored with each chunk in the vector store using EffectiveAcl and McpAclEvaluator.
  2. Relationship-Based Access Control (ReBAC) with OpenFGA: Decouples permission evaluation into a high-performance fine-grained authorization engine, enabling dynamic group membership changes, transitive role hierarchies, and immediate permission revocation without re-indexing chunks.
  3. Safe Migration Modes (shadow and intersection): Enables operators to validate OpenFGA against stored ACL decisions with zero downtime and zero false-positive access grants before cutting over.
flowchart TD
    subgraph Client["AI Agent / LLM Client Boundary"]
        Caller["Caller Identity<br/>(userPrincipal + userRoles)"]
    end

    subgraph Store["Search & Retrieval Engine Layer"]
        Backends["Vector / Search Backends (9 Engines)<br/>pgvector / Vespa / Qdrant / Milvus / OpenSearch / Solr / Luxir / OpenArch"]
        Candidates["Ranked Top-K Candidates<br/>(Candidate Window: limit × overfetch-factor)"]
    end

    subgraph GateLayer["Unified Enforcement Layer (Fail-Closed)"]
        Gate["DocumentAccessGate<br/>(Single access gate for all stores)"]
        SPI["AuthorizationProvider SPI<br/>(Batch evaluation per page)"]
    end

    subgraph Providers["Authorization Providers"]
        Stored["StoredAclAuthorizationProvider<br/>(EffectiveAcl / McpAclEvaluator)"]
        FGA["OpenFgaAuthorizationProvider<br/>(OpenFGA BatchCheck API v1.22.0)"]
    end

    Caller --> Backends
    Backends --> Candidates
    Candidates --> Gate
    Gate --> SPI
    SPI -.->|metadata mode| Stored
    SPI -.->|shadow mode| Stored & FGA
    SPI -.->|intersection mode| Stored & FGA
    SPI -.->|openfga mode| FGA
    Gate -->|"Only Authorized Documents (Fail-Closed)"| Caller
Loading

Key Architectural Principles:

  • One Access Gate for Every Store: All 9 MCP search backends (PostgreSQL/pgvector, Qdrant, Milvus, OpenSearch 2.x, OpenSearch 3.x, Apache Solr, Vespa, Luxir, and Hyland OpenArch) hand their search hits directly to DocumentAccessGate. No store backend makes its own access decision.
  • Batch Evaluation: The gate queries the configured AuthorizationProvider for the whole candidate page in a single batch call (canRead(Caller, List<ResourceRef>)), minimizing network round-trips.
  • Fail-Closed Semantics: A document with missing or unreadable permission data is immediately denied. If the authorization provider encounters an error (engine outage, network timeout, malformed payload), the entire request fails closed. OpenCrawling never falls back to weaker security rules.

🏛️ 2. Permission Normalization (EffectiveAcl)

Connectors harvest document security from heterogeneous repositories as a structured SecurityConfig (a collection of PermissionRule entries defining identity, identityType, and access), optionally accompanied by legacy comma-separated acl strings.

The unified normalizer EffectiveAcl translates these rules into consistent fields indexed across every search engine (security_allowed_read, security_denied_read, security_inheritance) and mirrored into OpenFGA:

Source Permission Rule Normalized Effect
access = "read", "write", or "admin" Grants read access to the specified identity.
access = "deny" Explicitly denies read access. An explicit deny strictly supersedes any allow grant.
identityType = "public" (e.g. CMIS anonymous, GROUP_EVERYONE) Indexed as public: readable by all callers.
Structured rules present Structured rules authoritatively decide; legacy flat acl strings are ignored.
Default public placeholder rule + legacy acl naming identities The legacy acl identities take precedence: an unauthenticated connector cannot accidentally make restricted documents public.
No structured rules, only legacy acl Tokens in acl grant read access ("read" itself is filtered as a reserved keyword).
Neither structured rules nor legacy acl Nobody may read the document (fail-closed default).

Identity Matching Rules:

  • Case-insensitive comparison across all identities, user principals, and roles.
  • Leading user: or group: prefixes are stripped symmetrically during comparison (e.g., a grant to group:finance matches caller role finance or group:finance).
  • Identities containing commas (such as LDAP Distinguished Names: CN=Alice,OU=Finance,DC=corp) are parsed and matched as intact whole strings without improper delimiter splitting.

Note

Today the caller's principal and roles are asserted by the MCP client in tool arguments (userPrincipal, userRoles). Token-based verification (OAuth2/OIDC RBAC) is governed by issue #187; the Caller abstraction already incorporates an authenticated flag ready for token propagation.

🎛️ 3. Operational Authorization Modes

Configured via opencrawling.security.authorization.mode (OPENCRAWLING_SECURITY_AUTHORIZATION_MODE):

Mode Authoritative Decision Maker Recommended Phase Behavior & Rollout Context
metadata (default) Permissions stored with each chunk in the vector store Production (standalone) Default mode. OpenFGA services are not started. Uses McpAclEvaluator and stored vector payloads.
shadow Stored vector permissions decide; OpenFGA evaluated asynchronously Pre-migration validation Stored ACLs govern the response returned to the caller. OpenFGA is queried concurrently; disagreements increment oc.authz.shadow.mismatch and log warnings.
intersection Both stored ACLs and OpenFGA must allow Cautious cut-over Dual-gate security: OpenFGA can only restrict access further, never inadvertently expand it.
openfga OpenFGA alone decides Target ReBAC state True relationship-based access control. User group memberships and permission updates apply immediately without re-indexing vector stores.

Important

Selecting shadow, intersection, or openfga requires the oc-authorization-openfga module. If required configuration properties are missing, the runtime fails startup immediately with an actionable error rather than silently falling back to metadata mode.

🧠 4. How OpenFGA Decides (ReBAC Architecture)

The OpenFGA Model (model.fga)

OpenCrawling bundles a fine-grained authorization schema (oc-authorization-openfga/src/main/resources/openfga/model.fga):

model
  schema 1.1

type user

type group
  relations
    define member: [user, user:*, group#member]

type document
  relations
    define grantee: [user, user:*, group#member]
    define blocked: [user, user:*, group#member]
    define can_read: grantee but not blocked

Identifier & Tuple Encoding:

  1. Document Identifiers: Encoded as document:<sha256-hex(document_id)>. This produces fixed-length 64-character tokens safe against arbitrary characters, URIs, and path separators.
  2. Identity Tuples:
    • Each permission grant creates a grantee relation tuple on the document.
    • Each explicit deny rule creates a blocked relation tuple on the document.
    • Public access maps to user:* as a grantee.
    • Because stored-ACL matching supports both literal and prefix-stripped tokens, each identity generates two group representations: r_<raw_identity> and s_<stripped_identity>. Identifiers are lowercased and percent-encoded.
  3. Contextual Evaluation:
    • At query time, the caller's asserted userPrincipal and userRoles are supplied as contextual tuples to OpenFGA's BatchCheck API.
    • Group memberships stored directly in OpenFGA (e.g., group:s_finance#member@user:p_dave) apply transitively, enabling enterprise LDAP/IdP synchronization.

🔄 5. Decoupled Permission Synchronization

The AuthorizationSyncConsumer (oc-authorization-openfga) runs in the decoupled stack listening to the primary documents Kafka topic (opencrawling-documents) under consumer group opencrawling-authz-sync.

sequenceDiagram
    autonumber
    participant K as Kafka (opencrawling-documents)
    participant C as AuthorizationSyncConsumer
    participant F as OpenFGA Engine
    participant DLT as Kafka (opencrawling-documents.authz-dlt)

    K->>C: Document Ingestion / Update / Delete Message
    C->>F: Read current tuples for document:<hash>
    C->>C: Compute diff (additions & deletions)
    alt Diff > 0
        C->>F: Apply tuple diff (add blocked -> add grantee -> del grantee -> del blocked)
    end
    alt OpenFGA Error / Timeout
        C->>C: Retry with Exponential Backoff (6 attempts, 0.5s -> 10s)
        opt Retries Exhausted
            C->>DLT: Publish message to dead-letter topic
        end
    end
Loading

Sync Characteristics:

  • Diff Calculation & Ordering: Syncs compute minimal tuple diffs. When changes exceed 100 tuples, mutations are strictly ordered: add blocked → add grantee → remove grantee → remove blocked. This ensures a document is never temporarily more permissive during write windows.
  • Tombstones & Deletions: OIS DELETE messages delete all relationship tuples for the target document, ensuring purged files immediately vanish from search results even if downstream store purges lag.
  • Retry with Exponential Backoff: Transient errors retry up to 6 times (backing off from 500ms to 10s).
  • Dead-Letter Topic (opencrawling-documents.authz-dlt): Exhausted failures are isolated to the DLT without blocking the partition.

Warning

Do not replay the dead-letter topic directly. Replaying older failure messages verbatim could overwrite newer permission updates. To recover, resolve the OpenFGA outage and reset the consumer group offset to the original failed record offset (preserved in record header kafka_dlt-original-offset) or trigger a targeted re-crawl.

🔍 6. Result Completeness & Overfetching

When operating in intersection or openfga modes, OpenFGA may deny documents that the vector store ranked highly. To prevent returned result pages from truncating or starving, search backends dynamically widen their candidate retrieval window:

$$\text{Candidate Count} = \min(\text{requested_limit} \times \text{overfetch-factor}, ; \text{candidate-budget})$$

  • overfetch-factor: Default 10 (e.g. requesting 5 results fetches 50 candidate hits).
  • candidate-budget: Default 500 maximum candidates per query.
  • Pre-Filter Bypass: In intersection and openfga modes, backends disable engine-level ACL pre-filters (e.g. PostgreSQL security_allowed_read SQL clauses) so that candidates can be evaluated by OpenFGA.

⚙️ 7. Configuration Reference

Property Environment Variable Default Description
opencrawling.security.authorization.mode OPENCRAWLING_SECURITY_AUTHORIZATION_MODE metadata Mode: metadata, shadow, intersection, or openfga.
opencrawling.security.authorization.overfetch-factor OPENCRAWLING_SECURITY_AUTHORIZATION_OVERFETCH_FACTOR 10 Candidate multiplication factor for ReBAC queries.
opencrawling.security.authorization.candidate-budget OPENCRAWLING_SECURITY_AUTHORIZATION_CANDIDATE_BUDGET 500 Hard cap on candidate chunks passed to the access gate.
opencrawling.security.authorization.openfga.api-url OPENCRAWLING_SECURITY_AUTHORIZATION_OPENFGA_API_URL http://localhost:8080 OpenFGA HTTP endpoint.
opencrawling.security.authorization.openfga.api-token OPENCRAWLING_SECURITY_AUTHORIZATION_OPENFGA_API_TOKEN (none) Pre-shared key or Bearer token for OpenFGA authentication.
opencrawling.security.authorization.openfga.store-id OPENCRAWLING_SECURITY_AUTHORIZATION_OPENFGA_STORE_ID (none) Pinned OpenFGA Store ID. Pin in production.
opencrawling.security.authorization.openfga.model-id OPENCRAWLING_SECURITY_AUTHORIZATION_OPENFGA_MODEL_ID (none) Pinned OpenFGA Authorization Model ID. Pin in production.
opencrawling.security.authorization.openfga.auto-provision OPENCRAWLING_SECURITY_AUTHORIZATION_OPENFGA_AUTO_PROVISION false Automatically creates store and model if unpinned (Dev/Testing).
opencrawling.security.authorization.openfga.store-name OPENCRAWLING_SECURITY_AUTHORIZATION_OPENFGA_STORE_NAME opencrawling Store name used during auto-provisioning.
opencrawling.security.authorization.openfga.timeout OPENCRAWLING_SECURITY_AUTHORIZATION_OPENFGA_TIMEOUT 5s Timeout duration for OpenFGA API calls.
opencrawling.security.authorization.openfga.batch-size OPENCRAWLING_SECURITY_AUTHORIZATION_OPENFGA_BATCH_SIZE 50 Maximum check operations bundled per BatchCheck request.
opencrawling.security.authorization.openfga.max-parallel-requests OPENCRAWLING_SECURITY_AUTHORIZATION_OPENFGA_MAX_PARALLEL_REQUESTS 10 Concurrent BatchCheck HTTP calls allowed.
opencrawling.security.authorization.openfga.sync.enabled OPENCRAWLING_SECURITY_AUTHORIZATION_OPENFGA_SYNC_ENABLED true Enables AuthorizationSyncConsumer in the runtime container.
opencrawling.security.authorization.openfga.sync-group-id OPENCRAWLING_SECURITY_AUTHORIZATION_OPENFGA_SYNC_GROUP_ID opencrawling-authz-sync Consumer group ID for the authorization sync worker.

🚀 8. Safe Production Rollout Guide

Follow this phased zero-downtime migration path:

[Phase 1: Deploy OpenFGA] ──► [Phase 2: Shadow Mode] ──► [Phase 3: Backfill] ──► [Phase 4: Cutover]
  docker-compose-openfga         mode=shadow               Kafka replay + recrawl    mode=intersection / openfga

Phase 1: Deploy OpenFGA

Deploy OpenFGA v1.22.0 with PostgreSQL storage overlay using docker-compose-openfga.yml:

export OPENFGA_PRESHARED_KEY="$(openssl rand -hex 32)"
docker compose -f docker-compose.yml -f docker-compose-openfga.yml up -d openfga postgres-openfga

Phase 2: Run in Shadow Mode

Set OPENCRAWLING_SECURITY_AUTHORIZATION_MODE=shadow on oc-mcp-server. The sync consumer begins populating OpenFGA in real-time from active crawler jobs while stored ACLs remain authoritative.

Phase 3: Backfill Existing Documents

  1. Replay Kafka Topic: Reset the opencrawling-authz-sync group to the beginning of the retained opencrawling-documents topic: 
    kafka-consumer-groups.sh --bootstrap-server kafka:9092 --group opencrawling-authz-sync \
  --topic opencrawling-documents --reset-offsets --to-earliest --execute
  2. Re-crawl Historical Repositories: Trigger crawls for documents older than the Kafka retention window.

Phase 4: Validate & Cutover

Monitor Micrometer metric oc.authz.shadow.mismatch. Once mismatches stay at zero across normal production traffic, switch to intersection (conservative verification) or openfga (full ReBAC agility).

📊 9. Operational Telemetry & Metrics

OpenCrawling exposes dedicated Micrometer authorization metrics under /actuator/metrics, tagged with provider:

Metric Name Type Description
oc.authz.check Timer Latency distribution of candidate page authorization decisions.
oc.authz.candidates Counter Total candidate chunks evaluated by DocumentAccessGate.
oc.authz.denied Counter Total candidate chunks rejected due to insufficient permissions.
oc.authz.errors Counter Requests rejected due to provider timeouts or infrastructure failures.
oc.authz.shadow.mismatch Counter Decisions where OpenFGA disagreed with stored vector ACLs.
oc.authz.shadow.errors Counter OpenFGA errors during shadow evaluation (does not impact caller).

🧪 10. Verification & Test Suites

  • Unit Tests:
    • EffectiveAcl: Validates 20,000+ permutations ensuring stores holding full security objects and stores holding flattened fields make identical decisions.
    • OpenFgaIdentifiersTest: Verifies deterministic hashing, percent-encoding, and prefix stripping.
    • TuplePlanTest: Validates fail-closed tuple diff ordering.
  • Integration Tests (-DskipITs=false):
    • OpenFgaAuthorizationIT: Testcontainers test exercising sync, live check, deletes, and fail-closed behavior during outages.
    • OpenFgaDifferentialIT: Validates 37,500 random document and caller combinations comparing stored ACL decisions against real OpenFGA server evaluations.
  • End-to-End Decoupled Script: 
    ./scripts/test-openfga-decoupled.sh
    Executes an 8-stage automated decoupled pipeline verification testing OIS document ingestion, per-caller search isolation, dynamic OpenFGA group membership additions/removals without re-indexing, tombstone deletions, fail-closed behavior under service shutdown, and metric emissions.

🗺️ Documentation

🧪 Testing

🤝 Community

Clone this wiki locally