Skip to content

Confluence Repository Connector

Piergiorgio Lucidi edited this page Nov 21, 2026 · 1 revision

Atlassian Confluence Repository Connector

Confluence Java Spring Boot OIS Compliance Claim Check

Module: oc-confluence-repository-connector
Connector Class: org.opencrawling.confluence.ConfluenceRepositoryConnector
Client Class: org.opencrawling.confluence.ConfluenceClient
Storage Converter: org.opencrawling.confluence.ConfluenceStorageConverter
Document Builder: org.opencrawling.confluence.ConfluenceDocumentBuilder
Security Mapper: org.opencrawling.confluence.ConfluenceSecurityMapper
Properties Class: org.opencrawling.confluence.ConfluenceProperties
Crawl State: org.opencrawling.confluence.ConfluenceCrawlState & org.opencrawling.runtime.connector.ControlPlaneConfluenceStateStore
Discovery & Single-Doc SPI: Implements DocumentFetchingRepositoryConnector (DiscoverableRepositoryConnector: explore, dryRun, crawlStart, fetchDocument)
Spring Bean: @Component (activated when spring.opencrawling.connector.type=confluence or SPRING_OPENCRAWLING_REPOSITORY_CONNECTOR_TYPE=confluence)
Introduced in: Issue #341, PR #440


📌 Overview

The Atlassian Confluence Repository Connector (oc-confluence-repository-connector) crawls Confluence Cloud (REST API v2 + v1 restrictions/CQL) and Confluence Data Center / Server (REST API v1) using native Java 25 java.net.http.HttpClient, Jackson 2, and jsoup—with zero external Atlassian SDK dependencies.

It transforms Confluence Storage Format XHTML (body.storage) into clean, semantic HTML5 preserving headings (h1–h6), lists, tables, blockquotes, code blocks, and content-bearing macros (expand, panel, info, note, warning, tip, status, excerpt, code, noformat) while dropping boilerplate navigation macros (toc, children, pagetree, jira, recently-updated, contributors, contentbylabel, livesearch). Pages, blog posts, and coupled binary attachments are emitted as Open Ingestion Standard (OIS) documents with clickable url citations, ancestor breadcrumbs, filterable tags (Confluence labels), multi-replica ControlPlaneStore CQL watermarks, periodic full reconciliation with an empty-listing guard, explicit TOO_LARGE ledger reporting, pre-crawl discovery, and fail-closed Zero-Trust Space + inherited page read restriction ACLs.

Important

Cross-Origin Redirect Security & Fail-Closed ACLs: When downloading binary attachments, ConfluenceClient follows redirects manually and automatically strips the Authorization header whenever a redirect targets a different origin (such as Confluence Cloud's signed CDN download URLs). Anonymous Space permissions are ignored by default (allow-anonymous-public=false), and unreadable Space permissions or page restrictions always fail closed (SecurityConfig.none()).


🏗️ Architecture & Flow

graph TD
    subgraph Atlassian["Atlassian Confluence (Cloud & Data Center)"]
        Auth["Auth (Cloud API Token / OAuth 2.0 3LO / DC Bearer PAT / Basic)"]
        CloudApi["Confluence Cloud REST API v2 + v1 (/api/v2/spaces, /pages, /labels, /attachments + /rest/api/content/search & restrictions)"]
        DcApi["Confluence Data Center REST API v1 (/rest/api/space, /content, /restriction, /content/search)"]
        CDN["Signed CDN Attachment Download (Auth Stripped Cross-Origin)"]

        Auth --> CloudApi
        Auth --> DcApi
        CloudApi -. 302 Redirect .-> CDN
    end

    subgraph OpenCrawling["OpenCrawling Ingestion Runtime (Java 25)"]
        Client["ConfluenceClient (Java 25 HttpClient, Retry-After, Group UUID Cache, Server Date)"]
        Connector["ConfluenceRepositoryConnector (CQL Delta, Tree, Discovery SPI, Cascade Tombstones)"]
        StateStore["ControlPlaneConfluenceStateStore (Shared ControlPlaneStore / oc_setting)"]
        VThreads["Virtual Threads (Bounded Parallelism)"]
        Converter["ConfluenceStorageConverter (jsoup XML -> Semantic HTML5 + Macro Policy)"]
        SecMap["ConfluenceSecurityMapper (Space View ∩ Inherited Read Restrictions, Fail-Closed)"]
        DocBld["ConfluenceDocumentBuilder (Pages, Blog Posts, Coupled Attachments, TOO_LARGE)"]
        TempStream["TempFileInputStream (Constant-Heap Spool & Auto-Delete)"]
        Orchestrator["JobOrchestrator (TOO_LARGE -> SKIPPED Ledger + Trace WARN)"]
        ClaimCheck["Claim-Check Store (Local / Apache Ozone / S3)"]
        Tika["Apache Tika 4 Text Extraction"]

        CloudApi <-->|REST JSON| Client
        DcApi <-->|REST JSON| Client
        CDN -->|Binary Stream| Client
        Client --> Connector
        Connector <--> StateStore
        Connector --> VThreads
        VThreads --> Converter
        VThreads --> SecMap
        VThreads --> TempStream
        Converter --> DocBld
        SecMap --> DocBld
        DocBld --> Orchestrator
        TempStream --> ClaimCheck
        ClaimCheck --> Tika

        OIS["OIS Document (UPSERT / Cascading DELETE Tombstone)"]
        Orchestrator --> OIS
        Tika --> OIS

        OIS --> Kafka["Kafka Pipeline (opencrawling-ingestion -> chunks -> embedded)"]
        Kafka --> Sinks[("Vector & Lakehouse Stores (PGVector, Vespa, Qdrant, Milvus, Solr, OpenSearch, Elasticsearch, Iceberg, ...)")]
    end

    subgraph AI["AI Applications & Zero-Trust MCP"]
        Sinks --> SecureMcp["OpenCrawling Secure MCP Server (McpAclEvaluator & DiscoveryMcpTools)"]
        SecureMcp -->|Zero-Trust Space & Page Read Restriction Enforcement| Agent["Enterprise RAG Assistant / Agent"]
    end
Loading

✅ Prerequisites

  1. Confluence Cloud:
    • API Token (recommended): Create an Atlassian API token for a dedicated read-only service account (username = Atlassian email, api-token = API token, auth-type: api_token, deployment-type: cloud). Grant the service account View permissions to the target Space(s).
    • OAuth 2.0 3LO (auth-type: oauth2): Register an Atlassian OAuth 2.0 app with read scopes (read:confluence-content.all, read:confluence-space.summary, read:confluence-props, read:confluence-groups, read:confluence-user, offline_access), configure cloud-id, client-id, client-secret, refresh-token, and set web-base-url to your tenant site (e.g. https://acme.atlassian.net/wiki) so citations stay clickable while API calls route through https://api.atlassian.com/ex/confluence/{cloudId}/wiki.
  2. Confluence Data Center / Server:
    • Personal Access Token (recommended): Issue a PAT for a read-only crawl user (auth-type: pat, api-token = PAT, deployment-type: data_center, base-url = full base URL including context path such as https://confluence.internal.corp/confluence).
    • Basic Auth: Set auth-type: basic with username and api-token (or password).
  3. Connection Verification: Click Test Connection in the Admin UI or run oc connector check --name <name> --type repository. The checker probes /api/v2/spaces?limit=1 (Cloud) or /rest/api/space?limit=1 (Data Center), verifies every configured space-keys entry, and checks reachability of root-page-id when set.

🚀 Key Features

  • Unified Cloud & Data Center Support:
    • Auto-detects cloud vs data_center when deployment-type: auto (based on cloud-id, .atlassian.net, or api.atlassian.com), or binds explicitly via cloud / data_center.
  • Space, Subtree & Label Scoping:
    • Filter spaces by space-keys (personal spaces ~user excluded by default unless include-personal-spaces: true).
    • Restrict crawling to a page subtree via root-page-id, toggle Space blog posts via include-blog-posts, and filter by included-labels / excluded-labels (excludes always take precedence, e.g. excluded-labels: draft,archived).
    • Emits Confluence labels as multi-value tags and confluence.labels metadata on both pages and their coupled attachments.
  • Structure-Aware Storage Format Conversion (#385):
    • Parses body.storage XHTML via jsoup's XML parser (Parser.xmlParser()) into clean semantic HTML5 preserving h1–h6, lists, tables (colspan/rowspan), blockquotes, code blocks, ac:task-list ([x]/[ ]), and ac:link titles.
    • Preserves content-bearing macros (expand, panel, info, note, warning, tip, status, excerpt, section, column, layout, code, noformat) and strips navigation boilerplate (toc, toc-zone, children, pagetree, pagetreesearch, jira, jiraissues, recently-updated, contributors, contentbylabel, livesearch, spaces, attachments, anchor, index), both customizable via keep-macros and drop-macros.
  • Coupled Binary Attachments, Cascading Tombstones & TOO_LARGE Reporting (#397):
    • Indexes MIME-filtered binary attachments (included-attachment-mime-types, excluded-attachment-mime-types excluding image/*, video/*, audio/*, and archives by default) as separate OIS documents inheriting the owning page's labels, breadcrumbs, and effective ACLs.
    • Streams downloads via auto-deleting TempFileInputStream while stripping Authorization across cross-origin CDN redirects.
    • Oversized attachments exceeding max-attachment-size-bytes (default 52428800 = 50 MB) emit opencrawling.document.status=TOO_LARGE, so JobOrchestrator records SKIPPED (errorCode="TOO_LARGE") in the execution ledger and a WARN event in traces.
    • Trashing, deleting, moving out of scope, or label-excluding a page automatically cascades DELETE tombstones to all its tracked attachments.
  • Incremental CQL Delta, Multi-Replica State (#395) & Periodic Full Reconciliation (#396):
    • cql_delta mode (default): Persists server-Date-anchored lastModifiedUtc, lastFullScanAt, renderSignature, and per-item (version, labelFingerprint, aclFingerprint, parentId, attachmentIds) in OpenCrawling's shared ControlPlaneStore (ControlPlaneConfluenceStateStore, key connector-state/confluence-<sha256>.json in oc_setting), allowing scheduled jobs to run on any runtime replica.
    • Queries modified pages/blogposts, modified attachments (reconciling owning pages even when only an attachment changed), and status = trashed content using a configurable cql-overlap-minutes lookback window.
    • Because Confluence does not bump version.number on label-only or restriction-only edits, full-scan-interval (default P7D in Admin UI, optionally constrained by full-scan-window and full-scan-timezone) runs periodic full reconciliation guarded by an empty-listing guard that prevents mass tombstoning on transient empty upstream listings.
  • Pre-Crawl Discovery & On-Demand Document Fetching:
    • Implements DiscoverableRepositoryConnector (explore, dryRun, crawlStart) and DocumentFetchingRepositoryConnector (fetchDocument) for pages, blog posts, and attachments with live page-by-page VisibilityFilter enforcement.

⚙️ Configuration Properties

Configure via Spring Boot properties (spring.opencrawling.connector.type=confluence and spring.opencrawling.connector.confluence.*) or via the Admin UI / Platform API / connectors.json (org.opencrawling.confluence.ConfluenceRepositoryConnector, camelCase keys).

Spring Property (spring.opencrawling.connector.confluence.*) connectors.json Key Default Description
base-url baseUrl https://example.atlassian.net/wiki Confluence base URL (Cloud /wiki or Data Center URL including context path)
web-base-url webBaseUrl (same as base-url) User-facing browser URL for clickable citations (recommended with OAuth 2.0 cloud-id)
deployment-type deploymentType cloud cloud, data_center, or auto
auth-type authType api_token api_token (Cloud email + token), pat (DC Bearer PAT), oauth2 (Cloud 3LO), or basic
username username — Atlassian email (api_token) or username (basic)
api-token apiToken — Cloud API token, Data Center PAT, or Basic password (masked in API/UI)
oauth2-token-url oauth2TokenUrl https://auth.atlassian.com/oauth/token OAuth 2.0 token endpoint URL
cloud-id, client-id, client-secret, refresh-token cloudId, clientId, clientSecret, refreshToken — Cloud OAuth 2.0 3LO credentials (auth-type: oauth2)
space-keys spaceKeys (all global spaces) Comma-separated Space keys (e.g., ENG,KB,SEC)
include-personal-spaces includePersonalSpaces false Include personal spaces (~user) when space-keys is empty
root-page-id rootPageId — Optional page ID restricting crawl to that page and its descendants
include-blog-posts includeBlogPosts true Index Space blog posts
include-attachments includeAttachments true Index coupled binary attachments
include-labels includeLabels true Emit Confluence page labels as tags and confluence.labels metadata
include-custom-properties includeCustomProperties true Emit custom content properties as confluence_prop_<key>
included-labels includedLabels — Optional label whitelist (at least one must match)
excluded-labels excludedLabels (empty in core; draft,archived in UI default) Label blacklist (always takes precedence over included-labels)
body-format bodyFormat storage storage (recommended XHTML→HTML5), export_view, or view
keep-macros keepMacros expand,panel,info,note,warning,tip,status,excerpt,section,column,layout,code,noformat Body-bearing macros converted to clean semantic HTML
drop-macros dropMacros toc,toc-zone,children,pagetree,pagetreesearch,jira,jiraissues,recently-updated,contributors,contentbylabel,livesearch,spaces,attachments,anchor,index Boilerplate/navigation macros stripped completely
included-attachment-mime-types includedAttachmentMimeTypes — Optional MIME allow-list (supports wildcards like application/pdf)
excluded-attachment-mime-types excludedAttachmentMimeTypes image/*,video/*,audio/*,application/zip,application/x-zip-compressed,application/gzip,application/octet-stream MIME types to skip ("" or "none" clears defaults)
max-attachment-size-bytes maxAttachmentSizeBytes 52428800 Maximum attachment size in bytes (50 MB); larger files report TOO_LARGE
crawl-mode crawlMode cql_delta cql_delta (incremental CQL) or tree (full space enumeration)
cql-overlap-minutes cqlOverlapMinutes 5 CQL watermark overlap lookback window in minutes
state-file stateFile (blank — shared ControlPlaneStore) Optional local JSON state file override
full-scan-interval fullScanInterval (empty; P7D in UI default) ISO-8601 duration for periodic full reconciliation (catches label/ACL drift)
full-scan-window / full-scan-timezone fullScanWindow / fullScanTimezone (anytime) / UTC Optional 24h maintenance window (e.g. 02:00-05:00) & IANA timezone
include-acls includeAcls true Enforce Space View + inherited ancestor/page read restrictions
user-identity userIdentity email_and_id email_and_id, email, or id (GDPR-safe Cloud accountId)
expand-group-members expandGroupMembers false Expand Confluence group members via /rest/api/group/...
allow-anonymous-public allowAnonymousPublic false Opt-in mapping of anonymous Space view to OIS public
page-size / parallelism pageSize / parallelism 50 / 4 API page size (1–250) and Virtual Thread concurrency
max-retries / initial-backoff-ms / timeout-seconds maxRetries / initialBackoffMs / timeoutSeconds 5 / 1000 / 30 Retry/backoff on HTTP 429/5xx and HTTP timeout controls

📄 Example application.yml

spring:
  opencrawling:
    repository-connector-type: confluence
    connector:
      type: confluence
      confluence:
        base-url: "https://acme.atlassian.net/wiki"
        deployment-type: "cloud"
        auth-type: "api_token"
        username: "crawler-bot@acme.com"
        api-token: "${CONFLUENCE_API_TOKEN}"
        space-keys: "ENG,KB,SEC"
        included-labels: "kb"
        excluded-labels: "draft,archived"
        include-blog-posts: true
        include-attachments: true
        body-format: "storage"
        max-attachment-size-bytes: 52428800
        crawl-mode: "cql_delta"
        cql-overlap-minutes: 5
        full-scan-interval: "P7D"
        full-scan-window: "02:00-05:00"
        full-scan-timezone: "UTC"
        include-acls: true
        user-identity: "email_and_id"
        allow-anonymous-public: false
        page-size: 50
        parallelism: 4

📦 OIS Document Mapping & Payloads

1. Page / Blog Post Document (action: "UPSERT")

Canonical URI format: confluence://<host>/spaces/<spaceKey>/pages/<contentId> (or /blogposts/<contentId>):

{
  "id": "confluence://acme.atlassian.net/spaces/ENG/pages/100",
  "action": "UPSERT",
  "acl": "alice@example.com,engineering,security",
  "metadata": {
    "source_type": ["confluence"],
    "title": ["Architecture Root"],
    "name": ["Architecture Root.html"],
    "filename": ["Architecture Root.html"],
    "mimeType": ["text/html"],
    "url": ["https://acme.atlassian.net/wiki/spaces/ENG/pages/100"],
    "confluence.webUrl": ["https://acme.atlassian.net/wiki/spaces/ENG/pages/100"],
    "path": ["/ENG/Engineering/Architecture Root.html"],
    "breadcrumbs": ["Engineering"],
    "confluence.breadcrumbs": ["Engineering"],
    "tags": ["kb", "architecture"],
    "confluence.labels": ["kb", "architecture"],
    "confluence.contentType": ["page"],
    "confluence.contentId": ["100"],
    "confluence.spaceKey": ["ENG"],
    "confluence.spaceId": ["10"],
    "confluence.version": ["1"],
    "confluence.versionKey": ["v:1"],
    "confluence_prop_owner": ["platform-team"]
  },
  "security": {
    "inheritanceEnabled": false,
    "permissions": [
      { "identity": "alice@example.com", "identityType": "user", "displayName": "Alice Engineer", "access": "read" },
      { "identity": "engineering", "identityType": "group", "displayName": "engineering", "access": "read" },
      { "identity": "security", "identityType": "group", "displayName": "security", "access": "read" }
    ]
  }
}

2. Coupled Binary Attachment (action: "UPSERT")

Canonical URI format: confluence://<host>/spaces/<spaceKey>/pages/<parentPageId>/attachments/<attachmentId>. Attachments automatically inherit the owning page's tags (confluence.labels), breadcrumbs, and effective security ACLs.

3. Cascading Deletion Tombstone (action: "DELETE")

When a page is deleted, trashed, moved out of scope, or excluded by labels (e.g. labeled archived), the connector emits a DELETE tombstone for the page and cascades DELETE tombstones for every tracked attachment ID in its state record:

{
  "id": "confluence://acme.atlassian.net/spaces/ENG/pages/100/attachments/att-pdf",
  "action": "DELETE",
  "metadata": {
    "source_type": ["confluence"],
    "confluence.contentId": ["att-pdf"],
    "confluence.spaceKey": ["ENG"],
    "confluence.contentType": ["attachment"],
    "confluence.status": ["DELETED"]
  }
}

4. Oversized Attachment Ledger Entry (TOO_LARGE)

Attachments exceeding max-attachment-size-bytes emit opencrawling.document.status=TOO_LARGE and opencrawling.document.reason=<detail>, which JobOrchestrator records as DocumentLedgerEntry.skipped(..., "TOO_LARGE", reason) in the audit ledger rather than failing the job.


🔒 Zero-Trust Security & ACL Mapping

For each item (include-acls: true), ConfluenceSecurityMapper combines Space View Permissions with every restricted ancestor and page read restriction (ignoring update restrictions, which do not restrict viewing in Confluence):

Confluence Permission Source Principal Emitted (identity) identityType Notes
Space View + Unrestricted Page Space read users & groups user / group Cloud group UUIDs resolved & cached via /rest/api/group/by-id
Inherited Ancestor / Page read Restrictions Intersection across every restricted ancestor level, the page, and Space View user / group Supports optional expand-group-members: true for Space-group ∩ page-user intersection
GDPR User Identity (user-identity) email_and_id (default), email, or id user Emits accountId/userKey cleanly when Atlassian Cloud GDPR settings hide emails
Anonymous Space Access public (only when allow-anonymous-public: true and page is unrestricted) public Ignored by default (allow-anonymous-public: false, fail-closed)
Empty / Unreadable Permissions (None — SecurityConfig.none()) — Fail-closed (permissions: [], acl: ""); never defaults to public

Querying with the Secure MCP Server (oc-mcp-server)

{
  "name": "secureVectorSearch",
  "arguments": {
    "query": "Confluence architecture ingestion pipeline security escalation",
    "userPrincipal": "alice@example.com",
    "userRoles": "engineering",
    "maxResults": 10
  }
}

🌐 Confluence REST API Endpoints Used

Operation Confluence Cloud (v2 + v1) Confluence Data Center (v1)
Connectivity Probe / List Spaces GET /api/v2/spaces GET /rest/api/space?expand=permissions
Space View Permissions GET /api/v2/spaces/{id}/permissions (+ /rest/api/group/by-id) Inline permissions on /rest/api/space/{key}?expand=permissions
List Current Pages / Blog Posts GET /api/v2/spaces/{id}/pages?status=current&body-format=..., /blogposts GET /rest/api/space/{key}/content/page?status=current&expand=..., /rest/api/content?type=blogpost
Fetch Single Page / Ancestors / Children GET /api/v2/pages/{id}, /ancestors, /children GET /rest/api/content/{id}?expand=..., /child/page
Labels & Custom Properties GET /api/v2/pages/{id}/labels, /properties Inline metadata.labels.results & GET /rest/api/content/{id}/property
Direct read Restrictions GET /rest/api/content/{id}/restriction/byOperation/read?expand=restrictions.user,restrictions.group Same v1 endpoint
Attachments & Binary Download GET /api/v2/pages/{id}/attachments + downloadLink (cross-origin auth stripped) GET /rest/api/content/{id}/child/attachment + _links.download
CQL Incremental Search GET /rest/api/content/search?cql=... Same v1 endpoint
OAuth 2.0 Token Refresh POST https://auth.atlassian.com/oauth/token N/A (PAT Bearer or Basic auth)

🖥️ Admin UI & CLI Integration

Admin UI (oc-admin-ui)

  1. Open Connectors → New Connector and select Atlassian Confluence (Cloud & Data Center) (org.opencrawling.confluence.ConfluenceRepositoryConnector).
  2. Select Deployment Type (cloud, data_center, or auto) and Authentication Mode (api_token, pat, oauth2, or basic). All secret fields (apiToken, clientSecret, refreshToken) are automatically masked as ********.
  3. Configure spaceKeys, optional rootPageId, includedLabels/excludedLabels, macro lists, attachment MIME filters & size cap, CQL periodic reconciliation schedule (fullScanInterval), and Zero-Trust ACL options.
  4. Click Test Connection to validate credentials, target spaces, and optional rootPageId.

OpenCrawling CLI (oc-cli)

# Validate Confluence credentials, target space keys, and rootPageId
oc connector check --name Confluence_Repository --type repository

🧪 Testing & Verification

1. Connector Unit & Embedded HTTP Mock Tests

mvn test -pl oc-confluence-repository-connector -am

Runs 19 unit and embedded HTTP mock tests (ConfluencePropertiesTest, ConfluenceStorageConverterTest, ConfluenceSecurityMapperTest, ConfluenceClientTest, ConfluenceRepositoryConnectorTest) covering Cloud v2+v1 and Data Center v1 parsing, OAuth 2.0 401 refresh, 429 Retry-After backoff, cross-origin CDN redirect auth stripping, Storage XHTML macro keep/drop conversion, label filtering + cascading attachment tombstones, TOO_LARGE reporting, inherited read restriction intersection, periodic reconciliation for ACL-only drift, empty-listing guard, and Discovery/DocumentFetching SPIs.

2. Standalone Emulator, Contract & Runtime Wiring Suite

./scripts/test-confluence-connector.sh

Starts the zero-dependency Python 3 Confluence Cloud & Data Center mock (oc-confluence-repository-connector/confluence-mock/confluence_mock.py), validates HTTP 401 rejection, Cloud API Token Basic auth (/wiki/api/v2/spaces, /pages, /attachments, PDF download), Data Center PAT Bearer auth (/confluence/rest/api/space, /content/page, /restriction/byOperation/read), and OAuth 2.0 /oauth/token, then executes all 19 connector tests and 158 oc-runtime wiring tests (ConnectorKindTest, ConnectorFactoryTest, ConnectorCheckerServiceTest, ControlPlaneConfluenceStateStoreTest verifying two independent JDBC replicas share CQL state and JobOrchestrator records TOO_LARGE attachments as SKIPPED, PlatformApiV1UnitTest secret masking for apiToken/personalAccessToken, and PlatformApiV1UiParityTest). Wired into .github/workflows/integration-tests.yml as "Confluence Connector Test Suite".

3. Full Decoupled Pipeline Integration Test (Kafka + Tika + Ollama + pgvector + MCP)

./scripts/test-confluence-decoupled.sh          # Production Dockerfiles
./scripts/test-confluence-decoupled.sh --local  # Fast local BuildKit ~/.m2 cache overlay
./scripts/test-confluence-decoupled.sh -s -k    # Skip build and keep containers for inspection

Boots oc-confluence-repository-connector/docker/docker-compose-decoupled-with-confluence.yml and verifies 9 end-to-end steps:

  1. Initial oc-crawler (cql_delta) crawl into pgvector (vector_store_1024): exact live count (5 docs covering Architecture page with kept info/expand macros and dropped toc/jira macros, Security Playbook page, Q4 Blog Post, fail-closed PUB anonymous page, and coupled Tika-extracted PDF attachment att-pdf), draft page & image/png attachment excluded (0 rows), 100 MB oversized PDF attachment reported as 1 too_large (0 rows, 0 failed).
  2. Rich metadata (title, tags, confluence_prop_owner), coupled attachment label & parent inheritance, fail-closed PUB anonymous space, and shared JDBC Control Plane lastModifiedUtc watermark persistence in PostgreSQL oc_setting.
  3. Secure MCP Server (secureVectorSearch) enforcement across Alice (engineering), Bob (security), and Outsider.
  4. Live Confluence mutations + incremental CQL delta sync (edit page 101, create new page 105, trash page 100 triggering cascading tombstone for coupled attachment att-pdf).
  5. Restriction-only update (without bumping lastModified) reconciled cleanly by a tree full reconciliation crawl.

⚠️ Known Limitations

  • Label & Permission Drift Between Full Scans: Confluence does not increment version.number (and in many deployments does not update CQL lastModified) when only labels or page restrictions are edited. Keep periodic full reconciliation enabled (full-scan-interval: P7D or shorter) in cql_delta jobs so label and ACL edits always converge.
  • Dynamic / Server-Side Macros: Macros that pull live external data at render time (jira, recently-updated, contentbylabel, livesearch, pagetree, children) are intentionally dropped in storage mode to avoid empty boilerplate chunks. Switch body-format to export_view if static server-rendered macro HTML is required.
  • Comments & Historical Revisions: Only current published versions (status=current) are indexed; inline/footer comments and old page revisions are not indexed as separate documents.

🗺️ Documentation


🎓 Tutorials


🧪 Testing


🤝 Community

Clone this wiki locally