Flowable Repository Connector

Flowable Repository Connector

Module: oc-flowable-repository-connector
Connector Class: org.opencrawling.flowable.FlowableRepositoryConnector
Security Mapper: org.opencrawling.flowable.FlowableSecurityMapper
Spring Bean: @Component (activated when spring.opencrawling.connector.type=flowable or CONNECTOR_TYPE=flowable)

---

📌 Overview

The Flowable Repository Connector enables OpenCrawling to crawl, extract, and index workflow instances, historic process executions, BPMN variables, and runtime/historic identity links from Flowable BPMN Process Engines (7.x+) via the Flowable REST API (/flowable-rest/service).

By uniting Flowable with the Open Ingestion Standard (OIS), Java 25 Virtual Threads (StructuredTaskScope), and zero-trust security mapping, this connector allows AI agents, LLM assistants, and enterprise RAG systems to query running and completed business workflows, identify operational bottlenecks, and perform semantic similarity searches that strictly adhere to Flowable's native authorization model.

flowchart TD
    subgraph Flowable["Flowable BPMN Engine (v7.x+)"]
        REST["Flowable REST API (/flowable-rest/service)"]
        HPI["Historic Process Instances (/history/historic-process-instances)"]
        HVI["Historic Variables (/history/historic-variable-instances)"]
        HIL["Historic Identity Links (/history/historic-process-instances/{id}/identitylinks)"]
        
        REST --> HPI
        REST --> HVI
        REST --> HIL
    end

    subgraph OpenCrawling["OpenCrawling Ingestion Runtime (Java 25)"]
        Connector["FlowableRepositoryConnector (oc-flowable-repository-connector)"]
        SecMapper["FlowableSecurityMapper (Identity Link Resolution)"]
        VThreads["Virtual Threads (StructuredTaskScope)"]
        
        REST <-->|HTTP/REST Basic Auth| Connector
        Connector --> VThreads
        VThreads --> SecMapper
        
        OIS["OIS Document Builder (RepositoryDocument with SecurityConfig & Entity Links)"]
        SecMapper --> OIS
        VThreads --> OIS
    end

    subgraph Destinations["Dual-Mode Zero-Trust Storage & Retrieval"]
        direction TB
        subgraph Collocated["1. Collocated DB Pattern (pgvector on Flowable PostgreSQL)"]
            PGVector[("PostgreSQL / pgvector (vector_store table)")]
            FlowableDB[("Flowable Tables: ACT_RU_IDENTITYLINK / ACT_HI_IDENTITYLINK")]
            SQLPushdown["Query-Time SQL EXISTS Pushdown (Zero Sync Lag, No Chunk Bloat)"]
            PGVector <-->|Direct SQL JOIN / Subquery| FlowableDB
            SQLPushdown --> PGVector
        end
        
        subgraph Distributed["2. Distributed DB Pattern (External Vector DBs)"]
            ExtVector[("Qdrant / Milvus / OpenSearch / Vespa")]
            SecFilter["Metadata Pre-Filtering (security_allowed_read / security_denied_read)"]
            SecFilter --> ExtVector
        end
    end

    OIS -->|Collocated Architecture| PGVector
    OIS -->|Decoupled Architecture| ExtVector

    subgraph AI["Enterprise AI & RAG Orchestrators"]
        Agents["AI Agents / Spring AI RAG / MCP Assistants"]
        Agents --> SQLPushdown
        Agents --> SecFilter
    end

Loading

---

🚀 Key Features

• Historic & Active Process Instance Crawling:
  • Paginates through historical and running workflow instances (GET /history/historic-process-instances).
  • Ingests core execution metadata: processDefinitionId, processDefinitionKey, businessKey, startUserId, startTime, endTime, durationInMillis, and tenantId.
  • Configurable scan scope: all, completed, or active.
• Dynamic BPMN Variables Extraction:
  • Fetches and parses process variables (/history/historic-variable-instances), indexing them as metadata attributes prefixed with flowable_var_<varName> and within the document JSON content body.
• Identity Links & Zero-Trust Security Mapping:
  • Queries /history/historic-process-instances/{id}/identitylinks for participants, candidates, owners, and starters.
  • FlowableSecurityMapper translates candidate groups and user links into OIS SecurityConfig and PermissionRule records, deduplicating identities and distinguishing user vs. group principals.
• Collocated Database Pattern (Entity-Level Linking):
  • Eliminates distributed transaction synchronization by stamping entity reference metadata (flowable_proc_inst_id, flowable_scope_id, flowable_scope_type = "processInstance").
  • Enables query-time authorization via SQL EXISTS subqueries against native ACT_RU_IDENTITYLINK and ACT_HI_IDENTITYLINK tables without duplicating identity links per vector chunk.
• Distributed Database Pattern (OIS Metadata Pre-Filtering):
  • Emits security_allowed_read, security_denied_read, and security_inheritance for external vector databases (Qdrant, Milvus, OpenSearch, Vespa).
• Java 25 Structured Concurrency:
  • Uses Java 25 Virtual Threads (Thread.ofVirtual()) and StructuredTaskScope.open() to process instance batches, variable queries, and identity links concurrently with high throughput.

---

⚙️ Configuration Properties

Configure the connector via application.yml, connectors.json (Admin UI), or environment variables:

Property Key	Environment Variable	Default	Description
spring.opencrawling.connector.flowable.url	FLOWABLE_URL	http://localhost:8080/flowable-rest/service	Base URL of Flowable REST service
spring.opencrawling.connector.flowable.username	FLOWABLE_USERNAME	admin	Basic auth username
spring.opencrawling.connector.flowable.password	FLOWABLE_PASSWORD	test	Basic auth password
spring.opencrawling.connector.flowable.batch-size	FLOWABLE_BATCH_SIZE	100	Process instances page batch size
spring.opencrawling.connector.flowable.process-definition-key	FLOWABLE_PROCESS_DEFINITION_KEY	""	Optional process definition key filter
spring.opencrawling.connector.flowable.include-variables	FLOWABLE_INCLUDE_VARIABLES	true	Extract historic BPMN process variables
spring.opencrawling.connector.flowable.include-acls	FLOWABLE_INCLUDE_ACLS	true	Extract identity links and map to SecurityConfig
spring.opencrawling.connector.flowable.scope	FLOWABLE_SCOPE	all	Filter by state: all, completed, or active

---

📂 SCAN_PATH Format

When launching a crawl job from the CLI, Web UI, or REST API:

Format	Behaviour
(empty) or /	Crawls all process instances according to configured scope
invoice-process or /invoice-process	Filters process instances to the specific process definition key invoice-process

---

📑 Metadata & Document Structure

Every crawled process instance is converted into a standard RepositoryDocument:

{
  "id": "proc-1234",
  "uri": "flowable://process-instances/proc-1234",
  "action": "UPSERT",
  "metadata": {
    "mimeType": ["application/json"],
    "processDefinitionId": ["invoice-process:2:98765"],
    "processDefinitionKey": ["invoice-process"],
    "businessKey": ["INV-2026-001"],
    "startUserId": ["finance_agent_1"],
    "startTime": ["2026-07-23T08:00:00Z"],
    "endTime": ["2026-07-23T08:15:00Z"],
    "durationInMillis": ["900000"],
    "flowable_proc_inst_id": ["proc-1234"],
    "flowable_process_instance_id": ["proc-1234"],
    "flowable_scope_id": ["proc-1234"],
    "flowable_scope_type": ["processInstance"],
    "flowable_identity_users": ["finance_agent_1"],
    "flowable_identity_groups": ["finance_managers"],
    "flowable_var_totalAmount": ["250.50"],
    "flowable_var_customerName": ["ACME Corp"]
  },
  "security": {
    "inheritanceEnabled": false,
    "permissions": [
      {"identity": "finance_agent_1", "identityType": "user", "access": "read"},
      {"identity": "finance_managers", "identityType": "group", "access": "read"}
    ]
  }
}

---

🛡️ Dual-Mode Zero-Trust Architecture

See Zero-Trust Security & ACL Management for the comprehensive architecture covering Flowable, Camunda, Alfresco, and CMIS.

1. Collocated Database Pattern (Query Pushdown)

When the Flowable engine and the vector database share the same underlying PostgreSQL instance running pgvector:

• No Distributed Sync Lag: Permissions are evaluated at query time using Flowable's native tables.
• No Chunk Row Explosion: Instead of creating an identity link row for every vector chunk, OpenCrawling stamps the parent entity reference (flowable_proc_inst_id).
• Query Pushdown SQL: Similarity searches join vector distance calculation with Flowable's runtime and historical identity tables:

SELECT *, embedding <=> :queryEmbedding AS distance 
FROM vector_store 
WHERE (
    -- Runtime identity check for active workflows
    EXISTS (
        SELECT 1 FROM ACT_RU_IDENTITYLINK I 
        WHERE I.PROC_INST_ID_ = (metadata->>'flowable_proc_inst_id')
          AND (I.USER_ID_ = :currentUserId OR I.GROUP_ID_ IN (:currentUserGroups))
    )
    -- Historical identity check for completed workflows
    OR EXISTS (
        SELECT 1 FROM ACT_HI_IDENTITYLINK HI 
        WHERE HI.PROC_INST_ID_ = (metadata->>'flowable_proc_inst_id')
          AND (HI.USER_ID_ = :currentUserId OR HI.GROUP_ID_ IN (:currentUserGroups))
    )
)
ORDER BY distance 
LIMIT :topK;

2. Distributed Database Pattern (OIS Metadata Pre-Filtering)

When indexing into standalone vector databases (Qdrant, Milvus, OpenSearch, Vespa):

• OpenCrawling maps the document's SecurityConfig into vector payload fields:

  {
    "security_allowed_read": ["finance_agent_1", "finance_managers"],
    "security_denied_read": [],
    "security_inheritance": false
  }

• At query time, the AI orchestrator passes the authenticated user's ID and group memberships as a metadata filter to the vector search engine.

---

🧪 Testing & Verification

Running Unit Tests

mvn test -pl oc-flowable-repository-connector

Running the Decoupled Integration Test

The project includes a multi-service integration test suite with Docker Compose:

./scripts/test-flowable-decoupled.sh

This script:

1. Starts Flowable REST (flowable/flowable-rest:7.1.0), PostgreSQL pgvector, Redis Stack, Ollama, Kafka, and OpenCrawling decoupled microservices.
2. Seeds a sample BPMN workflow (invoiceApproval.bpmn20.xml) with variables and process executions via scripts/seed-flowable-workflows.sh.
3. Runs oc-crawler-service to crawl the Flowable engine.
4. Verifies embedding generation, vector persistence in PostgreSQL, and semantic query responses via the OpenCrawling Secure MCP Server.