Skip to content

Elasticsearch Output Connector

Piergiorgio Lucidi edited this page Nov 9, 2026 · 1 revision

Elasticsearch Output Connector

Elasticsearch Java Spring Boot OIS Compliance

Module: oc-elasticsearch-output-connector
Direct Connector Class: org.opencrawling.elasticsearch.ElasticsearchOutputConnector
Decoupled Kafka Writer: org.opencrawling.elasticsearch.messaging.ElasticsearchStoreWriterConsumer
MCP Search Backend: org.opencrawling.runtime.mcp.backend.ElasticsearchMcpSearchBackend
Spring Bean: @Component (activated when spring.opencrawling.output.type=elasticsearch)


📌 Overview

The Elasticsearch Output Connector integrates Elasticsearch (v8.x and v9.x) into the OpenCrawling platform as a first-class vector database and hybrid search store. Powered by the official Elastic Java API client (co.elastic.clients:elasticsearch-java over Apache HttpClient 5 Rest5Client), it delivers automated dense_vector index provisioning, HNSW and scalar/binary quantized ANN indexing (int8_hnsw, int4_hnsw, bbq_hnsw), normalized Zero-Trust ACL security_read_keys pre-filtering inside kNN queries, stale chunk pruning on re-crawls, OIS tombstone deletion lifecycle support, and native Model Context Protocol (MCP) retrieval.

flowchart TD
    subgraph Ingestion["OpenCrawling Ingestion & Embedding Pipeline"]
        Doc["Crawled RepositoryDocument"]
        Direct["ElasticsearchOutputConnector<br/>(Direct Non-Kafka Path)"]
        EmbedSvc["oc-embedding-consumer<br/>(Spring AI / Ollama / OpenAI)"]
        Kafka[("Kafka Topic<br/>(opencrawling-embedded)")]
        Writer["ElasticsearchStoreWriterConsumer<br/>(Decoupled Writer Stage)"]
    end

    subgraph MapperLayer["Unified Mapping & Client Layer"]
        Mapper["ElasticsearchDocumentMapper<br/>(EffectiveAcl + ReadKeys Normalization)"]
        Init["ElasticsearchIndexInitializer<br/>(dense_vector + keyword ACL Schema)"]
        Client["ElasticsearchClient<br/>(compatible-with=8 | 9, API Key / Basic / TLS)"]
    end

    subgraph Elastic["Elasticsearch 8.x / 9.x Cluster"]
        Index[("Target Index ('enterprise_kb')<br/>dense_vector (HNSW / int8_hnsw / bbq_hnsw)<br/>+ security_read_keys (keyword)")]
    end

    subgraph MCP["Secure MCP Server (oc-mcp-server)"]
        Backend["ElasticsearchMcpSearchBackend<br/>(Top-Level kNN + ElasticsearchAclQueries)"]
        Gate["DocumentAccessGate<br/>(Fail-Closed Authoritative Check)"]
    end

    Doc -->|Direct Mode| Direct
    Doc -->|Decoupled Mode| EmbedSvc
    EmbedSvc --> Kafka --> Writer
    Direct & Writer --> Mapper --> Client
    Init --> Client --> Index
    Index <-->|"kNN + security_read_keys Filter"| Backend
    Backend --> Gate
Loading

🚀 Key Features

  • Dual Elasticsearch 8.x & 9.x Compatibility: Uses the official co.elastic.clients:elasticsearch-java client (managed by the Spring Boot BOM) and sends Accept / Content-Type: application/vnd.elasticsearch+json; compatible-with=8 by default (compatible-with=8). This enables a single connector binary to work against both Elasticsearch 8.x natively and Elasticsearch 9.x via REST API compatibility, while allowing 9.x-only clusters to opt into native mode (compatible-with=9).
  • dense_vector HNSW & Vector Quantization: Automatically maps embeddings as an indexed dense_vector field with configurable dimensionality (dimensions, default 1024), similarity metrics (COSINE, DOT_PRODUCT, L2_NORM, MAX_INNER_PRODUCT), and ANN index types:
    • HNSW (default float32 HNSW graph)
    • INT8_HNSW (scalar quantization reducing vector memory footprint by ~75%)
    • INT4_HNSW (4-bit scalar quantization)
    • BBQ_HNSW (Better Binary Quantization HNSW)
    • FLAT and INT8_FLAT (brute-force exact vector scoring)
  • Resilient Index Auto-Provisioning: On startup (create-index=true), ElasticsearchIndexInitializer checks whether the target index exists and creates it with date_detection: false (preventing free-form string metadata from triggering date-parsing rejections) and ignore_malformed: true on lastModified. Startup initialization retries up to 30 times (with a 3-second delay) while waiting for containerized clusters to come online, and safely handles concurrent creation (resource_already_exists_exception).
  • Zero-Trust ACL Pre-Filtering (security_read_keys): Both write paths normalize document permissions via EffectiveAcl and ReadKeys.forAllowed(...) (u:<principal>, g:<group>, public) into the security_read_keys keyword field alongside security_allowed_read, security_denied_read, and security_inheritance. During MCP search, ElasticsearchAclQueries.readableBy(...) injects a terms filter on security_read_keys directly inside the Elasticsearch top-level knn query, and every candidate hit is subsequently re-verified by DocumentAccessGate (where explicit denies always take precedence).
  • Re-Crawl Stale Chunk Cleanup:
    • Direct Path (ElasticsearchOutputConnector): Uses deterministic chunk IDs (<documentId>_<chunkIndex>), purges any existing chunks of documentId via deleteByQuery (Conflicts.Proceed), and bulk-indexes the new chunks in batches (bulk-batch-size, default 500).
    • Decoupled Kafka Path (ElasticsearchStoreWriterConsumer): When chunk 0 of a re-crawled document arrives with total_chunks, automatically executes purgeChunksPastNewLength (deleteByQuery where documentId == docId and chunk_index >= total_chunks) so shortened documents never leave trailing chunks with outdated text or permissions.
  • OIS Document Lifecycle & Tombstone Deletes: Processes DocumentAction.DELETE tombstones on both the direct and Kafka consumer paths by executing deleteByQuery (Conflicts.Proceed) matching documentId or chunk id.
  • Flexible Authentication & TLS Trust:
    • API Key Authentication: Supports Base64 API keys (api-key) for Elastic Cloud and Elastic Serverless (takes precedence over basic auth).
    • HTTP Basic Authentication: Supports username (default elastic) and password.
    • TLS / SSL: Supports cluster CA SHA-256 certificate fingerprints (ssl.ca-fingerprint for Elasticsearch 8/9 auto-generated certificates) or standard PKCS12 / JKS truststores (ssl.trust-store-path, ssl.trust-store-password, ssl.trust-store-type).
  • Admin UI & Cluster Identity Verification: Integrated into the OpenCrawling Admin UI (ConnectorForm.tsx) and ConnectorCheckerService, which verifies the X-Elastic-Product: Elasticsearch HTTP header to distinguish Elasticsearch from OpenSearch clusters running on port 9200.

⚙️ Configuration Properties

All properties are bound under spring.opencrawling.output.elasticsearch.* (ElasticsearchOutputProperties):

Property Environment Variable Default Description
spring.opencrawling.output.elasticsearch.uris SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_URIS http://localhost:9200 One node URL or a comma-separated list of Elasticsearch node URLs
spring.opencrawling.output.elasticsearch.api-key SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_API_KEY "" Base64 API key (Elastic Cloud / Serverless); takes precedence over basic auth
spring.opencrawling.output.elasticsearch.username SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_USERNAME elastic Basic-auth username; used only when password is non-empty and api-key is blank
spring.opencrawling.output.elasticsearch.password SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_PASSWORD "" Basic-auth password
spring.opencrawling.output.elasticsearch.index-name SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_INDEX_NAME enterprise_kb Target Elasticsearch index name
spring.opencrawling.output.elasticsearch.dimensions SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_DIMENSIONS 1024 dense_vector dimensionality (must match the active embedding model, e.g. 1024 for mxbai-embed-large)
spring.opencrawling.output.elasticsearch.similarity SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_SIMILARITY COSINE Vector similarity metric: COSINE, DOT_PRODUCT, L2_NORM, or MAX_INNER_PRODUCT
spring.opencrawling.output.elasticsearch.index-type SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_INDEX_TYPE HNSW ANN index type: HNSW, INT8_HNSW, INT4_HNSW, BBQ_HNSW, FLAT, or INT8_FLAT
spring.opencrawling.output.elasticsearch.bulk-batch-size SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_BULK_BATCH_SIZE 500 Maximum number of chunks sent in a single _bulk request on the direct path
spring.opencrawling.output.elasticsearch.create-index SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_CREATE_INDEX true Automatically create the target index with the expected mapping on startup if missing
spring.opencrawling.output.elasticsearch.compatible-with SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_COMPATIBLE_WITH 8 REST API major version requested via compatible-with header (8 works with 8.x and 9.x; 9 uses native 9.x headers; 0 keeps client default)
spring.opencrawling.output.elasticsearch.ssl.ca-fingerprint SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_SSL_CA_FINGERPRINT "" SHA-256 fingerprint of the cluster CA certificate (alternative to a truststore)
spring.opencrawling.output.elasticsearch.ssl.trust-store-path SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_SSL_TRUST_STORE_PATH "" File path to a truststore holding the cluster CA certificate
spring.opencrawling.output.elasticsearch.ssl.trust-store-password SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_SSL_TRUST_STORE_PASSWORD "" Truststore password
spring.opencrawling.output.elasticsearch.ssl.trust-store-type SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_SSL_TRUST_STORE_TYPE PKCS12 Truststore format (PKCS12 or JKS)

Note

Changing dimensions, similarity, or index-type only affects newly created indexes. To apply new vector parameters to existing data, reindex or configure a new index-name.


📋 Index Mapping & Document Schema

1. Auto-Provisioned Index Mapping (ElasticsearchIndexInitializer)

When create-index=true and the target index does not yet exist, ElasticsearchIndexInitializer creates the index with the following mapping:

{
  "mappings": {
    "date_detection": false,
    "properties": {
      "id": { "type": "keyword" },
      "documentId": { "type": "keyword" },
      "text": { "type": "text" },
      "uri": { "type": "keyword" },
      "acl": { "type": "keyword" },
      "lastModified": { "type": "date", "ignore_malformed": true },
      "security_inheritance": { "type": "boolean" },
      "security_allowed_read": { "type": "keyword" },
      "security_denied_read": { "type": "keyword" },
      "security_read_keys": { "type": "keyword" },
      "embeddings": {
        "type": "dense_vector",
        "dims": 1024,
        "index": true,
        "similarity": "cosine",
        "index_options": {
          "type": "hnsw"
        }
      }
    }
  }
}

2. Stored Chunk Document (ElasticsearchDocumentMapper)

Both ElasticsearchOutputConnector and ElasticsearchStoreWriterConsumer map chunks through ElasticsearchDocumentMapper, protecting reserved system and security fields (RESERVED_FIELDS) so arbitrary source metadata cannot overwrite them:

{
  "id": "doc-001_0",
  "documentId": "doc-001",
  "text": "Quarterly engineering roadmap and architecture milestones...",
  "uri": "file:///kb/doc-001.txt",
  "acl": "alice,engineering",
  "lastModified": "2026-01-02T03:04:05Z",
  "security_inheritance": true,
  "security_allowed_read": ["user:alice", "group:engineering"],
  "security_denied_read": ["user:mallory"],
  "security_read_keys": ["u:alice", "g:engineering"],
  "embeddings": [0.0123, -0.0456, 0.0891, ...],
  "title": ["IT doc-001"],
  "chunk_index": 0,
  "total_chunks": 3
}

🛡️ Zero-Trust MCP Search Backend (ElasticsearchMcpSearchBackend)

When spring.opencrawling.output.type=elasticsearch and opencrawling.mcp.server.enabled=true (the default), oc-runtime activates ElasticsearchMcpSearchBackend:

  1. In-Query kNN ACL Pre-Filtering: In metadata and shadow authorization modes (storePrefilterAllowed() == true), ElasticsearchAclQueries.readableBy(userPrincipal, userRoles) constructs a terms filter on security_read_keys matching the caller's normalized read keys (u:<principal>, g:<role>, and public) and attaches it directly to the top-level knn search clause (num_candidates = max(topK * 2, 100)).
  2. Vector Payload Exclusion: Search requests use _source filtering to exclude the embeddings float array (NO_VECTORS), reducing network payload overhead when returning metadata and text chunks to MCP clients.
  3. Authoritative Access Gate (DocumentAccessGate): Every candidate hit returned by Elasticsearch is passed through DocumentAccessGate (readable(...)), which evaluates the configured AuthorizationProvider (metadata, shadow, intersection, or openfga) with strict fail-closed semantics and explicit deny precedence.

🔍 Executing Queries against Elasticsearch

1. Document Chunk Count

curl -s "http://localhost:9200/enterprise_kb/_count" | jq .

2. Full-Text Search

curl -s "http://localhost:9200/enterprise_kb/_search" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "match": {
        "text": "OpenCrawling"
      }
    },
    "_source": {
      "excludes": ["embeddings"]
    }
  }' | jq .

3. Dense Vector kNN Search with Zero-Trust ACL Pre-Filter

curl -s "http://localhost:9200/enterprise_kb/_search" \
  -H "Content-Type: application/json" \
  -d '{
    "size": 5,
    "knn": {
      "field": "embeddings",
      "query_vector": [0.0123, -0.0456, 0.0891, ...],
      "k": 5,
      "num_candidates": 100,
      "filter": {
        "terms": {
          "security_read_keys": ["public", "u:alice", "g:engineering"]
        }
      }
    },
    "_source": ["id", "documentId", "uri", "text", "acl", "lastModified"]
  }' | jq .

🐳 Docker & Integration Test

1. Connector Unit & Testcontainers Integration Suite (ES 8.x & 9.x)

Run the standalone connector test script, which executes unit tests plus Testcontainers integration tests (Elasticsearch8OutputConnectorIT on Elasticsearch 8.19.20 and Elasticsearch9OutputConnectorIT on Elasticsearch 9.4.3, testing both compatible-with=8 and native compatible-with=9):

chmod +x scripts/test-elasticsearch-connector.sh
./scripts/test-elasticsearch-connector.sh

2. Decoupled Multi-Service Docker Compose Stack

The decoupled pipeline definition is located at oc-elasticsearch-output-connector/docker/docker-compose-decoupled-with-elasticsearch.yml and orchestrates:

  • Elasticsearch (elasticsearch-standalone): docker.elastic.co/elasticsearch/elasticsearch:${ELASTICSEARCH_VERSION:-8.19.20} on port 9200 (configurable to 9.4.3 or any 8.x/9.x tag via ELASTICSEARCH_VERSION).
  • Apache Kafka (kafka-decoupled-elasticsearch): KRaft message broker on port 9092.
  • Redis Stack (redis-stack-decoupled-elasticsearch): Distributed cache on ports 6379 / 8001.
  • Apache Ozone 2.2.0 (ozone-scm, ozone-om, ozone-datanode, ozone-s3g): Claim Check binary object store.
  • Ollama & Model Puller: Pulls mxbai-embed-large (1024d), nomic-embed-text (768d), and all-minilm (384d).
  • OpenCrawling Microservices: oc-crawler, oc-ingestion-consumer, oc-embedding-consumer, oc-writer-consumer, oc-mcp-server (port 8080), and oc-admin-ui (port 3000).
# Start with default Elasticsearch 8.19.20
docker compose -f oc-elasticsearch-output-connector/docker/docker-compose-decoupled-with-elasticsearch.yml up -d

# Or start with Elasticsearch 9.4.3
ELASTICSEARCH_VERSION=9.4.3 docker compose -f oc-elasticsearch-output-connector/docker/docker-compose-decoupled-with-elasticsearch.yml up -d

3. Automated Decoupled End-to-End Test Script

Run the full decoupled pipeline integration test:

chmod +x scripts/test-elasticsearch-decoupled.sh
./scripts/test-elasticsearch-decoupled.sh

# Or test against Elasticsearch 9.x:
ELASTICSEARCH_VERSION=9.4.3 ./scripts/test-elasticsearch-decoupled.sh

The script verifies:

  1. Container image builds and startup of Elasticsearch, Ollama, Redis, Kafka, Apache Ozone, and all OpenCrawling microservices.
  2. X-Elastic-Product: Elasticsearch header verification on http://localhost:9200.
  3. Automatic creation of the enterprise_kb index with dense_vector (dims: 1024) and security_read_keys (keyword) mappings.
  4. End-to-end filesystem crawl $\rightarrow$ Kafka $\rightarrow$ Tika text extraction & chunking $\rightarrow$ Ollama 1024-dim embedding $\rightarrow$ ElasticsearchStoreWriterConsumer persistence.
  5. Verification that stored chunks carry 1024-dimensional embeddings and populated security_read_keys.
  6. Live kNN vector search with the security_read_keys terms pre-filter.
  7. oc-mcp-server reachability on http://localhost:8080/mcp and confirmation that ElasticsearchMcpSearchBackend is active.
  8. OIS DELETE tombstone lifecycle verification via ElasticsearchStoreWriterConsumerTest.

🗺️ Documentation


🎓 Tutorials


🧪 Testing


🤝 Community

Clone this wiki locally