Repository navigation
Elasticsearch Output Connector
Module: oc-elasticsearch-output-connector
Direct Connector Class: org.opencrawling.elasticsearch.ElasticsearchOutputConnector
Decoupled Kafka Writer: org.opencrawling.elasticsearch.messaging.ElasticsearchStoreWriterConsumer
MCP Search Backend: org.opencrawling.runtime.mcp.backend.ElasticsearchMcpSearchBackend
Spring Bean: @Component (activated when spring.opencrawling.output.type=elasticsearch)
The Elasticsearch Output Connector integrates Elasticsearch (v8.x and v9.x) into the OpenCrawling platform as a first-class vector database and hybrid search store. Powered by the official Elastic Java API client (co.elastic.clients:elasticsearch-java over Apache HttpClient 5 Rest5Client), it delivers automated dense_vector index provisioning, HNSW and scalar/binary quantized ANN indexing (int8_hnsw, int4_hnsw, bbq_hnsw), normalized Zero-Trust ACL security_read_keys pre-filtering inside kNN queries, stale chunk pruning on re-crawls, OIS tombstone deletion lifecycle support, and native Model Context Protocol (MCP) retrieval.
flowchart TD
subgraph Ingestion["OpenCrawling Ingestion & Embedding Pipeline"]
Doc["Crawled RepositoryDocument"]
Direct["ElasticsearchOutputConnector<br/>(Direct Non-Kafka Path)"]
EmbedSvc["oc-embedding-consumer<br/>(Spring AI / Ollama / OpenAI)"]
Kafka[("Kafka Topic<br/>(opencrawling-embedded)")]
Writer["ElasticsearchStoreWriterConsumer<br/>(Decoupled Writer Stage)"]
end
subgraph MapperLayer["Unified Mapping & Client Layer"]
Mapper["ElasticsearchDocumentMapper<br/>(EffectiveAcl + ReadKeys Normalization)"]
Init["ElasticsearchIndexInitializer<br/>(dense_vector + keyword ACL Schema)"]
Client["ElasticsearchClient<br/>(compatible-with=8 | 9, API Key / Basic / TLS)"]
end
subgraph Elastic["Elasticsearch 8.x / 9.x Cluster"]
Index[("Target Index ('enterprise_kb')<br/>dense_vector (HNSW / int8_hnsw / bbq_hnsw)<br/>+ security_read_keys (keyword)")]
end
subgraph MCP["Secure MCP Server (oc-mcp-server)"]
Backend["ElasticsearchMcpSearchBackend<br/>(Top-Level kNN + ElasticsearchAclQueries)"]
Gate["DocumentAccessGate<br/>(Fail-Closed Authoritative Check)"]
end
Doc -->|Direct Mode| Direct
Doc -->|Decoupled Mode| EmbedSvc
EmbedSvc --> Kafka --> Writer
Direct & Writer --> Mapper --> Client
Init --> Client --> Index
Index <-->|"kNN + security_read_keys Filter"| Backend
Backend --> Gate
-
Dual Elasticsearch 8.x & 9.x Compatibility: Uses the official
co.elastic.clients:elasticsearch-javaclient (managed by the Spring Boot BOM) and sendsAccept/Content-Type: application/vnd.elasticsearch+json; compatible-with=8by default (compatible-with=8). This enables a single connector binary to work against both Elasticsearch 8.x natively and Elasticsearch 9.x via REST API compatibility, while allowing 9.x-only clusters to opt into native mode (compatible-with=9). -
dense_vectorHNSW & Vector Quantization: Automatically mapsembeddingsas an indexeddense_vectorfield with configurable dimensionality (dimensions, default1024), similarity metrics (COSINE,DOT_PRODUCT,L2_NORM,MAX_INNER_PRODUCT), and ANN index types:-
HNSW(defaultfloat32HNSW graph) -
INT8_HNSW(scalar quantization reducing vector memory footprint by ~75%) -
INT4_HNSW(4-bit scalar quantization) -
BBQ_HNSW(Better Binary Quantization HNSW) -
FLATandINT8_FLAT(brute-force exact vector scoring)
-
-
Resilient Index Auto-Provisioning: On startup (
create-index=true),ElasticsearchIndexInitializerchecks whether the target index exists and creates it withdate_detection: false(preventing free-form string metadata from triggering date-parsing rejections) andignore_malformed: trueonlastModified. Startup initialization retries up to 30 times (with a 3-second delay) while waiting for containerized clusters to come online, and safely handles concurrent creation (resource_already_exists_exception). -
Zero-Trust ACL Pre-Filtering (
security_read_keys): Both write paths normalize document permissions viaEffectiveAclandReadKeys.forAllowed(...)(u:<principal>,g:<group>,public) into thesecurity_read_keyskeyword field alongsidesecurity_allowed_read,security_denied_read, andsecurity_inheritance. During MCP search,ElasticsearchAclQueries.readableBy(...)injects atermsfilter onsecurity_read_keysdirectly inside the Elasticsearch top-levelknnquery, and every candidate hit is subsequently re-verified byDocumentAccessGate(where explicit denies always take precedence). -
Re-Crawl Stale Chunk Cleanup:
-
Direct Path (
ElasticsearchOutputConnector): Uses deterministic chunk IDs (<documentId>_<chunkIndex>), purges any existing chunks ofdocumentIdviadeleteByQuery(Conflicts.Proceed), and bulk-indexes the new chunks in batches (bulk-batch-size, default500). -
Decoupled Kafka Path (
ElasticsearchStoreWriterConsumer): When chunk0of a re-crawled document arrives withtotal_chunks, automatically executespurgeChunksPastNewLength(deleteByQuerywheredocumentId == docIdandchunk_index >= total_chunks) so shortened documents never leave trailing chunks with outdated text or permissions.
-
Direct Path (
-
OIS Document Lifecycle & Tombstone Deletes: Processes
DocumentAction.DELETEtombstones on both the direct and Kafka consumer paths by executingdeleteByQuery(Conflicts.Proceed) matchingdocumentIdor chunkid. -
Flexible Authentication & TLS Trust:
-
API Key Authentication: Supports Base64 API keys (
api-key) for Elastic Cloud and Elastic Serverless (takes precedence over basic auth). -
HTTP Basic Authentication: Supports
username(defaultelastic) andpassword. -
TLS / SSL: Supports cluster CA SHA-256 certificate fingerprints (
ssl.ca-fingerprintfor Elasticsearch 8/9 auto-generated certificates) or standardPKCS12/JKStruststores (ssl.trust-store-path,ssl.trust-store-password,ssl.trust-store-type).
-
API Key Authentication: Supports Base64 API keys (
-
Admin UI & Cluster Identity Verification: Integrated into the OpenCrawling Admin UI (
ConnectorForm.tsx) andConnectorCheckerService, which verifies theX-Elastic-Product: ElasticsearchHTTP header to distinguish Elasticsearch from OpenSearch clusters running on port9200.
All properties are bound under spring.opencrawling.output.elasticsearch.* (ElasticsearchOutputProperties):
| Property | Environment Variable | Default | Description |
|---|---|---|---|
spring.opencrawling.output.elasticsearch.uris |
SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_URIS |
http://localhost:9200 |
One node URL or a comma-separated list of Elasticsearch node URLs |
spring.opencrawling.output.elasticsearch.api-key |
SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_API_KEY |
"" |
Base64 API key (Elastic Cloud / Serverless); takes precedence over basic auth |
spring.opencrawling.output.elasticsearch.username |
SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_USERNAME |
elastic |
Basic-auth username; used only when password is non-empty and api-key is blank |
spring.opencrawling.output.elasticsearch.password |
SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_PASSWORD |
"" |
Basic-auth password |
spring.opencrawling.output.elasticsearch.index-name |
SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_INDEX_NAME |
enterprise_kb |
Target Elasticsearch index name |
spring.opencrawling.output.elasticsearch.dimensions |
SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_DIMENSIONS |
1024 |
dense_vector dimensionality (must match the active embedding model, e.g. 1024 for mxbai-embed-large) |
spring.opencrawling.output.elasticsearch.similarity |
SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_SIMILARITY |
COSINE |
Vector similarity metric: COSINE, DOT_PRODUCT, L2_NORM, or MAX_INNER_PRODUCT
|
spring.opencrawling.output.elasticsearch.index-type |
SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_INDEX_TYPE |
HNSW |
ANN index type: HNSW, INT8_HNSW, INT4_HNSW, BBQ_HNSW, FLAT, or INT8_FLAT
|
spring.opencrawling.output.elasticsearch.bulk-batch-size |
SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_BULK_BATCH_SIZE |
500 |
Maximum number of chunks sent in a single _bulk request on the direct path |
spring.opencrawling.output.elasticsearch.create-index |
SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_CREATE_INDEX |
true |
Automatically create the target index with the expected mapping on startup if missing |
spring.opencrawling.output.elasticsearch.compatible-with |
SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_COMPATIBLE_WITH |
8 |
REST API major version requested via compatible-with header (8 works with 8.x and 9.x; 9 uses native 9.x headers; 0 keeps client default) |
spring.opencrawling.output.elasticsearch.ssl.ca-fingerprint |
SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_SSL_CA_FINGERPRINT |
"" |
SHA-256 fingerprint of the cluster CA certificate (alternative to a truststore) |
spring.opencrawling.output.elasticsearch.ssl.trust-store-path |
SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_SSL_TRUST_STORE_PATH |
"" |
File path to a truststore holding the cluster CA certificate |
spring.opencrawling.output.elasticsearch.ssl.trust-store-password |
SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_SSL_TRUST_STORE_PASSWORD |
"" |
Truststore password |
spring.opencrawling.output.elasticsearch.ssl.trust-store-type |
SPRING_OPENCRAWLING_OUTPUT_ELASTICSEARCH_SSL_TRUST_STORE_TYPE |
PKCS12 |
Truststore format (PKCS12 or JKS) |
Note
Changing dimensions, similarity, or index-type only affects newly created indexes. To apply new vector parameters to existing data, reindex or configure a new index-name.
When create-index=true and the target index does not yet exist, ElasticsearchIndexInitializer creates the index with the following mapping:
{
"mappings": {
"date_detection": false,
"properties": {
"id": { "type": "keyword" },
"documentId": { "type": "keyword" },
"text": { "type": "text" },
"uri": { "type": "keyword" },
"acl": { "type": "keyword" },
"lastModified": { "type": "date", "ignore_malformed": true },
"security_inheritance": { "type": "boolean" },
"security_allowed_read": { "type": "keyword" },
"security_denied_read": { "type": "keyword" },
"security_read_keys": { "type": "keyword" },
"embeddings": {
"type": "dense_vector",
"dims": 1024,
"index": true,
"similarity": "cosine",
"index_options": {
"type": "hnsw"
}
}
}
}
}Both ElasticsearchOutputConnector and ElasticsearchStoreWriterConsumer map chunks through ElasticsearchDocumentMapper, protecting reserved system and security fields (RESERVED_FIELDS) so arbitrary source metadata cannot overwrite them:
{
"id": "doc-001_0",
"documentId": "doc-001",
"text": "Quarterly engineering roadmap and architecture milestones...",
"uri": "file:///kb/doc-001.txt",
"acl": "alice,engineering",
"lastModified": "2026-01-02T03:04:05Z",
"security_inheritance": true,
"security_allowed_read": ["user:alice", "group:engineering"],
"security_denied_read": ["user:mallory"],
"security_read_keys": ["u:alice", "g:engineering"],
"embeddings": [0.0123, -0.0456, 0.0891, ...],
"title": ["IT doc-001"],
"chunk_index": 0,
"total_chunks": 3
}When spring.opencrawling.output.type=elasticsearch and opencrawling.mcp.server.enabled=true (the default), oc-runtime activates ElasticsearchMcpSearchBackend:
-
In-Query kNN ACL Pre-Filtering: In
metadataandshadowauthorization modes (storePrefilterAllowed() == true),ElasticsearchAclQueries.readableBy(userPrincipal, userRoles)constructs atermsfilter onsecurity_read_keysmatching the caller's normalized read keys (u:<principal>,g:<role>, andpublic) and attaches it directly to the top-levelknnsearch clause (num_candidates = max(topK * 2, 100)). -
Vector Payload Exclusion: Search requests use
_sourcefiltering to exclude theembeddingsfloat array (NO_VECTORS), reducing network payload overhead when returning metadata and text chunks to MCP clients. -
Authoritative Access Gate (
DocumentAccessGate): Every candidate hit returned by Elasticsearch is passed throughDocumentAccessGate(readable(...)), which evaluates the configuredAuthorizationProvider(metadata,shadow,intersection, oropenfga) with strict fail-closed semantics and explicit deny precedence.
curl -s "http://localhost:9200/enterprise_kb/_count" | jq .curl -s "http://localhost:9200/enterprise_kb/_search" \
-H "Content-Type: application/json" \
-d '{
"query": {
"match": {
"text": "OpenCrawling"
}
},
"_source": {
"excludes": ["embeddings"]
}
}' | jq .curl -s "http://localhost:9200/enterprise_kb/_search" \
-H "Content-Type: application/json" \
-d '{
"size": 5,
"knn": {
"field": "embeddings",
"query_vector": [0.0123, -0.0456, 0.0891, ...],
"k": 5,
"num_candidates": 100,
"filter": {
"terms": {
"security_read_keys": ["public", "u:alice", "g:engineering"]
}
}
},
"_source": ["id", "documentId", "uri", "text", "acl", "lastModified"]
}' | jq .Run the standalone connector test script, which executes unit tests plus Testcontainers integration tests (Elasticsearch8OutputConnectorIT on Elasticsearch 8.19.20 and Elasticsearch9OutputConnectorIT on Elasticsearch 9.4.3, testing both compatible-with=8 and native compatible-with=9):
chmod +x scripts/test-elasticsearch-connector.sh
./scripts/test-elasticsearch-connector.shThe decoupled pipeline definition is located at oc-elasticsearch-output-connector/docker/docker-compose-decoupled-with-elasticsearch.yml and orchestrates:
-
Elasticsearch (
elasticsearch-standalone):docker.elastic.co/elasticsearch/elasticsearch:${ELASTICSEARCH_VERSION:-8.19.20}on port9200(configurable to9.4.3or any 8.x/9.x tag viaELASTICSEARCH_VERSION). -
Apache Kafka (
kafka-decoupled-elasticsearch): KRaft message broker on port9092. -
Redis Stack (
redis-stack-decoupled-elasticsearch): Distributed cache on ports6379/8001. -
Apache Ozone 2.2.0 (
ozone-scm,ozone-om,ozone-datanode,ozone-s3g): Claim Check binary object store. -
Ollama & Model Puller: Pulls
mxbai-embed-large(1024d),nomic-embed-text(768d), andall-minilm(384d). -
OpenCrawling Microservices:
oc-crawler,oc-ingestion-consumer,oc-embedding-consumer,oc-writer-consumer,oc-mcp-server(port8080), andoc-admin-ui(port3000).
# Start with default Elasticsearch 8.19.20
docker compose -f oc-elasticsearch-output-connector/docker/docker-compose-decoupled-with-elasticsearch.yml up -d
# Or start with Elasticsearch 9.4.3
ELASTICSEARCH_VERSION=9.4.3 docker compose -f oc-elasticsearch-output-connector/docker/docker-compose-decoupled-with-elasticsearch.yml up -dRun the full decoupled pipeline integration test:
chmod +x scripts/test-elasticsearch-decoupled.sh
./scripts/test-elasticsearch-decoupled.sh
# Or test against Elasticsearch 9.x:
ELASTICSEARCH_VERSION=9.4.3 ./scripts/test-elasticsearch-decoupled.shThe script verifies:
- Container image builds and startup of Elasticsearch, Ollama, Redis, Kafka, Apache Ozone, and all OpenCrawling microservices.
-
X-Elastic-Product: Elasticsearchheader verification onhttp://localhost:9200. - Automatic creation of the
enterprise_kbindex withdense_vector(dims: 1024) andsecurity_read_keys(keyword) mappings. - End-to-end filesystem crawl
$\rightarrow$ Kafka$\rightarrow$ Tika text extraction & chunking$\rightarrow$ Ollama 1024-dim embedding$\rightarrow$ ElasticsearchStoreWriterConsumerpersistence. - Verification that stored chunks carry 1024-dimensional embeddings and populated
security_read_keys. - Live kNN vector search with the
security_read_keystermspre-filter. -
oc-mcp-serverreachability onhttp://localhost:8080/mcpand confirmation thatElasticsearchMcpSearchBackendis active. - OIS
DELETEtombstone lifecycle verification viaElasticsearchStoreWriterConsumerTest.
- Home
- User & Installation Guide
- Quick Start
- VM & Bare-Metal Distribution Installation
- Local Development & Fast Docker Builds
- User Guide
- Architecture
- Zero-Trust Security & ACL Management
- Security and Identity Management
- Document Authorization & OpenFGA
- Enterprise Multi-Agent Integration
- Zero-Trust Data Provider Agent & Active Ingestion
- Document Lifecycle Actions & Tombstones
- Apache Tika 4 Text Extraction
- Configuration Guide
- Native Job Scheduler
- Platform API (
/api/v1) - Platform API: Traceability
- Connectors Reference
- Alfresco Content Services (ACS) Connector
- Alfresco Process Services (APS) Connector
- Flowable Repository Connector
- Camunda Repository Connector
- OASIS CMIS Repository Connector
- Doxis Repository Connector
- M-Files Repository Connector
- SharePoint Online Repository Connector
- Google Drive Repository Connector
- Atlassian Confluence Repository Connector
- IBM FileNet P8 Repository Connector
- OpenText Documentum Repository Connector
- Zammad Repository Connector
- Hyland OpenArch Integration (Repository, Output & MCP)
- Hyland Content Innovation Cloud (CIC) Connector
- Relational Database (JDBC) Connector
- Web Repository Connector
- Apache StormCrawler Repository Connector
- gRPC Push Repository Connector
- Elasticsearch Output Connector
- Apache Solr Output Connector
- Luxir Output Connector
- Apache SeaTunnel Output Connector
- Apache Ozone Output Connector (Migration Mode)
- AWS S3 Output Connector (Migration Mode)
- Alfresco Content Services Output Connector (Migration Mode)
- Apache Iceberg Output Connector
- Doxis Output Connector
- Connector Execution Modes
- Model Context Protocol
- Apache Ozone Claim Check Store
- Lakehouse Audit Reporting & Ledger
- AI-Powered Observability (AIOps)
- Auto-Narrativization Copilot
- Maven Connector Archetypes
- Java Client SDK
- Python Client SDK
- OpenCrawling CLI
- OpenAPI Specification & Swagger UI
- gRPC Internal Transport
- Migration Guide (ManifoldCF)
- Integration Test — Decoupled Pipeline
- M-Files Integration Test
- SharePoint Online Integration Test
- Google Drive Integration Test
- Confluence Integration Test
- IBM FileNet Integration Test
- OpenText Documentum Integration Test
- Zammad Integration Test
- Hyland OpenArch Integration Test
- Hyland CIC Integration Test
- Web Crawler Test
- Elasticsearch Integration Test
- Local Docker Build Test
- Root CLI Runner Scripts Test
- Secure MCP Server Integration Test
- Narrativization Integration Test
- Reporting Integration Test
- AWS S3 Migration Integration Test
- Alfresco Migration Integration Test
- OpenFGA Authorization Integration Test
- CLI Decoupled Integration Test
- Java Client SDK Decoupled Test
- Apache Iceberg Decoupled Test
- Doxis Output Integration Test
- Platform API v1 Test Suite