Apache Solr Output Connector

Apache Solr Output Connector

Module: oc-solr-output-connector
Class: SolrOutputConnector
Spring Bean: @Component (activated when spring.opencrawling.output.type=solr)

---

📌 Overview

The Apache Solr Output Connector integrates Apache Solr (v10.x and v9.x) into the OpenCrawling pipeline as a vector search engine and hybrid search store. It delivers high-throughput vector chunk indexing, HNSW ANN search, Solr 10 quantization, dynamic REST schema auto-provisioning, zero-trust ACL security metadata, and decoupled Kafka writer execution.

graph TD
    A["Crawled Document Chunk"] --> B["oc-embedding-consumer (Spring AI / Ollama)"]
    B --> C["Kafka Topic (opencrawling-embedded)"]
    C --> D["SolrStoreWriterConsumer (oc-solr-output-connector)"]
    D --> E["SolrJ 10 HttpJdkSolrClient / CloudSolrClient"]
    E --> F["Apache Solr Collection ('enterprise_kb')"]
    F --> G["solr.DenseVectorField (HNSW Indexing)"]

Loading

---

🚀 Key Features

• Solr 10 & 9.x Support: Native integration using SolrJ 10 JDK HTTP Client (HttpJdkSolrClient for standalone, CloudSolrClient for SolrCloud).
• solr.DenseVectorField & HNSW Indexing: Auto-provisions the knn_vector field type backed by solr.DenseVectorField with configurable HNSW graph parameters (hnswMaxConnections, hnswBeamWidth, similarityFunction).
• Solr 10 Advanced Vector Features:
  • Quantization: Supports none (Full Precision), scalar (Scalar Quantization SQ), and binary (Binary Quantization BQ).
  • Vector Encoding: Supports FLOAT32 or BYTE encodings.
  • efSearch Tuning: Precision/recall tuning parameter for Solr 10 KNN search queries.
• Automatic REST Schema Auto-Provisioning: SolrCollectionInitializer programmatically creates the required knn_vector field type and missing schema fields (embeddings, text, uri, acl, last_modified, security_allowed_read, security_denied_read) via Solr's Schema REST API.
• Decoupled Architecture: Features SolrStoreWriterConsumer for listening to the Kafka opencrawling-embedded topic and asynchronously indexing vector chunks into Solr.
• Zero-Trust Security: Maps Open Ingestion Standard (OIS) ACL permission rules into Solr fields (security_allowed_read, security_denied_read, security_inheritance_enabled).

---

⚙️ Configuration Properties

All properties are prefixed with spring.opencrawling.output.solr.*:

Property	Default	Description
spring.opencrawling.output.solr.mode	standalone	Solr deployment mode: standalone or cloud
spring.opencrawling.output.solr.url	http://localhost:8983/solr	Base HTTP URL of standalone Solr instance
spring.opencrawling.output.solr.zk-host	localhost:2181	ZooKeeper connection string for SolrCloud mode
spring.opencrawling.output.solr.collection	enterprise_kb	Target Solr collection / core name
spring.opencrawling.output.solr.dimensions	1024	Vector embedding dimension size
spring.opencrawling.output.solr.similarity	cosine	Similarity metric: cosine, dot_product, or euclidean
spring.opencrawling.output.solr.vector-encoding	FLOAT32	Solr 10 vector encoding: FLOAT32 or BYTE
spring.opencrawling.output.solr.quantization	none	Solr 10 quantization: none, scalar, or binary
spring.opencrawling.output.solr.hnsw-max-connections	16	HNSW graph max connections ($M$)
spring.opencrawling.output.solr.hnsw-beam-width	100	HNSW graph beam width ($efConstruction$)
spring.opencrawling.output.solr.ef-search	100	Solr 10 efSearch query precision parameter
spring.opencrawling.output.solr.commit-within-ms	1000	Soft-commit latency limit in milliseconds

---

📋 Auto-Provisioned Schema Definition

When initialized, SolrCollectionInitializer sends the following schema payload to Solr:

{
  "add-field-type": {
    "name": "knn_vector",
    "class": "solr.DenseVectorField",
    "vectorDimension": 1024,
    "similarityFunction": "cosine",
    "knnAlgorithm": "hnsw",
    "hnswMaxConnections": 16,
    "hnswBeamWidth": 100,
    "vectorEncoding": "FLOAT32"
  },
  "add-field": [
    { "name": "id", "type": "string", "stored": true, "indexed": true },
    { "name": "text", "type": "text_general", "stored": true, "indexed": true },
    { "name": "embeddings", "type": "knn_vector", "stored": true, "indexed": true },
    { "name": "uri", "type": "string", "stored": true, "indexed": true },
    { "name": "acl", "type": "string", "stored": true, "indexed": true },
    { "name": "security_allowed_read", "type": "string", "stored": true, "indexed": true, "multiValued": true },
    { "name": "security_denied_read", "type": "string", "stored": true, "indexed": true, "multiValued": true }
  ]
}

---

🔍 Executing Queries against Solr

1. Full-Text Search

curl "http://localhost:8983/solr/enterprise_kb/select?q=text:OpenCrawling&fl=id,text,uri,score"

2. Solr 10 Dense Vector Search (KNN Parser)

curl --data-urlencode "q={!knn f=embeddings topK=5}[0.0123,-0.0456,0.0891,...]" \
     --data-urlencode "fl=id,text,uri,score" \
     "http://localhost:8983/solr/enterprise_kb/select"

---

🐳 Docker & Integration Test

Standalone Solr Docker Compose

Spin up Apache Solr using the provided Compose file:

docker compose -f oc-solr-output-connector/docker/docker-compose.yml up -d

Automated Integration Test Script

Run the automated end-to-end multi-service pipeline integration test:

./scripts/test-solr-decoupled.sh