Luxir Output Connector

Luxir Output Connector

Module: oc-luxir-output-connector
Class: LuxirOutputConnector
Spring Bean: @Component (activated when spring.opencrawling.output.type=luxir)

---

📌 Overview

The Luxir Output Connector integrates Luxir into OpenCrawling as a high-performance, lightweight search engine and dense vector store. Luxir provides full-text search, dense vector kNN search, hybrid search using Reciprocal Rank Fusion (RRF), faceted filtering, dynamic typing schemas, and native JSON REST APIs over HTTP.

graph TD
    A["Crawled Document Chunk"] --> B["oc-embedding-consumer (Spring AI / Ollama)"]
    B --> C["Kafka Topic (opencrawling-embedded)"]
    C --> D["LuxirStoreWriterConsumer (oc-luxir-output-connector)"]
    D --> E["LuxirClient (HTTP/1.1 / JSON API)"]
    E --> F["Luxir Collection ('opencrawling')"]
    F --> G["Dense Vector Indexing ('embedding_v')"]

Loading

---

🚀 Key Features

• Native HTTP REST Integration: Lightweight, high-throughput integration using Java 25 HttpClient communicating with Luxir's native HTTP API on port 9400.
• Dense Vector & Dynamic Schema: Supports vector embeddings with customizable dimensionality (default 1024, matching mxbai-embed-large) and distance metrics (cosine, l2, ip), plus Luxir dynamic field suffix conventions (_t, _s, _ss, _v, _vs, _i).
• Automatic Collection & Schema Provisioning: LuxirCollectionInitializer automatically verifies collection existence on startup and initializes the target collection with explicit vector field schemas via POST /collections/{collection}/_schema.
• OIS v1.1 Document Lifecycle & Tombstone Deletes: Supports document removal and tombstone purge operations via Luxir's native delete_ids batch endpoint.
• Zero-Trust Security ACLs: Indexes Open Ingestion Standard (OIS) security tokens into multi-valued string array fields (acl_ss, security_allowed_read_ss, security_denied_read_ss) and inheritance flags (security_inheritance_s: "true"/"false").
• Admin UI & Health Checker Integration: Fully integrated into OpenCrawling Admin UI (ConnectorForm.tsx, JobTable.tsx) and verified dynamically by ConnectorCheckerService via GET /collections/{collection}.
• Decoupled Architecture: Features LuxirStoreWriterConsumer consuming from the Kafka opencrawling-embedded topic for horizontal scalability.

---

⚙️ Configuration Properties

All properties are prefixed with spring.opencrawling.output.luxir.*:

Property	Default	Description
spring.opencrawling.output.luxir.endpoint	http://localhost:9400	HTTP endpoint URL of the Luxir instance
spring.opencrawling.output.luxir.collection	opencrawling	Target Luxir collection name
spring.opencrawling.output.luxir.vector-field	embedding_v	Vector field name in Luxir (_v suffix convention)
spring.opencrawling.output.luxir.dimensions	1024	Vector dimensionality (matches default mxbai-embed-large)
spring.opencrawling.output.luxir.similarity	cosine	Distance metric: cosine, l2, or ip
spring.opencrawling.output.luxir.auto-commit	true	Whether to automatically commit index updates
spring.opencrawling.output.luxir.commit-within-ms	1000	Soft-commit window in milliseconds
spring.opencrawling.output.luxir.timeout-seconds	30	HTTP connection and socket timeout in seconds

---

📋 Schema & Document Representation

Vector Field Schema Provisioning

When initializing a collection, LuxirCollectionInitializer configures the vector field schema via:

POST /collections/opencrawling/_schema
{
  "fields": {
    "embedding_v": {
      "type": "vector",
      "dims": 1024,
      "metric": "cosine"
    },
    "passages_vs": {
      "type": "vector",
      "dims": 1024,
      "metric": "cosine",
      "multi": true
    }
  }
}

Document Indexing Payload

Documents and chunks are indexed via POST /collections/{collection}/_update:

{
  "docs": [
    {
      "id": "doc-001_chunk_0",
      "doc_id_s": "doc-001",
      "chunk_index_i": 0,
      "title_t": "Sample Knowledge Base Article",
      "text_t": "Extracted document content...",
      "uri_s": "https://example.org/docs/sample.pdf",
      "embedding_v": [0.0123, -0.0456, 0.0891, ...],
      "acl_ss": ["ROLE_USER", "group:engineering"],
      "security_allowed_read_ss": ["ROLE_USER", "group:engineering"],
      "security_denied_read_ss": [],
      "security_inheritance_s": "true"
    }
  ],
  "commit": {}
}

OIS v1.1 Tombstone Delete Payload

Tombstones (action: "DELETE") are processed using delete_ids:

{
  "delete_ids": ["doc-001_chunk_0"],
  "commit": {}
}

---

🔍 Executing Queries against Luxir

1. Document Count Query

Query total document count in the collection using the get_number flag:

curl -X POST "http://localhost:9400/collections/opencrawling/_search" \
     -H "Content-Type: application/json" \
     -d '{"query": "*:*", "get_number": true}'

2. Full-Text Search

Search using dynamic text fields (_t):

curl -X POST "http://localhost:9400/collections/opencrawling/_search" \
     -H "Content-Type: application/json" \
     -d '{"query": "text_t:OpenCrawling"}'

3. Dense Vector kNN Search

Execute dense vector similarity queries using the knn search clause:

curl -X POST "http://localhost:9400/collections/opencrawling/_search" \
     -H "Content-Type: application/json" \
     -d '{
       "query": {
         "knn": {
           "field": "embedding_v",
           "query": [0.01, 0.02, ...],
           "k": 5
         }
       }
     }'

---

🐳 Docker & Integration Test

Standalone Luxir Docker Compose

Spin up Luxir using the standalone Compose definition:

docker compose -f oc-luxir-output-connector/docker/docker-compose.yml up -d

Decoupled Multi-Service Deployment

The decoupled architecture definition is located at oc-luxir-output-connector/docker/docker-compose-decoupled-with-luxir.yml and includes:

• Luxir Search Engine: luxir/luxir:v0.1.0 (with platform: linux/amd64 emulation for ARM64 / Apple Silicon).
• Kafka & Zookeeper: Distributed messaging backbone.
• Redis: Distributed deduplication and Claim Check cache.
• Ollama & Ollama Model Puller: Automatically pulls mxbai-embed-large (1024 dims).
• OpenCrawling Microservices: oc-crawler, oc-ingestion-consumer, oc-embedding-consumer, oc-writer-consumer, and oc-mcp-server.

Automated Decoupled Integration Test Script

Run the automated end-to-end multi-service pipeline integration test:

./scripts/test-luxir-decoupled.sh

The script features detailed live logging, including:

1. Docker Compose stack startup and status verification (compose ps).
2. Health-check readiness counters for Luxir (:9400) and Ollama (:11434).
3. Init container tracking for the mxbai-embed-large embedding model.
4. Crawler execution tracking, exit code validation (System.exit(0)), and crawler scan log output (compose logs oc-crawler).
5. Live indexing progress polling against Luxir and writer consumer log output (compose logs oc-writer-consumer).
6. Full-text search and 1024-dimensional dense vector kNN query result verification.
7. MCP Server HTTP endpoint accessibility check on port 8080.
8. OIS Document Lifecycle Tombstone DELETE action integration testing via LuxirStoreWriterConsumerTest.
9. Clean environment teardown.