Logix

High-performance, strictly-typed log collection, validation, and routing platform. Prevents invalid, unstructured data from reaching Elasticsearch by enforcing JSON Schema contracts at the entry point.

---

Architecture

                      Business Applications
                             │
                             │  HTTP (JSON)
                             ▼
               ┌──────────────────────────┐
               │      lgx-ingestor        │
               │   Edge Gateway           │
               │   • JSON Schema validate │
               │   • Route valid → Kafka  │
               │   • Route invalid → DLQ  │
               │   • Backpressure (503)   │
               │   • WAL fallback         │
               └──────────┬───────────────┘
                          │
                    valid │  invalid
               ┌──────────┴──────────┐
               ▼                     ▼
        ┌────────────┐        ┌────────────┐
        │  Kafka     │        │  Kafka     │
        │  valid     │        │  DLQ       │
        │  topic     │        │  topic     │
        └─────┬──────┘        └─────┬──────┘
              │                     │
              ▼                     │
   ┌──────────────────────┐         │
   │     lgx-sink         │         │
   │  Background Worker   │         │
   │  • Poll + batch      │         │
   │  • Bulk index to ES  │         │
   │  • Retry + DLQ       │         │
   └──────────┬───────────┘         │
              │                     │
              ▼                     ▼
   ┌──────────────────┐   ┌──────────────────┐
   │  Elasticsearch   │   │  lgx-dashboard   │
   │  (logs storage)  │   │  (control plane) │
   └──────────────────┘   └──────────────────┘

Components

Service	Role	Port	Docs
lgx-ingestor	Edge gateway — accepts logs via HTTP, validates against JSON Schema, routes to Kafka	8080	INGESTOR.md
lgx-sink	Background worker — consumes from Kafka, batches and indexes to Elasticsearch	8081	SINK.md
lgx-dashboard	Control plane UI — schema viewer, DLQ inspector, health & metrics	(planned)	—

Infrastructure (external dependencies)

Component	Role
Kafka	Message broker, shock absorber between Ingestor and Sink. Topics: lgx.logs.valid, lgx.logs.retry, lgx.logs.dlq
Elasticsearch	Search engine, long-term log storage. Time-based indexes: lgx-YYYY.MM.dd

Documentation Map

File	What's inside
docs/INGESTOR.md	API endpoints, pipeline, validation, WAL, delivery guarantees, metrics, config reference
docs/SINK.md	Consumer architecture, retry/DLQ flow, idempotent indexing, metrics, config reference
config/lgx.yaml	Unified configuration file — single source of truth for all services
deployments/docker-compose/	Quick-start sandbox — Kafka, Zookeeper, Elasticsearch, Ingestor, Sink

Tech Stack

Layer	Technology
Framework	Quarkus 3.37.3 (Standard JVM)
Language	Java 25
Build	Maven (Maven Wrapper)
Messaging	Apache Kafka (native kafka-clients 4.0.0)
Validation	JSON Schema Draft 2020-12 (networknt/json-schema-validator)
Storage	Elasticsearch 8.17.x
Web	RESTEasy Reactive, Jackson, OpenAPI
Observability	Micrometer + Prometheus, SmallRye Health
Deployment	Docker, Docker Compose, Kubernetes (Helm)

Recommended Memory Limits

Service	Min Memory	Recommended	Max Observed
lgx-ingestor	128 MB	256 MB	252 MB @ 23K logs/s
lgx-sink	256 MB	512 MB	345 MB @ 23K logs/s
Kafka	512 MB	1 GB	—
Elasticsearch	512 MB heap	1 GB heap	—

Throughput (single instance, batch mode)

Logs/s	Ingestor Status	Errors	Notes
1,000	stable	0	~10 req/s, single-threaded
5,000	stable	0	~100 req/s, single-threaded
10,000	stable	0	~200 req/s
23,000	peak	0	20 parallel workers, 98% mem, 0 backpressure

Key findings:

• Ingestor throughput is bottlenecked by network I/O, not CPU or memory. At 23K logs/s with batch mode (50 logs/req), 256MB is sufficient with ~4% headroom.
• No data loss — all 2.6M ingested logs were delivered to Elasticsearch.
• No duplicates — idempotent Kafka producer + _id-based ES indexing.
• No backpressure — max-pending-sends=10000 was never reached in testing.
• For higher throughput, scale horizontally — add more ingestor instances behind a load balancer. Kafka acts as the buffer.

Scaling Guidance

Throughput Target	Ingestor Instances	Ingestor Memory (each)	Sink Instances
< 20K logs/s	1	256 MB	1
20K – 100K logs/s	2–3	256 MB	1–2
100K – 500K logs/s	5–10	256 MB	3–5
500K+ logs/s	10+	512 MB	5+

---

Quick Start

# 1. Start all services
docker compose -f deployments/docker-compose/docker-compose.yml up -d

# 2. Check health
curl http://localhost:8080/q/health     # ingestor
curl http://localhost:8081/q/health     # sink

# 3. Send a test log
curl -X POST http://localhost:8080/api/v1/logs \
  -H "Content-Type: application/json" \
  -d '{"@timestamp":"2026-07-24T10:00:00Z","level":"INFO","message":"Hello Logix","service":"demo"}'

# 4. Query Elasticsearch
curl "http://localhost:9200/lgx-$(date -u +%Y.%m.%d)/_search?pretty"

Project Structure

lgx-app/
├── lgx-ingestor/               # Quarkus — HTTP → Kafka (stateless edge)
├── lgx-sink/                   # Quarkus — Kafka → Elasticsearch (worker)
├── lgx-dashboard/              # Control plane UI (planned)
├── config/
│   └── lgx.yaml                # Unified configuration
├── deployments/
│   ├── docker-compose/         # Local sandbox
│   └── helm-chart/             # Production K8s
├── docs/
│   ├── INGESTOR.md
│   └── SINK.md
└── README.md