High-Throughput Concurrent Java Search Engine

A search engine built from scratch in pure Java 21 — no HashMap, no ArrayList, no String allocation during indexing. Every subsystem is designed around CPU cache locality, zero-GC ingestion, and lock-free concurrent reads.

Architecture

[Raw Files on Disk]
       │
       ▼  Zero-copy mmap (FileChannel → MappedByteBuffer)
[MemoryMappedIngestor]   Streams raw bytes via the OS page cache
       │
       ▼  Zero-allocation bitwise parsing
[TokenHasher]            Rolling 64-bit FNV-1a hash — tokens never become Strings
       │
       ▼  O(1) primitive-array storage
[HighPerformanceIndex]   Open-addressed hash table (long[] keys, int[] offsets)
       │                 + block-based postings arena (packed int[])
       ▼  StampedLock optimistic reads
[ConcurrentQueryEngine]  Multi-threaded Okapi BM25 ranking

Key Design Decisions

Problem	Standard Java	This Engine
Dictionary	HashMap<String, List> — 16B object headers, pointer chasing	Parallel long[]/int[] arrays, linear probing, cache-line packed
Tokenizing	String.split() — floods Young Gen, triggers GC	Rolling FNV-1a hash over raw bytes, zero allocations
File I/O	BufferedReader — kernel→user-space copies	mmap via MappedByteBuffer, OS page cache serves bytes directly
Postings	ArrayList<Posting> objects	Packed [capacity, count, docId, tf, ...] blocks in one int[] arena, doubling relocation on overflow
Read concurrency	synchronized / read locks block	StampedLock optimistic reads — lock-free unless a write races, then graceful fallback

Components

• HighPerformanceIndex — inverted index. Open addressing + linear probing, MurmurHash3 finalizer for slot spreading, amortized O(1) block growth for postings.
• TokenHasher — shared FNV-1a hashing so ingestion (bytes) and queries (strings) produce identical token hashes.
• MemoryMappedIngestor — zero-copy, zero-allocation document ingestion; returns per-document token counts for BM25 length normalization.
• ConcurrentQueryEngine — fixed thread pool sized to physical cores; BM25 scoring (k1=1.2, b=0.75); top-K result extraction.
• SearchEngineHarness — end-to-end demo: writes a corpus to disk, ingests it, runs ranked queries, then holds a steady-state loop for VisualVM/JConsole heap profiling.
• StressTest — validates table resizing (200K unique tokens), postings block relocation (1 token × 5,000 docs), and read consistency under concurrent writes.

Build & Run

javac *.java

# Demo with ranked query output
java SearchEngineHarness

# Correctness stress test
java StressTest

# With GC telemetry for profiling
java -XX:+UseG1GC -Xlog:gc -Xms512m -Xmx2g engine.SearchEngineHarness

Profiling with VisualVM

1. Run the harness with the GC flags above.
2. Attach VisualVM → Monitor tab during step [4/4].
3. Expected: a flat heap plateau (not a sawtooth) — ingestion allocates no temporary Strings, so Young Gen stays clean and GC pauses stay near 0 ms.

Complexity

Operation	Complexity
Token lookup	O(1) average (open addressing)
Posting insert	O(1) amortized (block doubling)
Query scoring	O(q × d) — q query terms × d matching docs
Table resize	O(n) rehash, amortized O(1) per insert