Reduced write amplification with faster data visibility

[bluestreak01]

Summary

Dramatically reduce write amplification and improve data visibility by enabling append-only writes with immediate query access to uncommitted data. The query engine reads WAL files directly before compaction, making new data visible in milliseconds rather than waiting for merge operations.

Current Limitation

[Write Path]                         [Query]
     │                                  │
  Sort on write ──────────────────►  Wait for merge
     │                                  │
  High write amplification           Delayed visibility

With This Change

[Write Path]                         [Query]
     │                                  │
  Append only (fast) ──────────────► Read WAL directly
     │                                  │
  Minimal amplification              Instant visibility
     │
  Background compaction (async)

Benefits

Metric	Improvement
Write amplification	Significantly reduced — append-only on hot path
Data visibility	Milliseconds instead of seconds — query reads WAL directly
Write throughput	Higher — no sorting on critical path
Resource efficiency	Compaction happens in background, doesn't block writes or reads

How It Works

• Query engine handles unordered data via predicate pushdown
• Writes append to WAL without immediate sorting
• Queries read both compacted partitions AND uncommitted WAL segments
• Background compaction merges WAL into sorted partitions when convenient

Use Cases

• High-frequency ingestion requiring immediate queryability
• Real-time dashboards showing latest data
• Trading systems where milliseconds of latency matter
• Any workload where write speed and data freshness are critical