perf(mapper): optimize Rust NIF hot paths

[djwhitt]

Summary

Optimize the Rust-backed mapper's existing single-document execution path without adding batching or changing the ClickHouse pipeline, mapping configuration, or public API.

The main strategy is to move reusable decisions into mapping compilation, share path work within each mapped document, and reduce intermediate BEAM terms and Rust heap allocations. The compiled mapping resource remains immutable; every map/3 invocation owns its own temporary cache, so the same compiled resource can be reused concurrently.

Scope and compatibility

This PR does not add batch mapping, change pipeline concurrency, or change the Elixir/NIF interface. It preserves the existing behavior for:

• defaults, coalesced paths, filters, transforms, allowed values, value maps, picks, and Enum8 inference
• nested-path and flat_keys: true lookup modes
• wildcard-array coercion and filter_nil behavior
• FlatMap exclusion/elevation rules and collision precedence
• scalar coercion boundaries and timestamp precision detection
• compound JSON serialization, including deterministic object-key ordering

Optimization details

1. Compile path metadata once

Each path is still parsed into key, index, and wildcard segments, but compilation now also records:

• the first wildcard position, allowing array fields with a single path to select a specialized execution path
• the dot-joined literal key used by flat_keys: true, avoiding rebuilding it per document
• cache slots for path prefixes referenced more than once
• cache slots used by ordinary fields, coalesced paths, JSON picks, and Enum8 inference conditions

The compiler counts shared prefixes across all path consumers and assigns compact integer cache indices. Runtime lookup therefore uses a pre-sized Vec<Option<Term>> instead of a per-document hash map. Missing and nil values are cached as well, preventing repeated failed traversals.

2. Adaptive root-map preloading

Production mappings often read many keys from a relatively small root map. When a compiled mapping contains at least eight root-key references, it records the referenced keys and can populate their cache slots in one map scan.

At runtime, preloading is only used when the document's root map is no larger than the compiled scan limit. Large or sparse roots keep direct map_get lookup, avoiding a full scan when only a small fraction of keys are needed. Flat-key mode remains isolated from nested root preloading.

The query cache is created inside map_single/4; no cached BEAM terms are stored in the reusable compiled resource. This is what keeps sequential documents and concurrent callers isolated.

3. Iterative path traversal and fused wildcard arrays

The common key/index path is evaluated iteratively. Wildcard fan-out is kept in a separate path so ordinary lookups do not pay for wildcard handling.

For array fields sourced from a single wildcard path, lookup and element coercion are combined in one pass. The mapper now walks the source list, resolves the suffix, applies the target array coercion, handles filter_nil, and builds the final BEAM list directly. This removes the previous intermediate wildcard-result list followed by a second coercion pass.

The specialization covers string, UInt64, Float64, DateTime64, JSON, map, and FlatMap arrays while preserving empty-list behavior for missing, empty, nil, or non-list wildcard prefixes.

4. Bulk map construction and fused FlatMap work

Output fields, JSON picks, exclusions, and flattened entries are collected into key/value arrays and emitted with enif_make_map_from_arrays through Rustler. This replaces repeated persistent map_put rebuilding where duplicate-key semantics do not require it.

FlatMap processing also avoids unnecessary intermediate maps:

• recursive flattening reuses one prefix buffer and truncates it while unwinding rather than constructing a new path string at each level
• existing valid binary values are reused instead of copied
• the common zero/one-elevation case combines exclusion, elevation, flattening, stringification, and output collection in one traversal
• multiple elevations use an ordered merge path so the first configured elevation wins over later elevations, while literal top-level keys retain highest precedence
• nested and already-flat inputs have separate implementations, so flat-key exclusion/elevation works on exact keys and dot-prefixes without rebuilding nested maps

The optimized path explicitly preserves non-map elevation parents, removes empty map parents, honors excluded elevation subtrees, and keeps literal dotted top-level keys ahead of nested elevated values.

5. Lower-allocation coercion and transforms

Several scalar hot paths now operate directly on borrowed BEAM binary slices:

• UInt64, Int32, Float64, boolean, and timestamp parsing no longer decodes through temporary Rust String values
• case-insensitive value-map and Enum8 lookup uses a stack buffer for typical ASCII values and falls back to Unicode allocation only when needed
• unchanged ASCII case transforms return the original BEAM binary; changed ASCII output is written directly into a NewBinary; Unicode retains the existing Unicode-aware fallback
• output strings are written directly into BEAM binaries, and integer-to-string conversion uses an itoa stack buffer
• timestamp precision uses numeric ranges and scaling uses saturating multiplication at overflow boundaries

Invalid UTF-8, invalid numeric text, booleans, Unicode case mapping, integer limits, and DateTime64 fallback behavior remain unchanged.

6. Stream compound JSON directly from BEAM terms

Compound list/map values previously required an intermediate serde_json::Value tree before conversion to a JSON string. JsonTerm now implements Serde's Serialize trait over the original BEAM term and streams scalars, lists, and maps directly into serde_json.

Map entries with valid UTF-8 keys are collected and sorted by key bytes before serialization. That small entry list preserves the legacy deterministic JSON representation—including byte-level key order—without allocating a second tree containing every compound value. Invalid UTF-8 values and non-finite floats retain the prior null representation.

Performance

Single-threaded local benchmarks on Linux ARM64, Elixir 1.19.5 / OTP 27:

Production log mapping	Before	After
Throughput	20.7–22.7K ops/s	81.7–85.6K ops/s
Average	44.0–48.4 µs	11.7–12.3 µs
Median	36.7–40.8 µs	11.4–11.9 µs
Reductions	787	122

Additional production-shaped workloads:

Workload	Before	After
Log	83.1K ops/s	261–269K ops/s
Trace arrays	53.4K ops/s	141–146K ops/s
Metric arrays	40.4K ops/s	129–133K ops/s
Flat keys	574.6K ops/s	1.22M ops/s
Compound JSON lists	7.16K ops/s	11.65K ops/s

Correctness and testing strategy

Focused hot-path regressions

optimized_mapper_test.exs exercises the optimization-specific invariants:

• Cache lifetime and shape: 120 documents mapped sequentially and concurrently through one compiled resource; repeated/missing/nil prefixes; coalescing; small preloaded roots versus 200-key roots; cached JSON picks and Enum8 inference; nested and flat-key lookup isolation.
• Wildcard fusion: every supported array family with and without filter_nil; missing, empty, nil, and non-list prefixes; root, indexed, nested, post-index, and repeated wildcard shapes.
• FlatMap fusion: exclusion/elevation interactions; top-level and literal dotted collisions; non-map and empty parents; multiple elevation precedence; overlapping flat prefixes; nested and flat_keys: true inputs.
• Coercion: unchanged/changed ASCII, Unicode, and invalid UTF-8 transforms; valid, invalid, and boundary numeric/boolean binaries; DateTime64 precision boundaries and saturating scale behavior.
• Compound JSON: mixed nested values, quoting, atoms, invalid UTF-8, and deterministic ordering for large 80-key maps in both nested and flat-key inputs.

Independent equivalence checks

Correctness is checked at several levels rather than only asserting selected examples:

1. A StreamData property test runs 100 generated production-shaped documents against a simple Elixir reference mapper.
2. A committed deterministic corpus runs 1,000 documents against the same independent reference model, covering log/metric/trace-like path reuse, arrays, FlatMaps, coercion, defaults, and malformed values.
3. A separate 1,000-document before/after differential corpus compares the optimized mapper with the baseline across log, metric, and trace configurations; outputs are byte-identical.
4. Existing mapper, mapping-config, ClickHouse adaptor/default, ingester, and ingest-queue tests verify behavior through the surrounding consumers rather than only at the NIF boundary.

Native and CI coverage

• 15 Rust unit tests for path parsing/compiled metadata and timestamp precision/scaling helpers
• dedicated cargo test -p mapper_ex CI job so native tests cannot silently fall outside the Elixir suite
• 385 focused mapper/config/ClickHouse consumer tests plus 1 property test
• retained deterministic 1,000-document reference-model corpus
• deterministic 1,000-document differential corpus is byte-identical to the baseline
• cargo fmt --all -- --check
• cargo clippy -p mapper_ex --all-targets -- -D warnings
• MIX_ENV=test ../bin/x mix compile --warnings-as-errors
• MIX_ENV=test ../bin/x mix format --check-formatted
• MIX_ENV=test ../bin/x mix lint.all
• all actionable PR checks passing; only the expected WIP check remains while the PR is a draft