Reduce per-field hashing and iteration on the composite/Parquet indexing hot path

[Bukhtawar]

Description

Removes per-field hashing, iteration, redundant lookup, and resize-allocation overhead from the composite/Parquet indexing hot path, motivated by an async-profiler CPU capture of bulk ingest where JVM_IHashCode was the single largest self-cost (~10.8%), and a follow-up allocation profile (-e alloc) where the per-document dedup set's HashMap$Node/Node[] churn was ~9.7% of total allocation.

1. ParquetDocumentInput: dedup fields by name instead of object identity

addField detected duplicate single-valued fields with a Collections.newSetFromMap(new IdentityHashMap<>()), so every field of every document paid System.identityHashCode (MappedFieldType inherits identity hashCode()). That call is the JVM_IHashCode frame in the profile.

The mapping layer forbids two mappers sharing a name (MappingLookup), and every addField caller passes fieldType(), so within one document same name ⇒ same MappedFieldType instance. Deduping in a HashSet<String> on fieldType.name() is therefore behaviourally equivalent, uses String's cached hash (no identity-hash intrinsic), and stays O(n) — no quadratic scan on wide documents.

2. CompositeDocumentInput: snapshot secondary inputs to arrays

addField iterated unmodifiableMap(IdentityHashMap).entrySet() per field per document, allocating an entry-set iterator and walking IdentityHashMap's sparse table on every call (IdentityHashMapIterator.hasNext / UnmodifiableEntrySet frames). The map is immutable after construction, so the constructor now snapshots it once into flat DataFormat[]/DocumentInput[] arrays and addField/setRowId loop those.

3. ParquetField SPI: resolve the Arrow vector once per field

VSRManager.addDocument resolved each field's vector by name twice per field — once for a null-check, then again inside every ParquetField.addToGroup implementation (ManagedVSR.getVector → HashMap.get). The SPI now passes the resolved FieldVector and row index into addToGroup; VSRManager resolves once, null-checks, and hands it through. A name-resolving createField overload is kept for callers that don't pre-resolve.

4. Pre-size the per-document collections from the mapped-field count

The allocation profile showed the dedup set and field list growing incrementally per document (HashMap$Node[] table copies 16 → 256 for a ~100-field document, plus ArrayList growth). ParquetIndexingEngine.newDocumentInput now sizes both from the Arrow schema's field count, cached against the mapping version so the schema walk only reruns on mapping changes.

Benchmarks

New jmh source sets in both plugins (same wiring as :libs:opensearch-concurrent-queue); all drive the real classes over genuine NumberFieldType/KeywordFieldType instances.

• Dedup (1): name-keyed set competitive at small field counts, clearly better at wide documents, no O(n²) cliff. An async-profiler run confirms the IdentityHashMap.hash frame is present in the old strategy and absent with the name-keyed set.
• Broadcast (2): with no-op per-format delegates (isolating broadcast overhead), array snapshot is ~10–13× map iteration across 5–50 fields × 1–3 secondaries (e.g. 20×1: 19.7 ns vs 249.5 ns).
• Vector lookup (3): single lookup is ~2× the double lookup at 50 fields (98.8 vs 186.8 ns). An ordinal-array layout measured ~10× further headroom — left as follow-up if Linux re-profiling still shows this frame.
• Pre-sizing (4), -prof gc at 200 fields: 18 649 → 14 201 B/op (−24%) and 2 689 → 1 908 ns/op vs default-sized collections. Remaining bytes are the per-field HashMap$Node/FieldValuePair entries themselves; eliminating those needs input pooling or a fused structure — measured trade-off, deferred.

Run: ./gradlew -Dsandbox.enabled=true :sandbox:plugins:parquet-data-format:jmh / :sandbox:plugins:composite-engine:jmh with -Pjmh.includes=<name>.

Honesty / scope notes for reviewers

• The microbenchmarks prove the mechanisms and isolated deltas; they cannot quantify fleet-level % — that needs re-profiling real bulk ingest on Linux. Isolated multiples shrink end-to-end since real per-format work sits on top.
• Allocation trade-off, stated plainly: the name-keyed HashSet allocates more per document than the old IdentityHashMap at small field counts (-prof gc @20 fields: 1 480 vs 592 B/op) because HashMap allocates a Node per entry while IdentityHashMap is a flat table. The old path's cost was CPU (identity-hash intrinsic + VM transition frames), the new path's cost is short-lived young-gen garbage. The production GC data so far (0 old-gen collections, 12.1 s young-gen over a 51-min ingest) says this is not pathological, but the Linux before/after should watch alloc-rate, not just CPU.
• The Lucene-internal KeywordFieldMapper.normalizeValue → AttributeSource.addAttribute JVM_IHashCode source and the Jackson DupDetector allocation hotspot (~8.4% of alloc, duplicate-JSON-key detection in parse) are not touched here.

Check List

• Functionality equivalent (name-dedup ≡ identity-dedup; broadcast map immutable after construction; row index captured before row count advances; schema cache keyed on mapping version)
• Commits are signed per the DCO
• Plugin unit tests pass (:sandbox:plugins:parquet-data-format:test)
• Reviewer to confirm production before/after profile (CPU and alloc) on Linux ingest host

[github-actions]

PR Code Analyzer ❗

AI-powered 'Code-Diff-Analyzer' found issues on commit 21485c5.

⛔ Hard block: Issues at Medium severity or above will block this PR from merging.

Path	Line	Severity	Description
sandbox/plugins/composite-engine/build.gradle	82	high	New dependency added: 'org.openjdk.jmh:jmh-core' under jmhImplementation configuration. Dependency artifact authenticity cannot be verified without reviewing the resolved version in versions.jmh and the artifact hash.
sandbox/plugins/composite-engine/build.gradle	83	high	New annotation processor added: 'org.openjdk.jmh:jmh-generator-annprocess' via jmhAnnotationProcessor. Annotation processors execute arbitrary code at compile time; this is a high-risk supply chain vector.
sandbox/plugins/parquet-data-format/build.gradle	68	high	New dependency added: 'org.openjdk.jmh:jmh-core' under jmhImplementation configuration. Dependency artifact authenticity cannot be verified without reviewing the resolved version in versions.jmh and the artifact hash.
sandbox/plugins/parquet-data-format/build.gradle	69	high	New annotation processor added: 'org.openjdk.jmh:jmh-generator-annprocess' via jmhAnnotationProcessor. Annotation processors execute arbitrary code at compile time; this is a high-risk supply chain vector.

The table above displays the top 10 most important findings.

Total: 4 | Critical: 0 | High: 4 | Medium: 0 | Low: 0

---

Pull Requests Author(s): Please update your Pull Request according to the report above.

Repository Maintainer(s): You can bypass diff analyzer by adding label skip-diff-analyzer after reviewing the changes carefully, then re-run failed actions. To re-enable the analyzer, remove the label, then re-run all actions.

---

⚠️ Note: The Code-Diff-Analyzer helps protect against potentially harmful code patterns. Please ensure you have thoroughly reviewed the changes beforehand.

Thanks.