feat(logging): bridge native IVF-PQ diagnostics to SLF4J

[jerry-024]

Summary

Route opt-in native IVF-PQ diagnostics through Java SLF4J/log4j when the JNI library is loaded, while retaining a stderr fallback for non-JNI consumers.

Changes

• Add opt-in IVF-PQ batch phase timing through PAIMON_VINDEX_LOG_IVFPQ_BATCH_TIMING.
• Route timing and table-reuse diagnostics through a process-wide core log sink.
• Install a JNI bridge during JNI_OnLoad and map native levels to SLF4J.
• Add a forked-JVM smoke test that verifies timing reaches SLF4J instead of stderr.

Testing

• cargo fmt --all -- --check
• mvn -f java/pom.xml test-compile
• cargo test -p paimon-vindex-core (GitHub x86 CI; local ARM is blocked by an existing unstable NEON intrinsic on main)
• JNI build and native log bridge smoke test in GitHub CI

Notes

• No public API or index format changes.

[shyjsarah]

[major] End read measurement and emit diagnostics on error paths

After begin_read_metrics(), several fallible operations can return early through ?, including list_payload_len, streamed reads, batch_read_end, and read_inverted_list_payloads. end_read_metrics() is only called on the successful path.

Consequently, failed searches emit no timing diagnostic and leave metrics enabled on the reader, so later operations continue collecting measurements and paying instrumentation overhead.

Please use an RAII guard or an equivalent finally-style scope to end measurement on every exit path. When timing is enabled, it would also be useful to emit a record with status=error, the completed phase timings, I/O metrics, and a non-sensitive error category.

[shyjsarah]

[major] Clarify or cover the single-query IVF-PQ search paths

PAIMON_VINDEX_LOG_IVFPQ_TIMING is only checked in the batch implementation. The JNI search and searchWithRoaringFilter methods use the single-query reader path, so those operations never produce IVF-PQ timing diagnostics.

Please either instrument search_with_reader_filter as well, or make the batch-only scope explicit by renaming the flag and documentation to something such as PAIMON_VINDEX_LOG_IVFPQ_BATCH_TIMING.

[shyjsarah]

[major] Assert SLF4J delivery rather than only checking System.err

This assertion only proves that the marker reaches Java's System.err. Both slf4j-simple and the fallback in NativeLogBridge.log() write to that stream, so an exception in the SLF4J call can fall back to System.err.println(message) and still pass this test.

Please use a recording SLF4J binding/appender and assert the logger name, level, and message directly. The stderr fallback should be exercised separately by an explicit failure-path test.

[shyjsarah]

[minor] Cover the table-reuse emitter in the JNI bridge test

This is the second diagnostic emitter migrated to emit_log, but the smoke test only enables PAIMON_VINDEX_LOG_IVFPQ_TIMING and builds an L2 residual index. It therefore never enters this !by_residual branch.

Please add a forked-JVM case using an inner-product or cosine index with PAIMON_VINDEX_LOG_IVFPQ_BATCH_REUSE=1, and verify that the ivfpq_batch_table_reuse record is delivered through the Java logging bridge.

[jerry-024]

Thanks. This extra forked-JVM case is not needed: the JNI bridge is a process-wide, message-agnostic sink, and the existing timing smoke test already exercises that transport end to end. The table-reuse behavior itself is covered by the core batch-reuse tests. A second IP/cosine JVM search would repeat the same bridge path without covering different delivery behavior.

[shyjsarah]

+1