fix: expand object store option references, uniquify constant metadata names, drop dead parquet JNI

[dwsmith1983]

Which issue does this PR close?

Part of the restructuring of #5365 requested in review: these are three small core fixes that were previously bundled into that PR and are independent of the Delta contrib work. The first two are prerequisites for it; the third is standalone cleanup.

Rationale for this change

1. NativeConfig.extractObjectStoreOptions iterates the Hadoop conf and forwards raw entry values. Configuration#get expands ${...} variable references while Configuration.Entry#getValue does not, so any object store option holding a reference (for example an endpoint composed from another property) reached native as the unexpanded literal and diverged from what every Hadoop-side consumer observes.

2. Comet appends synthetic constant metadata fields named _comet_metadata_<name> to the scan schema. DataFusion substitutes partition constants by name, so a user or partition column that happens to carry one of those names would silently receive the constant metadata value instead of its own data.

3. The Java_org_apache_comet_parquet_Native_* record batch reader entry points (initRecordBatchReader, readNextRecordBatch, currentColumnBatch, closeRecordBatchReader) and their BatchContext machinery have no JVM callers anywhere in the tree. Dead code in a JNI surface is worth removing promptly since nothing exercises or protects it.

What changes are included in this PR?

• extractObjectStoreOptions resolves each value through Configuration#get, falling back to the raw literal when get returns null (deprecated key aliasing) or raises on a substitution cycle, so extraction never aborts.
• A uniqueConstantMetadataFields helper uniquifies synthetic metadata field names against the data and partition schema names with trailing underscores; binding stays positional so renaming is always safe.
• Removal of the four dead JNI entry points and their supporting state (306 lines), plus one stale doc comment that referenced them.

How are these changes tested?

• Two new tests in NativeConfigSuite cover ${...} expansion and the cycle fallback.
• New CometNativeScanSuite covers prefixing and collision uniquification (written first and failing before the fix).
• cargo test -p datafusion-comet --lib (240 passed), clippy with -D warnings, fmt, and the Spark 3.5 build with NativeConfigSuite/CometNativeScanSuite all green. A repo-wide grep confirms no callers of the removed JNI symbols.

[sunchao]

Reviewed 8ffc026d8ea36fdcf08e684e03c93d40285002a8 against 10537e1402716896e9a21344998d9bd05198bcda. The configuration change expands Hadoop references, the metadata helper preserves positional binding, and the removed JNI methods have no remaining tracked callers. One P2 needs addressing: the new scan suite is not registered in the PR build matrices.

Current-head CI has four successful checks, one failed Preflight check and thirteen skipped checks. Preflight stopped at suite registration before the main build and test jobs could run. This is not a failed Spark test. The unchanged checker reproduced the registration failure in a reduced source fixture. Configuration component tests were also run, but no full Spark scan, JNI execution or native build was performed.

[sunchao]

[P2] Register the new scan suite in both CI matrices

Could you add org.apache.comet.serde.operator.CometNativeScanSuite to the suite lists in both .github/workflows/pr_build_linux.yml and .github/workflows/pr_build_macos.yml? The Preflight job for this head fails at Check missing suites, reports this class missing from the Linux workflow, and exits with code 255. The checker requires registration in both files, and the class is absent from both. This stops Detect changes and leaves the main build and test jobs skipped. Registering only the Linux entry would leave the macOS requirement unsatisfied.

[dwsmith1983]

Already caught this from the Preflight log and pushed the registration in b234d9c, both workflows. The checker passes locally now.

[sunchao]

Confirmed that CometNativeScanSuite is registered in both PR build matrices at b234d9cad0ed4116d9f018b88baf0fbf39dd8b10. The unchanged checker passes against the complete tracked suite set, and the old-head and Linux-only controls reproduce the expected failures. Product source, tests and dependencies are unchanged from the previous review. I have no remaining P1/P2 concerns.

The reviewed CI snapshot still has action_required workflows and no test results for this head. The local registration check is not a full Spark or native build.

[sunchao]

Re-reviewed 943437274cc16a61127cac3c0017c1dd0f47e300 against ef62b463. The PR-only patch and suite-registration fix are unchanged. I checked the incoming shuffle decoder's interaction with the removed Parquet JNI path and constant-field binding and found no new P1/P2. The existing approval stands.

This was a source/integration review using the unchanged prior component and registration evidence. I did not rerun tests or benchmarks. The three current-head workflows still require authorization, with no executed check results reported.

[sunchao]

Merged, thanks @dwsmith1983 !