fix: emit JDK libjvm search path from core build script to fix -ljvm CI link failures

[andygrove]

Which issue does this PR close?

No dedicated issue. This addresses a repo-wide CI failure where native builds fail at link time with:

error: linking with `cc` failed: exit status: 1
  = note: /usr/bin/ld.bfd: cannot find -ljvm: No such file or directory
error: could not compile `datafusion-comet` (lib)

It has been hitting PR Build (Linux), Iceberg Spark SQL Tests, and Spark SQL Tests jobs across open PRs.

Rationale for this change

The hdfs-sys dependency (pulled in by the default hdfs-opendal feature) links libjvm and, from its build script, bakes an absolute -L search path to the JDK's libjvm directory. Cargo caches that build-script output and replays it when only downstream crates change, without re-running the build script.

CI caches native/target keyed on hashFiles('native/**/*.rs', ...Cargo.toml/lock). When a change does not touch native sources, the same cache is restored. If the runner's JDK has moved in the meantime (setup-java floats the Zulu patch version, so JAVA_HOME's value can stay the same while the underlying install is swapped), the replayed -L path points at a directory that no longer exists, and the final link of libcomet.so fails with cannot find -ljvm.

core is the crate that produces the final cdylib, so the fix belongs there: emit a -L search path for the currently resolved JDK at build time. The linker then finds libjvm regardless of any stale path replayed from a dependency's cached build script.

What changes are included in this PR?

• Add native/core/build.rs that:
  • emits cargo:rustc-link-search=native=$JAVA_HOME/lib/server (the libjvm location for every supported JDK, 11+), and
  • emits cargo:rerun-if-env-changed=JAVA_HOME and cargo:rerun-if-changed=<that dir> so the search path is refreshed automatically whenever the JDK is swapped, rather than relying on a cache invalidation.

No workflow or dependency changes are needed; the fix is self-contained in the crate that performs the final link, so it applies to every native-build job.

How are these changes tested?

Verbose local build (cargo build -p datafusion-comet -vv) confirms the core build script now contributes the search path alongside the dependency's:

[hdfs-sys 0.3.0]     cargo:rustc-link-search=native=<JDK>/lib/server
[datafusion-comet 1.0.0] cargo:rerun-if-changed=<JDK>/lib/server
[datafusion-comet 1.0.0] cargo:rustc-link-search=native=<JDK>/lib/server

so -ljvm resolves against the current JDK even when a dependency replays a stale path. The existing native-build and Spark SQL CI jobs exercise the link path that was failing.

[mbutrovich]

I was just starting to look at what broke the builds. Thanks @andygrove!

[parthchandra]

Do we still need hdfs-sys ?

[mbutrovich]

Do we still need hdfs-sys ?

Related? #4867

[comphead]

Do we still need hdfs-sys ?

Currently not used.

[comphead]

Thanks @andygrove hdfs-sys really need JVM runtime lib path, jvmlib.so, wondering how come the path is flaky 🤔 we need to keep in mind to drop this file once we decide to remove local hdfs crate

[andygrove]

Merged. Thanks @mbutrovich @comphead

[parthchandra]

Do we still need hdfs-sys ?

Currently not used.

If we are going to then we should probably remove it before 1.0