Upgrade minimum Java version to 21

[jarohen]

Once this is:

• generally available in the
  • Manjaro (yes, being selfish here 😉)
    (not generally available but it's in the Arch repos and installing the manually downloaded packages works fine)
  • Mac repos
• Java Github action,
• Gradle
  • Clojurephant
    • NoSuchMethodError Instrumented.fileCollectionObserved upgrading to Gradle 8 clojurephant/clojurephant#206
• supported in Arrow: [Java][CI] Enable JDK 21 apache/arrow#36994 (to be released in 14.0.0)

let's bump the minimum requirements for XT2 core.

• Core package and server modules should have 21 as their minimum requirement - this is for us to (eventually) take advantage of Loom et al. Given that folks that don't want their applications to run on 21 can deploy XTDB separately and connect through the client, as well as the benefits it brings, I'm happy with imposing a high minimum standard here.
• API and client modules should keep 11 as their minimum requirement, so that people who can't upgrade can still embed these in their application. We'll want to set these explicitly in the build files, obv, which will save us from compile-time incompatibilities, but we'd also benefit from tests which actually use Java 11 connecting to a Java 21 server.
  • Discuss: we could make 17 the minimum requirement here - anyone who can't use 17 can still use PgWire/HTTP, but this would mean us being able to use records/sealed interfaces etc in our XTQL AST
• Anyone who can't upgrade to Java 11 can still use HTTP, but without our official client Clojure lib support.

[jarohen]

Doesn't seem like Arrow supports Java 21 due to its use of Unsafe

1. Unhandled java.lang.UnsupportedOperationException
   sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available

           MemoryUtil.java:  174  org.apache.arrow.memory.util.MemoryUtil/directBuffer
             ArrowBuf.java:  229  org.apache.arrow.memory.ArrowBuf/getDirectBuffer
             ArrowBuf.java:  224  org.apache.arrow.memory.ArrowBuf/nioBuffer
         WriteChannel.java:  133  org.apache.arrow.vector.ipc.WriteChannel/write
    MessageSerializer.java:  303  org.apache.arrow.vector.ipc.message.MessageSerializer/writeBatchBuffers
    MessageSerializer.java:  276  org.apache.arrow.vector.ipc.message.MessageSerializer/serialize
          ArrowWriter.java:  147  org.apache.arrow.vector.ipc.ArrowWriter/writeRecordBatch
          ArrowWriter.java:  133  org.apache.arrow.vector.ipc.ArrowWriter/writeBatch
                  util.clj:  416  xtdb.util/build-arrow-ipc-byte-buffer/fn/fn/fn/write-batch!

[danepitkin]

Have you tried running with --add-opens=java.base/java.nio=ALL-UNNAMED? I believe this is a known issue. See https://arrow.apache.org/docs/java/install.html#java-compatibility

[jarohen]

Hey @danepitkin, thanks for your comment 🙂

Yep, we're aware of that one, have got that in (and confirm we run into the known issue when it's removed). In our case, it works against Java 17 but not 21 - I think they changed the constructor of DirectByteBuffer in Java 21: https://issues.apache.org/jira/browse/SPARK-42369, which is depended on in MemoryUtil. Will write a note to this effect on the Arrow card 🙂

James

[jarohen]

@danepitkin aha, fixed already in apache/arrow#35053 🙂

[danepitkin]

Ah, thanks for digging in to it! Sorry for the rabbit hole. The next Arrow release (v14) should be released in October, sometime after Oct 10th (code freeze).

[jarohen]

Cheers @danepitkin - can confirm this now works for us under the new Arrow 14 release 🙏