[Java] Remove Java 8 support in Arrow v18

[danepitkin]

Describe the enhancement requested

1. Java 8 is holding back development of newer Java features. For example, the Java Platform Module System (JPMS)[1], which was introduced in Java 9.
2. Java 8 is preventing Arrow from using latest packages/dependencies in some places. See examples[2][3][4].
3. Arrow Java is quite stable, so Java 8 users can probably be fine pinning the Arrow dependency if they aren't interested in upgrading Java versions.
4. Java 8 is on the decline, and is not the most used Java version in 2023[5].

[1]https://en.wikipedia.org/wiki/Java_Platform_Module_System
[2]https://github.com/apache/arrow/blob/main/dev/release/verify-release-candidate.sh#L571
[3]#37723 (comment)
[4]#13072 (comment)
[5]https://newrelic.com/sites/default/files/2023-04/new-relic-2023-state-of-the-java-ecosystem-2023-04-20.pdf

Post-upgrade tasks

• Bump ErrorProne (MINOR: [Java] Bump com.google.errorprone:error_prone_core from 2.4.0 to 2.24.0 in /java #39409)
• Bump Mockito (MINOR: [Java] Bump org.mockito:mockito-core from 4.11.0 to 5.8.0 in /java #39410)
• Bump mockito-junit-jupiter (MINOR: [Java] Bump org.mockito:mockito-junit-jupiter from 2.25.1 to 5.8.0 in /java #39408)
• Bump Derby (MINOR: [Java] Bump org.apache.derby:derby from 10.14.2.0 to 10.17.1.0 in /java #39281)
• Bump Checkstyle
• Bump ORC (MINOR: [Java] Bump org.apache.orc:orc-core from 1.9.2 to 2.0.0 in /java #40779)
• Bump logback (MINOR: [Java] Bump ch.qos.logback:logback-classic from 1.3.14 to 1.5.3 in /java #40778)

Component(s)

Java

[kou]

The discussion thread on dev@arrow.apache.org: https://lists.apache.org/thread/s07jx58yw4mkl54t3bkggnyg0sftcrr8

[davisusanibar]

In addition, the following dependencies are pinned for JDK8:

• In order to enable support for JDK21, this draft PR will force us to pin Mockito dependency version needed for JDK8 and another version for JDK11+.

• Error prone 2.11+ requires JDK11 minimum supported version, thus it was necessary to pin JDK8 compatibility.

[danepitkin]

Apache Spark has dropped support for Java 8 and 11 on the main branch (targeting a 4.0 release) apache/spark#43005

Edit: Spark 4.0 release timeframe is 2024-06[1]

[1]https://lists.apache.org/thread/xhkgj60j361gdpywoxxz7qspp2w80ry6

[danepitkin]

Netty 5.0 will remove support for Java 8 netty/netty#10650

[danepitkin]

The current consensus on the Arrow mailing list[1] is to postpone Java 8 deprecation and to revisit it when Spark releases 4.0, which deprecates Java 8 (~2024-06).

[1] https://lists.apache.org/thread/kml53f81z1oskcf00xl7wlbcjssmn91g

[danepitkin]

Apache Derby continuously drops support for older JDK versions #38813

[kevingurney]

My apologies!

I accidentally unpinned this issue because I thought I had pinned it just for me, by accident. I just repinned it.

[danepitkin]

Apache Iceberg is considering dropping java 8 support https://lists.apache.org/thread/ntrk2thvsg9tdccwd4flsdz9gg743368

[danepitkin]

New mailing list discussion: https://lists.apache.org/thread/65vqpmrrtpshxo53572zcv91j1lb2y8g