Spark 4.0 - Dependencies Validation

[BentsiLeviav]

The following dependencies were upgraded in Spark 4.0; we need to make sure the release is aligned with these:

Library Name	Version Change
activation	1.1.1 -> REMOVED
aircompressor	0.27 -> 2.0.2
algebra	2.0.1 -> 2.8.0
aliyun-sdk-oss	3.13.0 -> 3.13.2
antlr4-runtime	4.9.3 -> 4.13.1
aopalliance-repackaged	2.6.1 -> 3.0.6
arrow-format	12.0.1 -> 18.1.0
arrow-memory-core	12.0.1 -> 18.1.0
arrow-memory-netty	12.0.1 -> 18.1.0
arrow-memory-netty-buffer-patch	-> 18.1.0 (NEW)
arrow-vector	12.0.1 -> 18.1.0
audience-annotations	0.5.0 -> 0.12.0
avro	1.11.4 -> 1.12.0
avro-ipc	1.11.4 -> 1.12.0
avro-mapred	1.11.4 -> 1.12.0
aws-java-sdk-bundle	1.12.262 -> REMOVED
bcprov-jdk18on	-> 1.80 (NEW)
bonecp	0.8.0.RELEASE -> REMOVED
bundle	-> 2.25.53 (NEW)
cats-kernel	2.1.1 -> 2.8.0
checker-qual	-> 3.43.0 (NEW)
chill	0.10.0 -> 0.10.0
commons-cli	1.5.0 -> 1.9.0
commons-codec	1.16.1 -> 1.17.2
commons-compress	1.23.0 -> 1.27.1
commons-io	2.16.1 -> 2.18.0
commons-lang3	3.12.0 -> 3.17.0
commons-text	1.10.0 -> 1.13.0
curator-client	2.13.0 -> 5.7.1
curator-framework	2.13.0 -> 5.7.1
curator-recipes	2.13.0 -> 5.7.1
datasketches-java	3.3.0 -> 6.1.1
datasketches-memory	2.1.0 -> 3.0.2
derby	10.14.2.0 -> 10.16.1.1
derbyshared	-> 10.16.1.1 (NEW)
derbytools	-> 10.16.1.1 (NEW)
error_prone_annotations	-> 2.36.0 (NEW)
esdk-obs-java	-> 3.20.4.2 (NEW)
failureaccess	-> 1.0.2 (NEW)
flatbuffers-java	1.12.0 -> 24.3.25
gcs-connector	hadoop3-2.2.14 -> hadoop3-2.2.26
gson	2.2.4 -> 2.11.0
guava	14.0.1 -> 33.4.0-jre
hadoop-aliyun	3.3.4 -> 3.4.1
hadoop-annotations	3.3.4 -> 3.4.1
hadoop-aws	3.3.4 -> 3.4.1
hadoop-azure	3.3.4 -> 3.4.1
hadoop-azure-datalake	3.3.4 -> 3.4.1
hadoop-client-api	3.3.4 -> 3.4.1
hadoop-client-runtime	3.3.4 -> 3.4.1
hadoop-cloud-storage	3.3.4 -> 3.4.1
hadoop-huaweicloud	-> 3.4.1 (NEW)
hadoop-openstack	3.3.4 -> REMOVED
hadoop-shaded-guava	1.1.1 -> 1.3.0
hadoop-yarn-server-web-proxy	3.3.4 -> REMOVED
hive-beeline	2.3.9 -> 2.3.10
hive-cli	2.3.9 -> 2.3.10
hive-common	2.3.9 -> 2.3.10
hive-exec	2.3.9 -> 2.3.10
hive-jdbc	2.3.9 -> 2.3.10
hive-llap-common	2.3.9 -> REMOVED
hive-metastore	2.3.9 -> 2.3.10
hive-serde	2.3.9 -> 2.3.10
hive-service-rpc	3.1.3 -> 4.0.0
hive-shims	2.3.9 -> 2.3.10
hive-shims-0.23	2.3.9 -> 2.3.10
hive-shims-common	2.3.9 -> 2.3.10
hive-shims-scheduler	2.3.9 -> 2.3.10
hk2-api	2.6.1 -> 3.0.6
hk2-locator	2.6.1 -> 3.0.6
hk2-utils	2.6.1 -> 3.0.6
icu4j	-> 76.1 (NEW)
istack-commons-runtime	3.0.8 -> 4.1.2
ivy	2.5.1 -> 2.5.3
j2objc-annotations	-> 3.0.0 (NEW)
jackson-annotations	2.15.2 -> 2.18.2
jackson-core	2.15.2 -> 2.18.2
jackson-core-asl	1.9.13 -> REMOVED
jackson-databind	2.15.2 -> 2.18.2
jackson-dataformat-cbor	2.15.2 -> 2.18.2
jackson-dataformat-yaml	2.15.2 -> 2.18.2
jackson-datatype-jsr310	2.15.2 -> 2.18.2
jackson-mapper-asl	1.9.13 -> REMOVED
jackson-module-scala	2.15.2 -> 2.18.2
jakarta.activation-api	-> 2.1.3 (NEW)
jakarta.annotation-api	1.3.5 -> 2.1.1
jakarta.inject	2.6.1 -> REMOVED
jakarta.inject-api	-> 2.0.1 (NEW)
jakarta.servlet-api	4.0.3 -> 5.0.0
jakarta.validation-api	2.0.2 -> 3.0.2
jakarta.ws.rs-api	2.1.6 -> 3.0.0
jakarta.xml.bind-api	2.3.2 -> 4.0.2
java-diff-utils	-> 4.15 (NEW)
java-xmlbuilder	-> 1.2 (NEW)
javassist	3.29.2-GA -> 3.30.2-GA
javax.servlet-api	4.0.1 -> REMOVED
jaxb-api	2.2.11 -> REMOVED
jaxb-core	-> 4.0.5 (NEW)
jaxb-runtime	2.3.2 -> 4.0.5
jcl-over-slf4j	2.0.7 -> 2.0.16
jersey-client	2.40 -> 3.0.16
jersey-common	2.40 -> 3.0.16
jersey-container-servlet	2.40 -> 3.0.16
jersey-container-servlet-core	2.40 -> 3.0.16
jersey-hk2	2.40 -> 3.0.16
jersey-server	2.40 -> 3.0.16
jettison	1.1 -> 1.5.4
jetty-util	9.4.56.v20240826 -> 11.0.24
jetty-util-ajax	9.4.56.v20240826 -> 11.0.24
jjwt-api	-> 0.12.6 (NEW)
jjwt-impl	-> 0.12.6 (NEW)
jjwt-jackson	-> 0.12.6 (NEW)
jline	2.14.6 -> 3.27.1
joda-time	2.12.5 -> 2.13.0
json4s-ast	3.7.0-M11 -> 4.0.7
json4s-core	3.7.0-M11 -> 4.0.7
json4s-jackson	3.7.0-M11 -> 4.0.7
json4s-jackson-core	-> 4.0.7 (NEW)
json4s-scalap	3.7.0-M11 -> 4.0.7
jul-to-slf4j	2.0.7 -> 2.0.16
kryo-shaded	4.0.2 -> 4.0.3
kubernetes-client	6.7.2 -> 7.1.0
kubernetes-client-api	6.7.2 -> 7.1.0
kubernetes-httpclient-okhttp	6.7.2 -> REMOVED
kubernetes-httpclient-vertx	-> 7.1.0 (NEW)
kubernetes-model-admissionregistration	6.7.2 -> 7.1.0
kubernetes-model-apiextensions	6.7.2 -> 7.1.0
kubernetes-model-apps	6.7.2 -> 7.1.0
kubernetes-model-autoscaling	6.7.2 -> 7.1.0
kubernetes-model-batch	6.7.2 -> 7.1.0
kubernetes-model-certificates	6.7.2 -> 7.1.0
kubernetes-model-common	6.7.2 -> 7.1.0
kubernetes-model-coordination	6.7.2 -> 7.1.0
kubernetes-model-core	6.7.2 -> 7.1.0
kubernetes-model-discovery	6.7.2 -> 7.1.0
kubernetes-model-events	6.7.2 -> 7.1.0
kubernetes-model-extensions	6.7.2 -> 7.1.0
kubernetes-model-flowcontrol	6.7.2 -> 7.1.0
kubernetes-model-gatewayapi	6.7.2 -> 7.1.0
kubernetes-model-metrics	6.7.2 -> 7.1.0
kubernetes-model-networking	6.7.2 -> 7.1.0
kubernetes-model-node	6.7.2 -> 7.1.0
kubernetes-model-policy	6.7.2 -> 7.1.0
kubernetes-model-rbac	6.7.2 -> 7.1.0
kubernetes-model-resource	6.7.2 -> 7.1.0
kubernetes-model-scheduling	6.7.2 -> 7.1.0
kubernetes-model-storageclass	6.7.2 -> 7.1.0
libthrift	0.12.0 -> 0.16.0
listenablefuture	-> 9999.0-empty-to-avoid-conflict-with-guava (NEW)
log4j-1.2-api	2.20.0 -> 2.24.3
log4j-api	2.20.0 -> 2.24.3
log4j-core	2.20.0 -> 2.24.3
log4j-layout-template-json	-> 2.24.3 (NEW)
log4j-slf4j2-impl	2.20.0 -> 2.24.3
logging-interceptor	3.12.12 -> REMOVED
mesos	1.4.3 -> REMOVED
metrics-core	4.2.19 -> 4.2.30
metrics-graphite	4.2.19 -> 4.2.30
metrics-jmx	4.2.19 -> 4.2.30
metrics-json	4.2.19 -> 4.2.30
metrics-jvm	4.2.19 -> 4.2.30
netty-all	4.1.96.Final -> 4.1.118.Final
netty-buffer	4.1.96.Final -> 4.1.118.Final
netty-codec	4.1.96.Final -> 4.1.118.Final
netty-codec-dns	-> 4.1.118.Final (NEW)
netty-codec-http	4.1.96.Final -> 4.1.118.Final
netty-codec-http2	4.1.96.Final -> 4.1.118.Final
netty-codec-socks	4.1.96.Final -> 4.1.118.Final
netty-common	4.1.96.Final -> 4.1.118.Final
netty-handler	4.1.96.Final -> 4.1.118.Final
netty-handler-proxy	4.1.96.Final -> 4.1.118.Final
netty-resolver	4.1.96.Final -> 4.1.118.Final
netty-resolver-dns	-> 4.1.118.Final (NEW)
netty-tcnative-boringssl-static	-> 2.0.70.Final (NEW)
netty-tcnative-classes	-> 2.0.70.Final (NEW)
netty-transport	4.1.96.Final -> 4.1.118.Final
netty-transport-classes-epoll	4.1.96.Final -> 4.1.118.Final
netty-transport-classes-kqueue	4.1.96.Final -> 4.1.118.Final
netty-transport-native-epoll	4.1.96.Final -> 4.1.118.Final
netty-transport-native-kqueue	4.1.96.Final -> 4.1.118.Final
netty-transport-native-unix-common	4.1.96.Final -> 4.1.118.Final
orc-core	1.9.6 -> 2.1.2
orc-format	-> 1.1.0 (NEW)
orc-mapreduce	1.9.6 -> 2.1.2
orc-shims	1.9.6 -> 2.1.2
parquet-column	1.13.1 -> 1.15.2
parquet-common	1.13.1 -> 1.15.2
parquet-encoding	1.13.1 -> 1.15.2
parquet-format-structures	1.13.1 -> 1.15.2
parquet-hadoop	1.13.1 -> 1.15.2
parquet-jackson	1.13.1 -> 1.15.2
pickle	1.3 -> 1.5
py4j	0.10.9.7 -> 0.10.9.9
rocksdbjni	8.3.2 -> 9.8.4
RoaringBitmap	0.9.45 -> 1.3.0
scala-collection-compat	2.7.0 -> 2.7.0
scala-compiler	2.12.18 -> 2.13.16
scala-library	2.12.18 -> 2.13.16
scala-parallel-collections	-> 1.2.0 (NEW)
scala-parser-combinators	2.3.0 -> 2.4.0
scala-reflect	2.12.18 -> 2.13.16
scala-xml	2.1.0 -> 2.3.0
shims	0.9.45 -> REMOVED
slf4j-api	2.0.7 -> 2.0.16
snakeyaml	2.0 -> 2.3
snakeyaml-engine	2.6 -> 2.9
snappy-java	1.1.10.5 -> 1.1.10.7
spire	0.17.0 -> 0.18.0
spire-macros	0.17.0 -> 0.18.0
spire-platform	0.17.0 -> 0.18.0
spire-util	0.17.0 -> 0.18.0
stream	2.9.6 -> 2.9.8
threeten-extra	1.7.1 -> 1.8.0
tink	1.9.0 -> 1.16.0
transaction-api	1.1 -> REMOVED
vertx-auth-common	-> 4.5.12 (NEW)
vertx-core	-> 4.5.12 (NEW)
vertx-web-client	-> 4.5.12 (NEW)
vertx-web-common	-> 4.5.12 (NEW)
wildfly-openssl	1.0.7.Final -> 2.2.5.Final
xbean-asm9-shaded	4.23 -> 4.26
xmlschema-core	-> 2.3.1 (NEW)
xz	1.9 -> 1.10
zjsonpatch	0.3.0 -> 7.1.0
zookeeper	3.6.3 -> 3.9.3
zookeeper-jute	3.6.3 -> 3.9.3
zstd-jni	1.5.5-4 -> 1.5.6-9

[BentsiLeviav]

@ShimonSte in #452, did we account for that? Did we validate these dependencies?