[SPARK-53191][CORE][SQL][MLLIB][YARN] Use Java InputStream.readAllBytes instead of ByteStreams.toByteArray

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to use Java InputStream.readAllBytes instead of ByteStreams.toByteArray in order to improve the performance.

Why are the changes needed?

Since Java 9+, we can use readAllBytes which is roughly 30% faster than ByteStreams.toByteArray.

BEFORE (ByteStreams.toByteArray)

scala> spark.time(com.google.common.io.ByteStreams.toByteArray(new java.io.FileInputStream("/tmp/1G.bin")).length)
Time taken: 386 ms
val res0: Int = 1073741824

AFTER (InputStream.readAllBytes)

scala> spark.time(new java.io.FileInputStream("/tmp/1G.bin").readAllBytes().length)
Time taken: 248 ms
val res0: Int = 1073741824

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the CIs.

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

Thank you, @yaooqinn ~

[dongjoon-hyun]

Merged to master for Apache Spark 4.1.0.