[SPARK-42488][BUILD] Upgrade commons-crypto to 1.2.0 #40082

LuciferYang · 2023-02-19T11:43:50Z

What changes were proposed in this pull request?

This pr aims upgrade commons-crypto from 1.1.0 to 1.2.0.

Why are the changes needed?

The new features in 1.2.0 as follows:

https://github.com/apache/commons-crypto/blob/1ebfddd0e77585884872416a0dff2dd114a88864/RELEASE-NOTES.txt#L12-L21

Does this PR introduce any user-facing change?

No

How was this patch tested?

Pass GitHub Actions

LuciferYang · 2023-02-19T11:44:03Z

test first

dongjoon-hyun

+1, LGTM. Thank you, @LuciferYang and @srowen .
Merged to master.

LuciferYang · 2023-02-20T05:03:36Z

Thanks @dongjoon-hyun @srowen

dongjoon-hyun · 2023-02-20T05:06:02Z

BTW, I have one question, @LuciferYang . We explicitly exclude jna from commons-crypto. Do you know what are we losing from the enumerated items in the commons-crypto releasenotes?

spark/pom.xml

Lines 2679 to 2689 in 5fc44da

    
           <dependency> 
        
             <groupId>org.apache.commons</groupId> 
        
             <artifactId>commons-crypto</artifactId> 
        
             <version>${commons-crypto.version}</version> 
        
             <exclusions> 
        
               <exclusion> 
        
                 <groupId>net.java.dev.jna</groupId> 
        
                 <artifactId>jna</artifactId> 
        
               </exclusion> 
        
             </exclusions> 
        
           </dependency>

LuciferYang · 2023-02-20T06:02:18Z

From the release notes, I think nothing is losing.

OpenSsl20XNativeJna, which is not mentioned in release notes, is a new support in version 1.2.0(Because jna is excluded, it cannot be used in Spark).

dongjoon-hyun · 2023-02-24T08:32:00Z

Oh, this seems to break Apple Silicon environment, @LuciferYang . I hit the failure on Macbook environment.

$ build/sbt "sql/testOnly *.RowQueueSuite"
...
[info] RowQueueSuite:
[info] - in-memory queue (33 milliseconds)
[info] - disk queue (encryption = off) (28 milliseconds)
[info] org.apache.spark.sql.execution.python.RowQueueSuite *** ABORTED *** (125 milliseconds)
[info]   java.lang.ExceptionInInitializerError:
[info]   at java.base/java.lang.Class.forName0(Native Method)
[info]   at java.base/java.lang.Class.forName(Class.java:398)
[info]   at org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93)
[info]   at org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64)
[info]   at org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189)
[info]   at org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138)
[info]   at org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56)
[info]   at org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151)
[info]   at scala.Option.map(Option.scala:230)
[info]   at org.apache.spark.serializer.SerializerManager.wrapForEncryption(SerializerManager.scala:151)
[info]   at org.apache.spark.sql.execution.python.DiskRowQueue.<init>(RowQueue.scala:119)
[info]   at org.apache.spark.sql.execution.python.RowQueueSuite.$anonfun$new$2(RowQueueSuite.scala:72)
[info]   at org.apache.spark.sql.execution.python.RowQueueSuite.$anonfun$new$2$adapted(RowQueueSuite.scala:68)
[info]   at org.apache.spark.security.EncryptionFunSuite.$anonfun$encryptionTest$2(EncryptionFunSuite.scala:32)
[info]   at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
[info]   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:207)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
[info]   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
[info]   at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:66)
[info]   at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
[info]   at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
[info]   at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:66)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
[info]   at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
[info]   at scala.collection.immutable.List.foreach(List.scala:431)
[info]   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
[info]   at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
[info]   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
[info]   at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
[info]   at org.scalatest.Suite.run(Suite.scala:1114)
[info]   at org.scalatest.Suite.run$(Suite.scala:1096)
[info]   at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
[info]   at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
[info]   at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:66)
[info]   at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
[info]   at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info]   at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info]   at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:66)
[info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info]   at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:413)
[info]   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info]   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[info]   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[info]   at java.base/java.lang.Thread.run(Thread.java:829)
[info]   Cause: java.lang.IllegalStateException: java.security.GeneralSecurityException: Native library is not loaded
[info]   at org.apache.commons.crypto.random.OpenSslCryptoRandom.<clinit>(OpenSslCryptoRandom.java:67)
[info]   at java.base/java.lang.Class.forName0(Native Method)
[info]   at java.base/java.lang.Class.forName(Class.java:398)
[info]   at org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93)
[info]   at org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64)
[info]   at org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189)
[info]   at org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138)
[info]   at org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56)
[info]   at org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151)
[info]   at scala.Option.map(Option.scala:230)
[info]   at org.apache.spark.serializer.SerializerManager.wrapForEncryption(SerializerManager.scala:151)
[info]   at org.apache.spark.sql.execution.python.DiskRowQueue.<init>(RowQueue.scala:119)
[info]   at org.apache.spark.sql.execution.python.RowQueueSuite.$anonfun$new$2(RowQueueSuite.scala:72)
[info]   at org.apache.spark.sql.execution.python.RowQueueSuite.$anonfun$new$2$adapted(RowQueueSuite.scala:68)
[info]   at org.apache.spark.security.EncryptionFunSuite.$anonfun$encryptionTest$2(EncryptionFunSuite.scala:32)
[info]   at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
[info]   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:207)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
[info]   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
[info]   at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:66)
[info]   at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
[info]   at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
[info]   at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:66)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
[info]   at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
[info]   at scala.collection.immutable.List.foreach(List.scala:431)
[info]   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
[info]   at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
[info]   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
[info]   at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
[info]   at org.scalatest.Suite.run(Suite.scala:1114)
[info]   at org.scalatest.Suite.run$(Suite.scala:1096)
[info]   at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
[info]   at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
[info]   at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:66)
[info]   at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
[info]   at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info]   at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info]   at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:66)
[info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info]   at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:413)
[info]   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info]   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[info]   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[info]   at java.base/java.lang.Thread.run(Thread.java:829)
[info]   Cause: java.security.GeneralSecurityException: Native library is not loaded
[info]   at org.apache.commons.crypto.random.OpenSslCryptoRandom.checkNative(OpenSslCryptoRandom.java:79)
[info]   at org.apache.commons.crypto.random.OpenSslCryptoRandom.<clinit>(OpenSslCryptoRandom.java:65)
[info]   at java.base/java.lang.Class.forName0(Native Method)
[info]   at java.base/java.lang.Class.forName(Class.java:398)
[info]   at org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93)
[info]   at org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64)
[info]   at org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189)
[info]   at org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138)
[info]   at org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56)
[info]   at org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151)
[info]   at scala.Option.map(Option.scala:230)
[info]   at org.apache.spark.serializer.SerializerManager.wrapForEncryption(SerializerManager.scala:151)
[info]   at org.apache.spark.sql.execution.python.DiskRowQueue.<init>(RowQueue.scala:119)
[info]   at org.apache.spark.sql.execution.python.RowQueueSuite.$anonfun$new$2(RowQueueSuite.scala:72)
[info]   at org.apache.spark.sql.execution.python.RowQueueSuite.$anonfun$new$2$adapted(RowQueueSuite.scala:68)
[info]   at org.apache.spark.security.EncryptionFunSuite.$anonfun$encryptionTest$2(EncryptionFunSuite.scala:32)
[info]   at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
[info]   at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:207)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
[info]   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
[info]   at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:66)
[info]   at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
[info]   at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
[info]   at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:66)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
[info]   at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
[info]   at scala.collection.immutable.List.foreach(List.scala:431)
[info]   at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
[info]   at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
[info]   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
[info]   at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
[info]   at org.scalatest.Suite.run(Suite.scala:1114)
[info]   at org.scalatest.Suite.run$(Suite.scala:1096)
[info]   at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
[info]   at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
[info]   at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:66)
[info]   at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
[info]   at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info]   at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info]   at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:66)
[info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info]   at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:413)
[info]   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[info]   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[info]   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[info]   at java.base/java.lang.Thread.run(Thread.java:829)
[error] Uncaught exception when running org.apache.spark.sql.execution.python.RowQueueSuite: java.lang.ExceptionInInitializerError
[error] sbt.ForkMain$ForkError: java.lang.ExceptionInInitializerError: null
[error] 	at java.base/java.lang.Class.forName0(Native Method)
[error] 	at java.base/java.lang.Class.forName(Class.java:398)
[error] 	at org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93)
[error] 	at org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64)
[error] 	at org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189)
[error] 	at org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138)
[error] 	at org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56)
[error] 	at org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151)
[error] 	at scala.Option.map(Option.scala:230)
[error] 	at org.apache.spark.serializer.SerializerManager.wrapForEncryption(SerializerManager.scala:151)
[error] 	at org.apache.spark.sql.execution.python.DiskRowQueue.<init>(RowQueue.scala:119)
[error] 	at org.apache.spark.sql.execution.python.RowQueueSuite.$anonfun$new$2(RowQueueSuite.scala:72)
[error] 	at org.apache.spark.sql.execution.python.RowQueueSuite.$anonfun$new$2$adapted(RowQueueSuite.scala:68)
[error] 	at org.apache.spark.security.EncryptionFunSuite.$anonfun$encryptionTest$2(EncryptionFunSuite.scala:32)
[error] 	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[error] 	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[error] 	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[error] 	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[error] 	at org.scalatest.Transformer.apply(Transformer.scala:22)
[error] 	at org.scalatest.Transformer.apply(Transformer.scala:20)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
[error] 	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:207)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
[error] 	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
[error] 	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:66)
[error] 	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
[error] 	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227
[error] 	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:66)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
[error] 	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413
[error] 	at scala.collection.immutable.List.foreach(List.scala:431)
[error] 	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
[error] 	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
[error] 	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
[error] 	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
[error] 	at org.scalatest.Suite.run(Suite.scala:1114)
[error] 	at org.scalatest.Suite.run$(Suite.scala:1096)
[error] 	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
[error] 	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
[error] 	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:66)
[error] 	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
[error] 	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[error] 	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[error] 	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:66)
[error] 	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[error] 	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[error] 	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:413)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[error] 	at java.base/java.lang.Thread.run(Thread.java:829)
[error] Caused by: sbt.ForkMain$ForkError: java.lang.IllegalStateException: java.security.GeneralSecurityException: Native library is not loaded
[error] 	at org.apache.commons.crypto.random.OpenSslCryptoRandom.<clinit>(OpenSslCryptoRandom.java:67)
[error] 	at java.base/java.lang.Class.forName0(Native Method)
[error] 	at java.base/java.lang.Class.forName(Class.java:398)
[error] 	at org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93)
[error] 	at org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64)
[error] 	at org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189)
[error] 	at org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138)
[error] 	at org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56)
[error] 	at org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151)
[error] 	at scala.Option.map(Option.scala:230)
[error] 	at org.apache.spark.serializer.SerializerManager.wrapForEncryption(SerializerManager.scala:151)
[error] 	at org.apache.spark.sql.execution.python.DiskRowQueue.<init>(RowQueue.scala:119)
[error] 	at org.apache.spark.sql.execution.python.RowQueueSuite.$anonfun$new$2(RowQueueSuite.scala:72)
[error] 	at org.apache.spark.sql.execution.python.RowQueueSuite.$anonfun$new$2$adapted(RowQueueSuite.scala:68)
[error] 	at org.apache.spark.security.EncryptionFunSuite.$anonfun$encryptionTest$2(EncryptionFunSuite.scala:32)
[error] 	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[error] 	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[error] 	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[error] 	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[error] 	at org.scalatest.Transformer.apply(Transformer.scala:22)
[error] 	at org.scalatest.Transformer.apply(Transformer.scala:20)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
[error] 	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:207)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
[error] 	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
[error] 	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:66)
[error] 	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
[error] 	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227
[error] 	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:66)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
[error] 	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413
[error] 	at scala.collection.immutable.List.foreach(List.scala:431)
[error] 	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
[error] 	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
[error] 	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
[error] 	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
[error] 	at org.scalatest.Suite.run(Suite.scala:1114)
[error] 	at org.scalatest.Suite.run$(Suite.scala:1096)
[error] 	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
[error] 	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
[error] 	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:66)
[error] 	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
[error] 	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[error] 	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[error] 	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:66)
[error] 	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[error] 	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[error] 	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:413)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[error] 	at java.base/java.lang.Thread.run(Thread.java:829)
[error] Caused by: sbt.ForkMain$ForkError: java.security.GeneralSecurityException: Native library is not loaded
[error] 	at org.apache.commons.crypto.random.OpenSslCryptoRandom.checkNative(OpenSslCryptoRandom.java:79)
[error] 	at org.apache.commons.crypto.random.OpenSslCryptoRandom.<clinit>(OpenSslCryptoRandom.java:65)
[error] 	at java.base/java.lang.Class.forName0(Native Method)
[error] 	at java.base/java.lang.Class.forName(Class.java:398)
[error] 	at org.apache.commons.crypto.utils.ReflectionUtils.getClassByNameOrNull(ReflectionUtils.java:93)
[error] 	at org.apache.commons.crypto.utils.ReflectionUtils.getClassByName(ReflectionUtils.java:64)
[error] 	at org.apache.commons.crypto.random.CryptoRandomFactory.getCryptoRandom(CryptoRandomFactory.java:189)
[error] 	at org.apache.spark.security.CryptoStreamUtils$.createInitializationVector(CryptoStreamUtils.scala:138)
[error] 	at org.apache.spark.security.CryptoStreamUtils$.createCryptoOutputStream(CryptoStreamUtils.scala:56)
[error] 	at org.apache.spark.serializer.SerializerManager.$anonfun$wrapForEncryption$3(SerializerManager.scala:151)
[error] 	at scala.Option.map(Option.scala:230)
[error] 	at org.apache.spark.serializer.SerializerManager.wrapForEncryption(SerializerManager.scala:151)
[error] 	at org.apache.spark.sql.execution.python.DiskRowQueue.<init>(RowQueue.scala:119)
[error] 	at org.apache.spark.sql.execution.python.RowQueueSuite.$anonfun$new$2(RowQueueSuite.scala:72)
[error] 	at org.apache.spark.sql.execution.python.RowQueueSuite.$anonfun$new$2$adapted(RowQueueSuite.scala:68)
[error] 	at org.apache.spark.security.EncryptionFunSuite.$anonfun$encryptionTest$2(EncryptionFunSuite.scala:32)
[error] 	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[error] 	at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[error] 	at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[error] 	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[error] 	at org.scalatest.Transformer.apply(Transformer.scala:22)
[error] 	at org.scalatest.Transformer.apply(Transformer.scala:20)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
[error] 	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:207)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
[error] 	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
[error] 	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:66)
[error] 	at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
[error] 	at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227
[error] 	at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:66)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
[error] 	at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413
[error] 	at scala.collection.immutable.List.foreach(List.scala:431)
[error] 	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
[error] 	at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
[error] 	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
[error] 	at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
[error] 	at org.scalatest.Suite.run(Suite.scala:1114)
[error] 	at org.scalatest.Suite.run$(Suite.scala:1096)
[error] 	at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
[error] 	at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
[error] 	at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
[error] 	at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:66)
[error] 	at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
[error] 	at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[error] 	at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[error] 	at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:66)
[error] 	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[error] 	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[error] 	at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:413)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[error] 	at java.base/java.lang.Thread.run(Thread.java:829)
[info] Run completed in 1 second, 175 milliseconds.
[info] Total number of tests run: 2
[info] Suites: completed 0, aborted 1
[info] Tests: succeeded 2, failed 0, canceled 0, ignored 0, pending 0
[info] *** 1 SUITE ABORTED ***
[error] Error during tests:
[error] 	org.apache.spark.sql.execution.python.RowQueueSuite
[error] (sql / Test / testOnly) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 176 s (02:56), completed Feb 24, 2023, 12:30:57 AM

dongjoon-hyun · 2023-02-24T08:33:08Z

In short, it's Native library is not loaded error. Could you verify from your side, @LuciferYang ?

java.lang.IllegalStateException: java.security.GeneralSecurityException: Native library is not loaded
[info]   at org.apache.commons.crypto.random.OpenSslCryptoRandom.<clinit>(OpenSslCryptoRandom.java:67)

LuciferYang · 2023-02-24T08:40:37Z

Thanks @dongjoon-hyun, it's my bad, I hit same issue, let me revert this one, I will remember check this part next time.

dongjoon-hyun · 2023-02-24T08:41:10Z

Well, let me revert directly~

dongjoon-hyun · 2023-02-24T08:41:22Z

Since it's on master only, it's easier.

LuciferYang · 2023-02-24T08:41:59Z

ok, thanks @dongjoon-hyun

dongjoon-hyun · 2023-02-24T08:42:19Z

It's reverted via ca22c41 .

dongjoon-hyun · 2023-02-24T08:49:19Z

FYI, I also confimed that the revert commit recovers the following streaming module test, too.

org.apache.spark.streaming.ReceivedBlockHandlerWithEncryptionSuite

ugprade

2674fc4

LuciferYang marked this pull request as draft February 19, 2023 11:43

github-actions bot added the BUILD label Feb 19, 2023

srowen approved these changes Feb 19, 2023

View reviewed changes

LuciferYang marked this pull request as ready for review February 20, 2023 03:06

dongjoon-hyun approved these changes Feb 20, 2023

View reviewed changes

dongjoon-hyun changed the title ~~[SPARK-42488][BUILD] Upgrade commons-crypto from 1.1.0 to 1.2.0~~ [SPARK-42488][BUILD] Upgrade commons-crypto to 1.2.0 Feb 20, 2023

dongjoon-hyun closed this in 5fc44da Feb 20, 2023

LuciferYang mentioned this pull request May 23, 2023

[SPARK-43739][BUILD] Upgrade commons-io to 2.12.0 #41271

Closed

LuciferYang mentioned this pull request Oct 21, 2023

[SPARK-45617][BUILD] Upgrade Apache Commons Crypto 1.2.0 #43467

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-42488][BUILD] Upgrade commons-crypto to 1.2.0 #40082

[SPARK-42488][BUILD] Upgrade commons-crypto to 1.2.0 #40082

LuciferYang commented Feb 19, 2023 •

edited

LuciferYang commented Feb 19, 2023

dongjoon-hyun left a comment

LuciferYang commented Feb 20, 2023

dongjoon-hyun commented Feb 20, 2023 •

edited

LuciferYang commented Feb 20, 2023 •

edited

dongjoon-hyun commented Feb 24, 2023 •

edited

dongjoon-hyun commented Feb 24, 2023

LuciferYang commented Feb 24, 2023

dongjoon-hyun commented Feb 24, 2023

dongjoon-hyun commented Feb 24, 2023

LuciferYang commented Feb 24, 2023

dongjoon-hyun commented Feb 24, 2023

dongjoon-hyun commented Feb 24, 2023

[SPARK-42488][BUILD] Upgrade commons-crypto to 1.2.0 #40082

[SPARK-42488][BUILD] Upgrade commons-crypto to 1.2.0 #40082

Conversation

LuciferYang commented Feb 19, 2023 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

LuciferYang commented Feb 19, 2023

dongjoon-hyun left a comment

Choose a reason for hiding this comment

LuciferYang commented Feb 20, 2023

dongjoon-hyun commented Feb 20, 2023 • edited

LuciferYang commented Feb 20, 2023 • edited

dongjoon-hyun commented Feb 24, 2023 • edited

dongjoon-hyun commented Feb 24, 2023

LuciferYang commented Feb 24, 2023

dongjoon-hyun commented Feb 24, 2023

dongjoon-hyun commented Feb 24, 2023

LuciferYang commented Feb 24, 2023

dongjoon-hyun commented Feb 24, 2023

dongjoon-hyun commented Feb 24, 2023

LuciferYang commented Feb 19, 2023 •

edited

dongjoon-hyun commented Feb 20, 2023 •

edited

LuciferYang commented Feb 20, 2023 •

edited

dongjoon-hyun commented Feb 24, 2023 •

edited