Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BIGTOP-4062. Upgrade Spark to 3.3.4. #1234

Merged
merged 1 commit into from
Feb 25, 2024
Merged

Conversation

iwasakims
Copy link
Member

https://issues.apache.org/jira/browse/BIGTOP-4062

The latest maintenance release of Spark 3.3 series with security and correctness fixes is available.

@iwasakims
Copy link
Member Author

ran smoke test of spark on Ubuntu 22.04 x86_64.

@iwasakims
Copy link
Member Author

iwasakims commented Feb 7, 2024

ran smoke test of spark on Rocky Linux 8 aarch64.

@sekikn
Copy link
Contributor

sekikn commented Feb 25, 2024

+1, tested on Fedora 38 (x86_64). Thanks @iwasakims.

$ ./gradlew allclean spark-pkg repo -Dbuildwithdeps=true
$ (cd docker/bigtop-puppet && ./build.sh trunk-fedora-38)
$ (cd docker/bigtop-slaves && ./build.sh trunk-fedora-38)
$ (cd provisioner/docker && ./docker-hadoop.sh -d -dcp -C config_fedora-38.yaml -F docker-compose-cgroupv2.yml -G -L -k hdfs,yarn,spark -s spark -c 3)

...

org.apache.bigtop.itest.spark.TestSpark > testSparkSQL STANDARD_OUT
    SPARK_MASTER isn't set. yarn will be used as the master URL. Refer to smoke-tests/README If this isn't what you you expect.

org.apache.bigtop.itest.spark.TestSpark > testSparkR STANDARD_OUT
    SPARK_MASTER isn't set. yarn will be used as the master URL. Refer to smoke-tests/README If this isn't what you you expect.

Gradle Test Executor 2 finished executing tests.

> Task :bigtop-tests:smoke-tests:spark:test
Finished generating test XML results (0.006 secs) into: /bigtop-home/bigtop-tests/smoke-tests/spark/build/test-results/test
Generating HTML test report...
Finished generating test html results (0.011 secs) into: /bigtop-home/bigtop-tests/smoke-tests/spark/build/reports/tests/test
Now testing...
:bigtop-tests:smoke-tests:spark:test (Thread[Daemon worker,5,main]) completed. Took 49.935 secs.

BUILD SUCCESSFUL in 2m 15s

@sekikn sekikn merged commit db4ddc1 into apache:master Feb 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants