[SPARK-41392][BUILD] Make maven build Spark master with Hadoop 3.4.0-SNAPSHOT successful

[LuciferYang]

What changes were proposed in this pull request?

This pr aims to add bc-java related to test dependencies to sql module to make maven build Spark master with Hadoop 3.4.0-SNAPSHOT successful.

Why are the changes needed?

Make maven build Spark master with Hadoop 3.4.0-SNAPSHOT successful.

Does this PR introduce any user-facing change?

No

How was this patch tested?

• Pass GitHub Actions
• Manual test:

build/mvn clean install -Phadoop-3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -DskipTests -Dhadoop.version=3.4.0-SNAPSHOT -Psnapshots-and-staging

Before

Failed with org.bouncycastle.jce.provider.BouncyCastleProvider ClassNotFoundException.

After

BUILD SUCCESS

[LuciferYang]

https://github.com/LuciferYang/make-distribution.sh/blob/master/.github/workflows/blank.yml

# This is a basic workflow to help you get started with Actions

name: CI

# Controls when the workflow will run
on:
  # Triggers the workflow on push or pull request events but only for the "master" branch
  push:
    branches: [ "master" ]
  pull_request:
    branches: [ "master" ]

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

jobs:
  build:

    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        java: [ '1.8', '11', '17' ]
    name: Build Spark with JDK ${{ matrix.java }}

    steps:
      - uses: actions/checkout@master
      - name: Set up JDK ${{ matrix.java }}
        uses: actions/setup-java@v1
        with:
          java-version: ${{ matrix.java }}
          distribution: 'zulu'
      - name: Install Python 3.8
        uses: actions/setup-python@v4
        with:
          python-version: 3.8
          architecture: x64
      - name: Install Python packages (Python 3.8)
        run: |
          python3.8 -m pip install 'numpy>=1.20.0' pyarrow pandas scipy unittest-xml-reporting 'grpcio==1.48.1' 'protobuf==4.21.6'
          python3.8 -m pip list
      - name: build
        env:
          GH_TOKEN: ${{ github.token }}
        run: |
          git clone https://github.com/apache/spark.git
          cd spark
          gh pr checkout 38974
          build/mvn clean install -DskipTests -Phadoop-3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -Dhadoop.version=3.4.0-SNAPSHOT -Psnapshots-and-staging

https://github.com/LuciferYang/make-distribution.sh/actions/runs/3645098350

c1539ee (with scala-maven-plugin 4.8.0)build with above Github Actions success

[LuciferYang]

Run

build/mvn clean install -Phadoop-3 -Phadoop-cloud -Pmesos -Pyarn -Pkinesis-asl -Phive-thriftserver -Pspark-ganglia-lgpl -Pkubernetes -Phive -DskipTests -Dhadoop.version=3.4.0-SNAPSHOT -Psnapshots-and-staging

locally with

openjdk version "1.8.0_352"
OpenJDK Runtime Environment (Zulu 8.66.0.15-CA-macos-aarch64) (build 1.8.0_352-b08)
OpenJDK 64-Bit Server VM (Zulu 8.66.0.15-CA-macos-aarch64) (build 25.352-b08, mixed mode)

Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
Maven home: /Users/yangjie01/SourceCode/tools/maven
Java version: 1.8.0_352, vendor: Azul Systems, Inc., runtime: /Users/yangjie01/SourceCode/tools/zulu8u352/zulu-8.jdk/Contents/Home/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "mac os x", version: "11.4", arch: "aarch64", family: "mac"

SUCCESS

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 3.4.0-SNAPSHOT:
[INFO] 
[INFO] Spark Project Parent POM ........................... SUCCESS [  4.019 s]
[INFO] Spark Project Tags ................................. SUCCESS [  8.647 s]
[INFO] Spark Project Sketch ............................... SUCCESS [  9.546 s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 10.114 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 14.079 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 11.954 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 12.983 s]
[INFO] Spark Project Launcher ............................. SUCCESS [  7.930 s]
[INFO] Spark Project Core ................................. SUCCESS [02:03 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 26.217 s]
[INFO] Spark Project GraphX ............................... SUCCESS [ 30.525 s]
[INFO] Spark Project Streaming ............................ SUCCESS [ 49.190 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [03:59 min]
[INFO] Spark Project SQL .................................. SUCCESS [03:30 min]
[INFO] Spark Project ML Library ........................... SUCCESS [02:26 min]
[INFO] Spark Project Tools ................................ SUCCESS [  6.388 s]
[INFO] Spark Project Hive ................................. SUCCESS [01:14 min]
[INFO] Spark Project REPL ................................. SUCCESS [ 17.813 s]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 35.463 s]
[INFO] Spark Project YARN ................................. SUCCESS [ 36.535 s]
[INFO] Spark Project Mesos ................................ SUCCESS [ 26.025 s]
[INFO] Spark Project Kubernetes ........................... SUCCESS [ 54.553 s]
[INFO] Spark Project Hive Thrift Server ................... SUCCESS [ 35.123 s]
[INFO] Spark Ganglia Integration .......................... SUCCESS [  8.270 s]
[INFO] Spark Project Hadoop Cloud Integration ............. SUCCESS [ 53.754 s]
[INFO] Spark Project Assembly ............................. SUCCESS [ 14.055 s]
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [ 15.740 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 23.491 s]
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [ 31.237 s]
[INFO] Spark Kinesis Integration .......................... SUCCESS [ 23.730 s]
[INFO] Spark Project Examples ............................. SUCCESS [ 57.467 s]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [  6.690 s]
[INFO] Spark Avro ......................................... SUCCESS [ 29.417 s]
[INFO] Spark Project Connect .............................. SUCCESS [ 52.574 s]
[INFO] Spark Protobuf ..................................... SUCCESS [ 25.662 s]
[INFO] Spark Project Kinesis Assembly ..................... SUCCESS [  9.153 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  25:43 min
[INFO] Finished at: 2022-12-08T12:24:31+08:00
[INFO] ------------------------------------------------------------------------

maven-build-java8-hadoop34.log

[LuciferYang]

Downgrading scala-maven-plugin will reach 4.7.2, and the local maven build will still pass. I will update pr and test the GA compilation

[INFO] Reactor Summary for Spark Project Parent POM 3.4.0-SNAPSHOT:
[INFO] 
[INFO] Spark Project Parent POM ........................... SUCCESS [  3.641 s]
[INFO] Spark Project Tags ................................. SUCCESS [  6.959 s]
[INFO] Spark Project Sketch ............................... SUCCESS [  7.881 s]
[INFO] Spark Project Local DB ............................. SUCCESS [  7.379 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 11.208 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [  9.188 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 10.616 s]
[INFO] Spark Project Launcher ............................. SUCCESS [  5.707 s]
[INFO] Spark Project Core ................................. SUCCESS [02:22 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 29.559 s]
[INFO] Spark Project GraphX ............................... SUCCESS [ 35.464 s]
[INFO] Spark Project Streaming ............................ SUCCESS [01:01 min]
[INFO] Spark Project Catalyst ............................. SUCCESS [04:02 min]
[INFO] Spark Project SQL .................................. SUCCESS [03:49 min]
[INFO] Spark Project ML Library ........................... SUCCESS [02:45 min]
[INFO] Spark Project Tools ................................ SUCCESS [  6.600 s]
[INFO] Spark Project Hive ................................. SUCCESS [01:27 min]
[INFO] Spark Project REPL ................................. SUCCESS [ 18.706 s]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 30.993 s]
[INFO] Spark Project YARN ................................. SUCCESS [ 36.845 s]
[INFO] Spark Project Mesos ................................ SUCCESS [ 30.759 s]
[INFO] Spark Project Kubernetes ........................... SUCCESS [ 37.600 s]
[INFO] Spark Project Hive Thrift Server ................... SUCCESS [ 43.337 s]
[INFO] Spark Ganglia Integration .......................... SUCCESS [  7.996 s]
[INFO] Spark Project Hadoop Cloud Integration ............. SUCCESS [ 17.219 s]
[INFO] Spark Project Assembly ............................. SUCCESS [  5.834 s]
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [ 17.551 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 24.742 s]
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [ 38.248 s]
[INFO] Spark Kinesis Integration .......................... SUCCESS [ 27.528 s]
[INFO] Spark Project Examples ............................. SUCCESS [01:03 min]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [  9.940 s]
[INFO] Spark Avro ......................................... SUCCESS [ 33.881 s]
[INFO] Spark Project Connect .............................. SUCCESS [ 54.508 s]
[INFO] Spark Protobuf ..................................... SUCCESS [ 32.005 s]
[INFO] Spark Project Kinesis Assembly ..................... SUCCESS [ 13.541 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  26:48 min
[INFO] Finished at: 2022-12-08T13:07:58+08:00
[INFO] ------------------------------------------------------------------------

maven-build-java8-hadoop34-plugin472.log

[LuciferYang]

https://github.com/LuciferYang/make-distribution.sh/actions/runs/3645502265/jobs/6155706406

9706cf5 (with scala-maven-plugin 4.7.2) test build with Github Actions successful

[LuciferYang]

This pr can fix the compile issue in the dev mail list reported by @steveloughran, but should we wait until Hadoop 3.4 is upgraded?. What do you think? @HyukjinKwon @dongjoon-hyun @srowen

[dongjoon-hyun]

Since we don't want to depend on any SNAPSHOT dependencies, let's wait for the official Apache Hadoop 3.4.0, @LuciferYang .

[LuciferYang]

fine to me, close first ~

[steveloughran]

yeah, not going to happen for a while; 3.3.5 RC0 coming soon though; just trying to wrap up an abfs prefetch bug

[steveloughran]

time to revisit this; 3.4.0 is in RC phase