Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix typo in spark-tests.sh [skip ci] #10328

Merged
merged 1 commit into from
Jan 30, 2024
Merged

Conversation

NvTimLiu
Copy link
Collaborator

Fix typo in spark-tests.sh for '--packages' test

Follow up: https://github.com/NVIDIA/spark-rapids/pull/10279/files#r1471168516

Signed-off-by: Tim Liu timl@nvidia.com

Fix typo in spark-tests.sh for '--packages' test

Follow up: https://github.com/NVIDIA/spark-rapids/pull/10279/files#r1471168516

Signed-off-by: Tim Liu <timl@nvidia.com>
@NvTimLiu NvTimLiu added the build Related to CI / CD or cleanly building label Jan 30, 2024
@NvTimLiu NvTimLiu self-assigned this Jan 30, 2024
@NvTimLiu
Copy link
Collaborator Author

NvTimLiu commented Jan 30, 2024

Sorry I had a typo error when filing the PR10279

Test script with below env, got PASS or SKIP

  if [[ "$CLASSIFIER" == "" || "$CLASSIFIER" == "cuda11" ]]; then
    SPARK_SHELL_SMOKE_TEST=1 \
    PYSP_TEST_spark_jars_packages=com.nvidia:rapids-4-spark_${SCALA_BINARY_VER}:${PROJECT_VER} \
    PYSP_TEST_spark_jars_repositories=${PROJECT_REPO} \
      ./run_pyspark_from_build.sh
  fi

CLASSIFIER='cuda11' or CLASSIFIER='', scala2.12

+ SPARK_SHELL_ARGS_ARR+=(--packages "${PYSP_TEST_spark_jars_packages}")
+ [[ -n $ART_URL ]]
+ SPARK_SHELL_ARGS_ARR+=(--repositories "${PYSP_TEST_spark_jars_repositories}")
+ spark-3.3.0-bin-hadoop3/bin/spark-shell --conf spark.plugins=com.nvidia.spark.SQLPlugin --conf spark.deploy.maxExecutorRetries=0 --packages com.nvidia:rapids-4-spark_2.12:24.02.0-SNAPSHOT --repositories $ART_URL
+ grep -F 'res0: Array[org.apache.spark.sql.Row] = Array([4950])'
res0: Array[org.apache.spark.sql.Row] = Array([4950])
+ echo 'SUCCESS spark-shell smoke test'
SUCCESS spark-shell smoke test

CLASSIFIER='cuda12' or CLASSIFIER='cuda11-arm64', scala2.12

SKIP

CLASSIFIER='cuda11' or CLASSIFIER='', scala2.13

+ SPARK_SHELL_ARGS_ARR+=(--packages "${PYSP_TEST_spark_jars_packages}")
+ [[ -n $ART_URL ]]
+ SPARK_SHELL_ARGS_ARR+=(--repositories "${PYSP_TEST_spark_jars_repositories}")
+ spark-3.3.0-bin-hadoop3-scala2.13/bin/spark-shell --conf spark.plugins=com.nvidia.spark.SQLPlugin --conf spark.deploy.maxExecutorRetries=0 --packages com.nvidia:rapids-4-spark_2.13:24.02.0-SNAPSHOT --repositories $ART_URL
+ grep -F 'res0: Array[org.apache.spark.sql.Row] = Array([4950])'
scala> spark.range(100).agg(Map("id" -> "sum")).collect()val res0: Array[org.apache.spark.sql.Row] = Array([4950])
+ echo 'SUCCESS spark-shell smoke test'

CLASSIFIER='cuda12' or CLASSIFIER='cuda11-arm64', scala2.13

SKIP

@NvTimLiu
Copy link
Collaborator Author

Skip CI as this script will not be executed in pre-merge CI, it is only be called in integration test CI

@NvTimLiu
Copy link
Collaborator Author

build

@jlowe jlowe merged commit 252111a into NVIDIA:branch-24.02 Jan 30, 2024
41 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build Related to CI / CD or cleanly building
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants