Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] non-UTC Nightly CI failed #10118

Closed
res-life opened this issue Dec 29, 2023 · 0 comments · Fixed by #10135
Closed

[BUG] non-UTC Nightly CI failed #10118

res-life opened this issue Dec 29, 2023 · 0 comments · Fixed by #10135
Assignees
Labels
bug Something isn't working

Comments

@res-life
Copy link
Collaborator

Describe the bug

jenkins/spark-tests.sh: line 276: jenkins/test-timezones.sh: No such file or directory

Steps/Code to reproduce bug
Refer to the last line of the below log

[2023-12-28T15:53:41.904Z] ----------------------------START TEST------------------------------------
[2023-12-28T15:53:41.904Z] + pushd /home/jenkins/agent/workspace/jenkins-rapids_it-non-utc-dev-22/jars/integration_tests/
[2023-12-28T15:53:41.904Z] /home/jenkins/agent/workspace/jenkins-rapids_it-non-utc-dev-22/jars/integration_tests /home/jenkins/agent/workspace/jenkins-rapids_it-non-utc-dev-22
[2023-12-28T15:53:41.904Z] + export TEST_TYPE=nightly
[2023-12-28T15:53:41.904Z] + TEST_TYPE=nightly
[2023-12-28T15:53:41.904Z] + export LOCAL_JAR_PATH=/home/jenkins/agent/workspace/jenkins-rapids_it-non-utc-dev-22/jars
[2023-12-28T15:53:41.904Z] + LOCAL_JAR_PATH=/home/jenkins/agent/workspace/jenkins-rapids_it-non-utc-dev-22/jars
[2023-12-28T15:53:41.904Z] + export SPARK_TASK_MAXFAILURES=1
[2023-12-28T15:53:41.904Z] + SPARK_TASK_MAXFAILURES=1
[2023-12-28T15:53:41.904Z] + export PATH=/home/jenkins/agent/workspace/jenkins-rapids_it-non-utc-dev-22/jars/spark-3.1.1-bin-hadoop3.2/bin:/home/jenkins/agent/workspace/jenkins-rapids_it-non-utc-dev-22/jars/spark-3.1.1-bin-hadoop3.2/sbin:/home/jenkins/agent/workspace/jenkins-rapids_it-non-utc-dev-22/jars/spark-3.1.1-bin-hadoop3.2/bin:/home/jenkins/agent/workspace/jenkins-rapids_it-non-utc-dev-22/jars/spark-3.1.1-bin-hadoop3.2/sbin:/opt/conda/bin:/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[2023-12-28T15:53:41.904Z] + PATH=/home/jenkins/agent/workspace/jenkins-rapids_it-non-utc-dev-22/jars/spark-3.1.1-bin-hadoop3.2/bin:/home/jenkins/agent/workspace/jenkins-rapids_it-non-utc-dev-22/jars/spark-3.1.1-bin-hadoop3.2/sbin:/home/jenkins/agent/workspace/jenkins-rapids_it-non-utc-dev-22/jars/spark-3.1.1-bin-hadoop3.2/bin:/home/jenkins/agent/workspace/jenkins-rapids_it-non-utc-dev-22/jars/spark-3.1.1-bin-hadoop3.2/sbin:/opt/conda/bin:/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[2023-12-28T15:53:41.905Z] + export 'SPARK_WORKER_OPTS= -Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.interval=120 -Dspark.worker.cleanup.appDataTtl=60'
[2023-12-28T15:53:41.905Z] + SPARK_WORKER_OPTS=' -Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.interval=120 -Dspark.worker.cleanup.appDataTtl=60'
[2023-12-28T15:53:41.905Z] + stop-worker.sh
[2023-12-28T15:53:42.160Z] no org.apache.spark.deploy.worker.Worker to stop
[2023-12-28T15:53:42.160Z] + stop-master.sh
[2023-12-28T15:53:42.160Z] no org.apache.spark.deploy.master.Master to stop
[2023-12-28T15:53:42.160Z] + start-master.sh
[2023-12-28T15:53:42.160Z] starting org.apache.spark.deploy.master.Master, logging to /home/jenkins/agent/workspace/jenkins-rapids_it-non-utc-dev-22/jars/spark-3.1.1-bin-hadoop3.2/logs/spark--org.apache.spark.deploy.master.Master-1-it-jenkins-rapids-it-non-utc-dev-22-2w9h7-qnkzm.out
[2023-12-28T15:53:44.674Z] + start-worker.sh spark://it-jenkins-rapids-it-non-utc-dev-22-2w9h7-qnkzm:7077
[2023-12-28T15:53:44.674Z] starting org.apache.spark.deploy.worker.Worker, logging to /home/jenkins/agent/workspace/jenkins-rapids_it-non-utc-dev-22/jars/spark-3.1.1-bin-hadoop3.2/logs/spark--org.apache.spark.deploy.worker.Worker-1-it-jenkins-rapids-it-non-utc-dev-22-2w9h7-qnkzm.out
[2023-12-28T15:53:47.192Z] + jps
[2023-12-28T15:53:47.192Z] 1248 Jps
[2023-12-28T15:53:47.192Z] 1086 Master
[2023-12-28T15:53:47.192Z] 1167 Worker
[2023-12-28T15:53:47.192Z] + export PYSP_TEST_spark_master=spark://it-jenkins-rapids-it-non-utc-dev-22-2w9h7-qnkzm:7077
[2023-12-28T15:53:47.192Z] + PYSP_TEST_spark_master=spark://it-jenkins-rapids-it-non-utc-dev-22-2w9h7-qnkzm:7077
[2023-12-28T15:53:47.192Z] + export PYSP_TEST_spark_sql_shuffle_partitions=12
[2023-12-28T15:53:47.192Z] + PYSP_TEST_spark_sql_shuffle_partitions=12
[2023-12-28T15:53:47.192Z] + export PYSP_TEST_spark_task_maxFailures=1
[2023-12-28T15:53:47.192Z] + PYSP_TEST_spark_task_maxFailures=1
[2023-12-28T15:53:47.192Z] + export PYSP_TEST_spark_dynamicAllocation_enabled=false
[2023-12-28T15:53:47.192Z] + PYSP_TEST_spark_dynamicAllocation_enabled=false
[2023-12-28T15:53:47.192Z] + export PYSP_TEST_spark_driver_extraJavaOptions=-Duser.timezone=UTC
[2023-12-28T15:53:47.192Z] + PYSP_TEST_spark_driver_extraJavaOptions=-Duser.timezone=UTC
[2023-12-28T15:53:47.192Z] + export PYSP_TEST_spark_executor_extraJavaOptions=-Duser.timezone=UTC
[2023-12-28T15:53:47.192Z] + PYSP_TEST_spark_executor_extraJavaOptions=-Duser.timezone=UTC
[2023-12-28T15:53:47.192Z] + export PYSP_TEST_spark_sql_session_timeZone=UTC
[2023-12-28T15:53:47.192Z] + PYSP_TEST_spark_sql_session_timeZone=UTC
[2023-12-28T15:53:47.192Z] + [[ true == \t\r\u\e ]]
[2023-12-28T15:53:47.192Z] + export PYSP_TEST_spark_cores_max=1
[2023-12-28T15:53:47.192Z] + PYSP_TEST_spark_cores_max=1
[2023-12-28T15:53:47.192Z] + export PYSP_TEST_spark_executor_memory=4g
[2023-12-28T15:53:47.192Z] + PYSP_TEST_spark_executor_memory=4g
[2023-12-28T15:53:47.192Z] + export PYSP_TEST_spark_executor_cores=1
[2023-12-28T15:53:47.192Z] + PYSP_TEST_spark_executor_cores=1
[2023-12-28T15:53:47.192Z] + export PYSP_TEST_spark_task_cores=1
[2023-12-28T15:53:47.192Z] + PYSP_TEST_spark_task_cores=1
[2023-12-28T15:53:47.192Z] + export PYSP_TEST_spark_rapids_sql_concurrentGpuTasks=1
[2023-12-28T15:53:47.192Z] + PYSP_TEST_spark_rapids_sql_concurrentGpuTasks=1
[2023-12-28T15:53:47.192Z] + export PYSP_TEST_spark_rapids_memory_gpu_minAllocFraction=0
[2023-12-28T15:53:47.192Z] + PYSP_TEST_spark_rapids_memory_gpu_minAllocFraction=0
[2023-12-28T15:53:47.192Z] + [[ '' == '' ]]
[2023-12-28T15:53:47.192Z] ++ nvidia-smi --query-gpu=memory.free --format=csv,noheader
[2023-12-28T15:53:47.192Z] ++ awk '{if (MAX < $1){ MAX = $1}} END {print int(MAX / (2 * 1024))}'
[2023-12-28T15:53:47.192Z] + PARALLELISM=11
[2023-12-28T15:53:47.192Z] + [[ 11 -gt 5 ]]
[2023-12-28T15:53:47.192Z] + PARALLELISM=5
[2023-12-28T15:53:47.192Z] ++ python -c 'print(1/(5 + 0.1))'
[2023-12-28T15:53:47.192Z] + MEMORY_FRACTION=0.19607843137254904
[2023-12-28T15:53:47.192Z] + export TEST_PARALLEL=5
[2023-12-28T15:53:47.192Z] + TEST_PARALLEL=5
[2023-12-28T15:53:47.192Z] + export PYSP_TEST_spark_rapids_memory_gpu_allocFraction=0.19607843137254904
[2023-12-28T15:53:47.192Z] + PYSP_TEST_spark_rapids_memory_gpu_allocFraction=0.19607843137254904
[2023-12-28T15:53:47.193Z] + export PYSP_TEST_spark_rapids_memory_gpu_maxAllocFraction=0.19607843137254904
[2023-12-28T15:53:47.193Z] + PYSP_TEST_spark_rapids_memory_gpu_maxAllocFraction=0.19607843137254904
[2023-12-28T15:53:47.193Z] ++ pwd -P
[2023-12-28T15:53:47.193Z] + export SCRIPT_PATH=/home/jenkins/agent/workspace/jenkins-rapids_it-non-utc-dev-22/jars/integration_tests
[2023-12-28T15:53:47.193Z] + SCRIPT_PATH=/home/jenkins/agent/workspace/jenkins-rapids_it-non-utc-dev-22/jars/integration_tests
[2023-12-28T15:53:47.193Z] + export TARGET_DIR=/home/jenkins/agent/workspace/jenkins-rapids_it-non-utc-dev-22/jars/integration_tests/target
[2023-12-28T15:53:47.193Z] + TARGET_DIR=/home/jenkins/agent/workspace/jenkins-rapids_it-non-utc-dev-22/jars/integration_tests/target
[2023-12-28T15:53:47.193Z] + mkdir -p /home/jenkins/agent/workspace/jenkins-rapids_it-non-utc-dev-22/jars/integration_tests/target
[2023-12-28T15:53:47.193Z] + TEST_MODE=NON_UTC_TZ
[2023-12-28T15:53:47.193Z] + [[ NON_UTC_TZ == \D\E\F\A\U\L\T ]]
[2023-12-28T15:53:47.193Z] + [[ NON_UTC_TZ == \D\E\F\A\U\L\T ]]
[2023-12-28T15:53:47.193Z] + [[ NON_UTC_TZ == \D\E\L\T\A\_\L\A\K\E\_\O\N\L\Y ]]
[2023-12-28T15:53:47.193Z] + [[ NON_UTC_TZ == \D\E\F\A\U\L\T ]]
[2023-12-28T15:53:47.193Z] + [[ NON_UTC_TZ == \I\C\E\B\E\R\G\_\O\N\L\Y ]]
[2023-12-28T15:53:47.193Z] + [[ NON_UTC_TZ == \D\E\F\A\U\L\T ]]
[2023-12-28T15:53:47.193Z] + [[ NON_UTC_TZ == \A\V\R\O\_\O\N\L\Y ]]
[2023-12-28T15:53:47.193Z] + [[ NON_UTC_TZ == \D\E\F\A\U\L\T ]]
[2023-12-28T15:53:47.193Z] + [[ NON_UTC_TZ == \M\U\L\T\I\T\H\R\E\A\D\E\D\_\S\H\U\F\F\L\E ]]
[2023-12-28T15:53:47.193Z] + [[ NON_UTC_TZ == \C\U\D\F\_\U\D\F\_\O\N\L\Y ]]
[2023-12-28T15:53:47.193Z] + [[ NON_UTC_TZ == \D\E\F\A\U\L\T ]]
[2023-12-28T15:53:47.193Z] + [[ NON_UTC_TZ == \P\Y\A\R\R\O\W\_\O\N\L\Y ]]
[2023-12-28T15:53:47.193Z] + [[ NON_UTC_TZ == \N\O\N\_\U\T\C\_\T\Z ]]
[2023-12-28T15:53:47.193Z] + run_non_utc_time_zone_tests
[2023-12-28T15:53:47.193Z] ++ dirname jenkins/spark-tests.sh
[2023-12-28T15:53:47.193Z] + source jenkins/test-timezones.sh
[2023-12-28T15:53:47.193Z] jenkins/spark-tests.sh: line 276: jenkins/test-timezones.sh: No such file or directory
script returned exit code 1
@res-life res-life added bug Something isn't working ? - Needs Triage Need team to review and classify labels Dec 29, 2023
@res-life res-life self-assigned this Jan 2, 2024
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Jan 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants