[VL] Optimize GHA workflow with nightly built docker image (part-1) #4293

PZD-CHINA · 2024-01-05T07:37:07Z

The patch targets to do CI cleanup.

.github/workflows/velox_be.yml static-build-test and build-script-test need further debug.

github-actions · 2024-01-05T07:37:24Z

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/oap-project/gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

Other pull requests

github-actions · 2024-01-05T07:37:38Z

Run Gluten Clickhouse CI

github-actions · 2024-01-05T07:52:59Z

Run Gluten Clickhouse CI

.github/workflows/velox_be.yml

github-actions · 2024-01-05T07:59:07Z

Run Gluten Clickhouse CI

github-actions · 2024-01-05T07:59:35Z

Run Gluten Clickhouse CI

github-actions · 2024-01-05T08:00:31Z

Run Gluten Clickhouse CI

github-actions · 2024-01-05T08:00:55Z

Run Gluten Clickhouse CI

tools/gluten-te/centos/buildenv.sh

tools/gluten-te/centos/defaults.conf

.github/workflows/velox_be.yml

github-actions · 2024-01-05T08:37:10Z

Run Gluten Clickhouse CI

github-actions · 2024-01-05T08:37:30Z

Run Gluten Clickhouse CI

github-actions · 2024-01-05T08:39:50Z

Run Gluten Clickhouse CI

github-actions · 2024-01-05T08:42:04Z

Run Gluten Clickhouse CI

PHILO-HE · 2024-01-05T08:48:31Z

Please replace /opt/spark331 by /opt/spark342 in the below two pieces of code.
Using spark 331 binary for spark 342 tests has some potential issues. In the new docker, suppose spark-342 source is downloaded and then built. See code link.

https://github.com/oap-project/gluten/blob/29f850086609d8922b30c163ea1687fbd25a2e8d/.github/workflows/velox_be.yml#L238

https://github.com/oap-project/gluten/blob/29f850086609d8922b30c163ea1687fbd25a2e8d/.github/workflows/velox_be.yml#L276

github-actions · 2024-01-05T08:49:53Z

Run Gluten Clickhouse CI

github-actions · 2024-01-05T08:51:14Z

Run Gluten Clickhouse CI

github-actions · 2024-01-05T08:54:12Z

Run Gluten Clickhouse CI

github-actions · 2024-01-05T09:05:11Z

Run Gluten Clickhouse CI

zhouyuan · 2024-01-05T10:46:09Z

.github/workflows/velox_be.yml

  DOCKER_REGISTRY_PORT: 5000
-
+  PATH_TO_GLUTEN_TE: ./tools/gluten-te


I think we should be able to use the Github environment variables

Hongze once removed the use of github environment in a revert commit: 385b252. I will check this with him.

github-actions · 2024-01-05T10:46:49Z

Run Gluten Clickhouse CI

github-actions · 2024-01-05T10:48:36Z

Run Gluten Clickhouse CI

github-actions · 2024-01-06T12:14:23Z

Run Gluten Clickhouse CI

github-actions · 2024-01-06T12:14:58Z

Run Gluten Clickhouse CI

zhouyuan · 2024-01-08T00:24:48Z

please help to do a rebase, the CK backend failure is fixed in main branch

github-actions · 2024-01-09T08:18:42Z

Run Gluten Clickhouse CI

ayushi-agarwal · 2024-01-09T09:14:13Z

@PZD-CHINA I'm not sure why these changes caused the failure in spark 34. Could you please disable these two failed unit tests and added them in this list for spark34 failed unit tests #3559? Thanks.

@JkSelf This PR will solve the issue #4203. As part of this PR #4273 we found that these were failing in our local when we give correct path and many other UT's pass when given the correct path.

revert DEFAULT_DOCKER_CACHE_REGISTRY and DEFAULT_DOCKER_PUSH_REGISTRY set. set DOCKER_CACHE_REGISTRY and DOCKER_PUSH_REGISTRY in velxo_be.yml update velxo_be.yml build-script-test job

github-actions · 2024-01-10T00:39:04Z

Run Gluten Clickhouse CI

github-actions · 2024-01-10T01:55:51Z

Run Gluten Clickhouse CI

github-actions · 2024-01-10T01:57:30Z

Run Gluten Clickhouse CI

github-actions · 2024-01-10T01:58:41Z

Run Gluten Clickhouse CI

github-actions · 2024-01-10T01:59:54Z

Run Gluten Clickhouse CI

github-actions · 2024-01-10T02:49:02Z

Run Gluten Clickhouse CI

PHILO-HE · 2024-01-10T03:28:39Z

Let's leave the change for yml in a follow-up PR and then really use the new docker after that.

PHILO-HE

Thanks for your work!

GlutenPerfBot · 2024-01-10T04:22:11Z

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query	log/native_4293_time.csv	log/native_master_01_09_2024_d3df97aa0_time.csv	difference	percentage
q1	33.60	33.38	-0.225	99.33%
q2	25.39	25.66	0.271	101.07%
q3	38.36	37.88	-0.479	98.75%
q4	39.32	39.11	-0.209	99.47%
q5	73.23	71.57	-1.652	97.74%
q6	6.65	6.96	0.309	104.65%
q7	85.38	86.82	1.441	101.69%
q8	86.09	86.62	0.530	100.62%
q9	126.71	125.62	-1.088	99.14%
q10	43.79	42.71	-1.082	97.53%
q11	20.12	20.21	0.086	100.43%
q12	27.36	29.40	2.047	107.48%
q13	47.13	47.13	0.002	100.00%
q14	19.52	16.36	-3.158	83.82%
q15	29.13	27.94	-1.186	95.93%
q16	14.44	14.99	0.550	103.81%
q17	104.29	156.46	52.173	150.03%
q18	150.89	194.42	43.528	128.85%
q19	12.54	16.64	4.101	132.71%
q20	27.49	28.42	0.923	103.36%
q21	227.92	225.07	-2.850	98.75%
q22	13.84	14.11	0.271	101.96%
total	1253.19	1347.49	94.301	107.52%

vibhaska · 2024-01-10T14:28:01Z

Please replace /opt/spark331 by /opt/spark342 in the below two pieces of code. Using spark 331 binary for spark 342 tests has some potential issues. In the new docker, suppose spark-342 source is downloaded and then built. See code link.

https://github.com/oap-project/gluten/blob/29f850086609d8922b30c163ea1687fbd25a2e8d/.github/workflows/velox_be.yml#L238

https://github.com/oap-project/gluten/blob/29f850086609d8922b30c163ea1687fbd25a2e8d/.github/workflows/velox_be.yml#L276

Has this been done already?

vibhaska · 2024-01-10T14:45:35Z

Please replace /opt/spark331 by /opt/spark342 in the below two pieces of code. Using spark 331 binary for spark 342 tests has some potential issues. In the new docker, suppose spark-342 source is downloaded and then built. See code link.
https://github.com/oap-project/gluten/blob/29f850086609d8922b30c163ea1687fbd25a2e8d/.github/workflows/velox_be.yml#L238

https://github.com/oap-project/gluten/blob/29f850086609d8922b30c163ea1687fbd25a2e8d/.github/workflows/velox_be.yml#L276

Has this been done already?

@JkSelf Possibly missed in checkin because its a ".github" folder, ignored by git.

PHILO-HE · 2024-01-10T15:12:05Z

Hi @vibhaska, the spark-3.4 test home issue will be fixed by another PR: https://github.com/oap-project/gluten/pull/4362/files. Will merge that patch after CI passes.

PHILO-HE changed the title ~~[VL] GHA CI cleanup~~ [VL] Optimize GHA workflow with nightly built docker image Jan 5, 2024

PHILO-HE reviewed Jan 5, 2024

View reviewed changes

.github/workflows/velox_be.yml Outdated Show resolved Hide resolved

PHILO-HE reviewed Jan 5, 2024

View reviewed changes

tools/gluten-te/centos/buildenv.sh Show resolved Hide resolved

PHILO-HE reviewed Jan 5, 2024

View reviewed changes

tools/gluten-te/centos/defaults.conf Outdated Show resolved Hide resolved

PHILO-HE reviewed Jan 5, 2024

View reviewed changes

.github/workflows/velox_be.yml Outdated Show resolved Hide resolved

PHILO-HE force-pushed the GHA-workflow branch from d9d978e to 42622eb Compare January 5, 2024 08:36

zhouyuan reviewed Jan 5, 2024

View reviewed changes

PHILO-HE force-pushed the GHA-workflow branch from 5f10f0f to c9fb89b Compare January 6, 2024 12:14

zhouyuan mentioned this pull request Jan 8, 2024

[VL] Turn on abfs connector in CI #4304

Merged

PHILO-HE and others added 6 commits January 10, 2024 08:37

Initial commit

b0ffb36

Ignore two spark-3.4 tests

0c44685

set registry node. update build-script-test

499c48d

revert DEFAULT_DOCKER_CACHE_REGISTRY and DEFAULT_DOCKER_PUSH_REGISTRY set. set DOCKER_CACHE_REGISTRY and DOCKER_PUSH_REGISTRY in velxo_be.yml update velxo_be.yml build-script-test job

Update velox_be.yml

01e3992

Update velox_be.yml

8b057d3

Update velox_be.yml

6f23a18

PHILO-HE force-pushed the GHA-workflow branch from b974092 to 6f23a18 Compare January 10, 2024 00:38

更新 defaults.conf

7d17edc

更新 build.sh

8a7e812

更新 defaults.conf

5aa2406

更新 defaults.conf

11148bf

Update velox_be.yml

2d826b1

PHILO-HE approved these changes Jan 10, 2024

View reviewed changes

PHILO-HE changed the title ~~[VL] Optimize GHA workflow with nightly built docker image~~ [VL] Optimize GHA workflow with nightly built docker image (part-1) Jan 10, 2024

PHILO-HE merged commit f80408c into apache:main Jan 10, 2024
15 of 20 checks passed

vibhaska mentioned this pull request Jan 10, 2024

[GLUTEN-3559][VL] Fix unit tests in GlutenSQLQueryTestSuite suite - Part 1 #4273

Closed

PZD-CHINA deleted the GHA-workflow branch January 16, 2024 05:32

		DOCKER_REGISTRY_PORT: 5000

		PATH_TO_GLUTEN_TE: ./tools/gluten-te

[VL] Optimize GHA workflow with nightly built docker image (part-1) #4293

[VL] Optimize GHA workflow with nightly built docker image (part-1) #4293

Conversation

PZD-CHINA commented Jan 5, 2024

github-actions bot commented Jan 5, 2024

github-actions bot commented Jan 5, 2024

github-actions bot commented Jan 5, 2024

github-actions bot commented Jan 5, 2024

github-actions bot commented Jan 5, 2024

github-actions bot commented Jan 5, 2024

github-actions bot commented Jan 5, 2024

github-actions bot commented Jan 5, 2024

github-actions bot commented Jan 5, 2024

github-actions bot commented Jan 5, 2024

github-actions bot commented Jan 5, 2024

PHILO-HE commented Jan 5, 2024

github-actions bot commented Jan 5, 2024

github-actions bot commented Jan 5, 2024

github-actions bot commented Jan 5, 2024

github-actions bot commented Jan 5, 2024

zhouyuan Jan 5, 2024

Choose a reason for hiding this comment

PHILO-HE Jan 5, 2024

Choose a reason for hiding this comment

github-actions bot commented Jan 5, 2024

github-actions bot commented Jan 5, 2024

github-actions bot commented Jan 6, 2024

github-actions bot commented Jan 6, 2024

zhouyuan commented Jan 8, 2024

github-actions bot commented Jan 9, 2024

ayushi-agarwal commented Jan 9, 2024

github-actions bot commented Jan 10, 2024

github-actions bot commented Jan 10, 2024

github-actions bot commented Jan 10, 2024

github-actions bot commented Jan 10, 2024

github-actions bot commented Jan 10, 2024

github-actions bot commented Jan 10, 2024

PHILO-HE commented Jan 10, 2024

PHILO-HE left a comment

Choose a reason for hiding this comment

GlutenPerfBot commented Jan 10, 2024

vibhaska commented Jan 10, 2024

vibhaska commented Jan 10, 2024

PHILO-HE commented Jan 10, 2024