[GLUTEN-4398][IT] Add Golden Files for TPC-H Spark32 + Gluten Execution Plan #4432

zwangsheng · 2024-01-17T12:25:40Z

What changes were proposed in this pull request?

Add tpc-h golden file for spark32 and check whether PR changed Spark Execution Plan

Continue worker with #4399

Add golden files for tpc-h(except delta related)
for now only work for spark32, other spark version will be made later.
Each PR will check those golden files though Velox Backend Unit Test, and will upload actual golden files to github action if some query mis-match.

Users can download and replace the local corresponding file for quick correction.
Golden files located at ${GLUTEN_REPO_HOME}/backends-velox/src/test/resources/tpch-approved-plan/${TPC_H_TEST_SUB_TYPE}/${SPARK_VERSION}/${QUERY_ID}.txt,
for example:
${GLUTEN_REPO_HOME}/backends-velox/src/test/resources/tpch-approved-plan/gluten-bhj-vanilla-be/spark322/1.txt.

And when user run unit test in local, them can find actual golden file in tmp dir, find this with unit test output log.

- TPC-H q5 *** FAILED ***
  Mismatch for query 5
  Actual Plan path: /tmp/tpch-approved-plan/v2-bhj/spark322/5.txt
  Golden Plan path: /opt/gluten/backends-velox/target/scala-2.12/test-classes/tpch-approved-plan/v2-bhj/spark322/5.txt (VeloxTPCHSuite.scala:101)

How was this patch tested?

Add Unit Test for Spark32 to check golden files

github-actions · 2024-01-17T12:25:59Z

#4398

github-actions · 2024-01-17T12:26:12Z

Run Gluten Clickhouse CI

github-actions · 2024-01-17T12:26:35Z

Run Gluten Clickhouse CI

github-actions · 2024-01-17T12:29:09Z

Run Gluten Clickhouse CI

github-actions · 2024-01-17T12:30:55Z

Run Gluten Clickhouse CI

github-actions · 2024-01-18T03:03:39Z

Run Gluten Clickhouse CI

github-actions · 2024-01-18T03:07:31Z

Run Gluten Clickhouse CI

github-actions · 2024-01-18T03:29:53Z

Run Gluten Clickhouse CI

zhztheplayer · 2024-01-18T05:07:12Z

@zwangsheng Thanks in advance.

Would you like to introduce a bit about the term "golden file"? Thanks.

If I understand it correctly, when execution plan for a query gets changed, developer should regenerate the files using gluten-it, is that correct?

If yes, I am worried that this way puts extra burden to developers since compiling gluten-it takes quite some time as it packages everything. Did you consider adding this check to Gluten's main code?

After all, would you like to share a bit about the background why you wanted to add this check to Gluten (and its CI)? Thanks.

zwangsheng · 2024-01-18T05:53:44Z

Thanks for noticing this feature. @zhztheplayer

Would you like to introduce a bit about the term "golden file"? Thanks.

Base on Spark SQL Module has PlanStabilitySuite to sense how code changes (from PR) change the execution plan.

Gluten is closely related to spark execution plans, and Golden File should be introduced for gluten to ensure that code changes to the execution plan are the first to be detected, so that we can better evaluate code changes.

If I understand it correctly, when execution plan for a query gets changed, developer should regenerate the files using gluten-it, is that correct?

Yes, if devs make same change to related code, the Golden Files should be re-generate by gluten-it module

If yes, I am worried that this way puts extra burden to developers since compiling gluten-it takes quite some time as it packages everything.

At present, it seems that the process of user contribution PR will be affected, and maybe we can make some changes based on this PR to reduce the compilation cost of the user side(in the process of CI detection, we can upload the changed Golden File to the log store section of Github Action, and users can download it instead of compiling it locally, that seems to allay the concerns you mentioned)

But in general, we still should have such a Golden File mechanism in case we miss the relevant changes.

Thanks again, if have new ideas to help us check spark execution plan changes, be open to them.

github-actions · 2024-01-18T08:22:55Z

Run Gluten Clickhouse CI

github-actions · 2024-01-18T10:59:20Z

Run Gluten Clickhouse CI

github-actions · 2024-01-18T12:11:41Z

Run Gluten Clickhouse CI

github-actions · 2024-01-19T01:58:44Z

Run Gluten Clickhouse CI

zhztheplayer · 2024-01-19T02:25:55Z

Thanks for the detailed explanation @zwangsheng.

At present, it seems that the process of user contribution PR will be affected, and maybe we can make some changes based on this PR to reduce the compilation cost of the user side(in the process of CI detection, we can upload the changed Golden File to the log store section of Github Action, and users can download it instead of compiling it locally, that seems to allay the concerns you mentioned)

I still feel the procedure is probably a little too complicated for developers. Gluten has in-place TPC-H unit test code here

https://github.com/oap-project/gluten/blob/main/backends-velox/src/test/scala/io/glutenproject/execution/VeloxTPCHSuite.scala

So if possible I might be inclined to have the golden checking integrated with current unit tests rather than integration testing. Then developers could avoid re-building the whole project to change golden files when plan gets changed. I understand it might require for a bunch of changes to this PR but let's see if it is worth it.

I was considering bring some gluten-it's features into unit tests but that may mess up the responsibility between ut and it. I am not sure. But currently the cost of using gluten-it to manually verify code changes is high.

A simple comparison between using it and ut:

it:
Developer change code -> Commits code change -> Run GHA CI -> Download golden files -> Replace local golden files -> Rerun GHA CI

ut (probably):
Developer change code -> Run UT to generate new golden files -> Replace local golden files -> Commit code change -> Run GHA CI

I assume the ut solution should take much shorter time than using it?

zwangsheng · 2024-01-19T10:53:41Z

ut (probably):
Developer change code -> Run UT to generate new golden files -> Replace local golden files -> Commit code change -> Run GHA CI

Thanks, i will give a try to move those golden files test to unit test.

zhztheplayer · 2024-01-22T01:36:38Z

ut (probably):
Developer change code -> Run UT to generate new golden files -> Replace local golden files -> Commit code change -> Run GHA CI

Thanks, i will give a try to move those golden files test to unit test.

Thank you!

github-actions · 2024-01-23T12:13:54Z

Run Gluten Clickhouse CI

github-actions · 2024-01-23T12:18:47Z

Run Gluten Clickhouse CI

github-actions · 2024-01-24T03:36:34Z

Run Gluten Clickhouse CI

github-actions · 2024-01-24T05:37:41Z

Run Gluten Clickhouse CI

github-actions · 2024-01-24T06:13:12Z

Run Gluten Clickhouse CI

github-actions · 2024-01-24T06:54:04Z

Run Gluten Clickhouse CI

github-actions · 2024-01-24T07:08:54Z

Run Gluten Clickhouse CI

github-actions · 2024-01-24T08:39:03Z

Run Gluten Clickhouse CI

github-actions · 2024-02-05T13:22:39Z

Run Gluten Clickhouse CI

github-actions · 2024-02-23T07:16:20Z

Run Gluten Clickhouse CI

github-actions · 2024-02-23T08:07:24Z

Run Gluten Clickhouse CI

zhztheplayer · 2024-02-26T02:36:28Z

tools/gluten-te/centos/gha/gha-checkout/cp.sh

+if [ -z "$GITHUB_RUN_ID" ]
+then
+  echo "Unable to parse GITHUB_RUN_ID."
+  exit 1
+fi


Should we check $GITHUB_JOB too?

zhztheplayer

Overall looks good. Thanks @zwangsheng !

ulysses-you

it seems overkill to generate golden files for each type

gluten-bhj-vanilla-be
vanilla-bhj-gluten-be
partitioned
v1
v1-bhj
v2
v2-bhj

Can we just maintain v1 and v1-bhj ?

zwangsheng · 2024-02-26T03:47:09Z

it seems overkill to generate golden files for each type

gluten-bhj-vanilla-be

vanilla-bhj-gluten-be

partitioned

v1

v1-bhj

v2

v2-bhj

Can we just maintain v1 and v1-bhj ?

Reducing type seems feasible, but considering Spark retains v2 related, should we also retain v2 related?

ulysses-you · 2024-02-26T03:53:08Z

I think no body would use v2 code path to read parquet.. it is mainly used for data lake.

zhztheplayer · 2024-02-26T04:30:43Z

Another question: After this patch, is there any best practice to re-generate all the golden files through a single command?

- TPC-H q5 *** FAILED ***
 Mismatch for query 5
 Actual Plan path: /tmp/tpch-approved-plan/v2-bhj/spark322/5.txt
 Golden Plan path: /opt/gluten/backends-velox/target/scala-2.12/test-classes/tpch-approved-plan/v2-bhj/spark322/5.txt (VeloxTPCHSuite.scala:101)

The example is good but it seems developer should fix the files one by one?

zwangsheng · 2024-02-26T07:10:31Z

Another question: After this patch, is there any best practice to re-generate all the golden files through a single command?
- TPC-H q5 *** FAILED ***
 Mismatch for query 5
 Actual Plan path: /tmp/tpch-approved-plan/v2-bhj/spark322/5.txt
 Golden Plan path: /opt/gluten/backends-velox/target/scala-2.12/test-classes/tpch-approved-plan/v2-bhj/spark322/5.txt (VeloxTPCHSuite.scala:101)
The example is good but it seems developer should fix the files one by one?

Nope, github action will cache those actual golden files in tgz mode, so users can download and replace all in one click.

github-actions · 2024-02-27T02:06:14Z

Run Gluten Clickhouse CI

github-actions · 2024-02-27T02:44:32Z

Run Gluten Clickhouse CI

github-actions · 2024-02-27T03:39:26Z

Run Gluten Clickhouse CI

ulysses-you · 2024-02-27T03:54:35Z

lgtm, please add a step by step docs to help contributors check and update golden files.

github-actions · 2024-02-27T05:47:08Z

Run Gluten Clickhouse CI

github-actions · 2024-02-27T08:12:51Z

Run Gluten Clickhouse CI

zwangsheng · 2024-02-27T08:14:12Z

Hi @ulysses-you @zhztheplayer Golden files related introduction was added in NewToGluten.md, please help to correct it

ulysses-you · 2024-02-27T08:54:57Z

docs/developers/NewToGluten.md

+  Actual Plan path: /tmp/tpch-approved-plan/v2-bhj/spark322/5.txt
+  Golden Plan path: /opt/gluten/backends-velox/target/scala-2.12/test-classes/tpch-approved-plan/v2-bhj/spark322/5.txt (VeloxTPCHSuite.scala:101)
+```
+For developers to update the golden plan, you can find the actual plan in Github CI Artifacts or in local `/tmp/` directory. 


can we add a .png to make it clear ?

github-actions · 2024-02-28T10:41:49Z

Run Gluten Clickhouse CI

ulysses-you

lgtm, cc @zhztheplayer @PHILO-HE if you have other comments

zhztheplayer

Thanks for your contribution!

GlutenPerfBot · 2024-02-29T02:00:55Z

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query	log/native_4432_time.csv	log/native_master_02_27_2024_94c8e7e0e_time.csv	difference	percentage
q1	32.86	33.59	0.729	102.22%
q2	24.58	24.40	-0.186	99.25%
q3	37.48	38.91	1.426	103.81%
q4	36.89	36.13	-0.755	97.95%
q5	70.37	70.98	0.615	100.87%
q6	5.51	7.08	1.572	128.51%
q7	84.79	84.44	-0.347	99.59%
q8	84.63	86.39	1.753	102.07%
q9	122.91	121.48	-1.432	98.84%
q10	43.11	44.06	0.955	102.22%
q11	20.57	19.96	-0.611	97.03%
q12	29.38	29.43	0.057	100.19%
q13	44.42	45.22	0.798	101.80%
q14	19.15	16.33	-2.820	85.27%
q15	28.84	28.90	0.056	100.19%
q16	16.22	15.96	-0.252	98.44%
q17	102.65	103.10	0.455	100.44%
q18	149.35	148.22	-1.134	99.24%
q19	12.63	12.53	-0.098	99.22%
q20	26.36	26.71	0.349	101.32%
q21	227.75	224.21	-3.548	98.44%
q22	13.67	13.66	-0.006	99.96%
total	1234.13	1231.71	-2.423	99.80%

zwangsheng changed the title ~~[GLUTEN-4398][IT] Add TPC-H Spark + Gluten Execution Plan Golden File~~ [GLUTEN-4398][IT] Add Golden Files for TPC-H Spark + Gluten Execution Plan Jan 18, 2024

zwangsheng marked this pull request as draft January 19, 2024 10:52

zwangsheng changed the title ~~[GLUTEN-4398][IT] Add Golden Files for TPC-H Spark + Gluten Execution Plan~~ [GLUTEN-4398][IT] Add Golden Files for TPC-H Spark32 + Gluten Execution Plan Feb 5, 2024

zwangsheng force-pushed the GLUTEN#4398 branch from 3973536 to b648a09 Compare February 5, 2024 13:22

reach lastest

ff48876

zwangsheng marked this pull request as ready for review February 23, 2024 09:57

zhztheplayer reviewed Feb 26, 2024

View reviewed changes

ulysses-you reviewed Feb 26, 2024

View reviewed changes

reduce subtype

8c3a2cb

fix q17

04140c3

mark querystageexec id

da789d3

fix golden files

d1ec4c2

Add Golden Files Guide in NewToGluten.md

105cd79

ulysses-you reviewed Feb 27, 2024

View reviewed changes

upload screen shoot

22a012d

ulysses-you approved these changes Feb 28, 2024

View reviewed changes

zhztheplayer approved these changes Feb 29, 2024

View reviewed changes

ulysses-you merged commit 7af6d0c into apache:main Feb 29, 2024
19 checks passed

[GLUTEN-4398][IT] Add Golden Files for TPC-H Spark32 + Gluten Execution Plan #4432

[GLUTEN-4398][IT] Add Golden Files for TPC-H Spark32 + Gluten Execution Plan #4432

Conversation

zwangsheng commented Jan 17, 2024 • edited Loading

What changes were proposed in this pull request?

How was this patch tested?

github-actions bot commented Jan 17, 2024

github-actions bot commented Jan 17, 2024

github-actions bot commented Jan 17, 2024

github-actions bot commented Jan 17, 2024

github-actions bot commented Jan 17, 2024

github-actions bot commented Jan 18, 2024

github-actions bot commented Jan 18, 2024

github-actions bot commented Jan 18, 2024

zhztheplayer commented Jan 18, 2024 • edited Loading

zwangsheng commented Jan 18, 2024

github-actions bot commented Jan 18, 2024

github-actions bot commented Jan 18, 2024

github-actions bot commented Jan 18, 2024

github-actions bot commented Jan 19, 2024

zhztheplayer commented Jan 19, 2024 • edited Loading

zwangsheng commented Jan 19, 2024

zhztheplayer commented Jan 22, 2024

github-actions bot commented Jan 23, 2024

github-actions bot commented Jan 23, 2024

github-actions bot commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

github-actions bot commented Jan 24, 2024

github-actions bot commented Feb 5, 2024

github-actions bot commented Feb 23, 2024

github-actions bot commented Feb 23, 2024

zhztheplayer Feb 26, 2024

Choose a reason for hiding this comment

zhztheplayer left a comment

Choose a reason for hiding this comment

ulysses-you left a comment

Choose a reason for hiding this comment

zwangsheng commented Feb 26, 2024

ulysses-you commented Feb 26, 2024

zhztheplayer commented Feb 26, 2024

zwangsheng commented Feb 26, 2024 • edited Loading

github-actions bot commented Feb 27, 2024

github-actions bot commented Feb 27, 2024

github-actions bot commented Feb 27, 2024

ulysses-you commented Feb 27, 2024

github-actions bot commented Feb 27, 2024

github-actions bot commented Feb 27, 2024

zwangsheng commented Feb 27, 2024

ulysses-you Feb 27, 2024

Choose a reason for hiding this comment

github-actions bot commented Feb 28, 2024

ulysses-you left a comment

Choose a reason for hiding this comment

zhztheplayer left a comment

Choose a reason for hiding this comment

GlutenPerfBot commented Feb 29, 2024

zwangsheng commented Jan 17, 2024 •

edited

Loading

zhztheplayer commented Jan 18, 2024 •

edited

Loading

zhztheplayer commented Jan 19, 2024 •

edited

Loading

zwangsheng commented Feb 26, 2024 •

edited

Loading