[SPARK-44023][SQL][TESTS] Add `System.gc` at `beforeEach` in `PruneFileSourcePartitionsSuite` by LuciferYang · Pull Request #41533 · apache/spark

LuciferYang · 2023-06-09T08:48:42Z

What changes were proposed in this pull request?

This pr aims to add System.gc at beforeEach in PruneFileSourcePartitionsSuite to avoid sql-others GA testing task OOM.

Why are the changes needed?

Make GA testing stable

Does this PR introduce any user-facing change?

No, just for test

How was this patch tested?

Should monitor CI

connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/application/ReplE2ESuite.scala

LuciferYang · 2023-06-11T07:46:51Z

...c/test/scala/org/apache/spark/sql/execution/datasources/PruneFileSourcePartitionsSuite.scala


 class PruneFileSourcePartitionsSuite extends PrunePartitionSuiteBase with SharedSparkSession {

+  override def beforeEach(): Unit = {


Let's check if this is really useful @HyukjinKwon @zhengruifeng

LuciferYang · 2023-06-12T03:02:10Z

The latest rebase did not actually run GA test, let me trigger it again

zhengruifeng · 2023-06-12T10:52:04Z

it seems sql - other was already timeout,

what about figuring out the problematic test suites to disable for now?

LuciferYang · 2023-06-12T11:09:15Z

The running one should be org.apache.spark.sql.execution.command.v2.AlterTableRenamePartitionSuite

LuciferYang · 2023-06-12T11:21:51Z

2023-06-12T07:47:10.5042643Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[32mAlterTableRenamePartitionSuite:�[0m�[0m
2023-06-12T07:47:10.7210341Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[32m- ALTER TABLE .. RENAME PARTITION using V2 catalog V2 command: rename without explicitly specifying database (171 milliseconds)�[0m�[0m
2023-06-12T07:47:10.7333258Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[32m- ALTER TABLE .. RENAME PARTITION using V2 catalog V2 command: table to alter does not exist (13 milliseconds)�[0m�[0m
2023-06-12T07:47:10.7959978Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[32m- ALTER TABLE .. RENAME PARTITION using V2 catalog V2 command: partition to rename does not exist (62 milliseconds)�[0m�[0m
2023-06-12T07:47:10.8771267Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[32m- ALTER TABLE .. RENAME PARTITION using V2 catalog V2 command: target partitions exist (80 milliseconds)�[0m�[0m
2023-06-12T07:47:10.9971419Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[32m- ALTER TABLE .. RENAME PARTITION using V2 catalog V2 command: single part partition (119 milliseconds)�[0m�[0m
2023-06-12T07:47:11.1922848Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[32m- ALTER TABLE .. RENAME PARTITION using V2 catalog V2 command: multi part partition (195 milliseconds)�[0m�[0m
2023-06-12T07:47:11.3404610Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[32m- ALTER TABLE .. RENAME PARTITION using V2 catalog V2 command: partition spec in RENAME PARTITION should be case insensitive (147 milliseconds)�[0m�[0m

hang at org.apache.spark.sql.execution.command.v2.AlterTableRenamePartitionSuite, ignore it and re-testing

LuciferYang · 2023-06-12T11:32:58Z

Another frequently killed is JavaRowSuite, but I don't think there are any issues with it.

~~How about downgrade sbt to 1.8.3 for more test?~~ Ignore this

LuciferYang · 2023-06-12T12:11:50Z

https://github.com/apache/spark/pull/41552/files is making another try, add a new test group to the sql module to alleviate the workload of group sql others

LuciferYang · 2023-06-12T14:07:21Z

...rc/test/scala/org/apache/spark/sql/execution/command/v2/AlterTableRenamePartitionSuite.scala

 * The class contains tests for the `ALTER TABLE .. RENAME PARTITION` command
 * to check V2 table catalogs.
 */
+@Ignore


Although GA passed, I am not sure we really need ignore this case ...

Let's ignore it for now.

HyukjinKwon · 2023-06-13T00:08:12Z

Merged to master.

panbingkun · 2023-06-13T02:01:47Z

https://github.com/apache/spark/pull/41552/files is making another try, add a new test group to the sql module to alleviate the workload of group sql others

Perhaps this is a good idea.

dongjoon-hyun

Thank you for all your efforts about sql - others failures.
Although the master branch still fails after merging this, I believe this PR helps us narrow-down the investigation scope.

I'm also +1 for adding a new test group and splitting sql - others into two groups. Thank you again all.

…TableRenamePartitionSuite` ### What changes were proposed in this pull request? #41533 ignore `AlterTableRenamePartitionSuite` try to restore stability of `sql-others` test task, but it seems that it is not the root cause that affects stability, so this pr has removed the previously added `ignore` identifier to restore testing. ### Why are the changes needed? Resume testing of `AlterTableRenamePartitionSuite` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? should monitor ci Closes #41647 from LuciferYang/SPARK-44089. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Max Gekk <max.gekk@gmail.com>

…leSourcePartitionsSuite` ### What changes were proposed in this pull request? This pr aims to add `System.gc` at `beforeEach` in `PruneFileSourcePartitionsSuite` to avoid `sql-others` GA testing task OOM. ### Why are the changes needed? Make GA testing stable ### Does this PR introduce _any_ user-facing change? No, just for test ### How was this patch tested? Should monitor CI Closes apache#41533 from LuciferYang/GA-test. Lead-authored-by: yangjie01 <yangjie01@baidu.com> Co-authored-by: YangJie <yangjie01@baidu.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

…TableRenamePartitionSuite` ### What changes were proposed in this pull request? apache#41533 ignore `AlterTableRenamePartitionSuite` try to restore stability of `sql-others` test task, but it seems that it is not the root cause that affects stability, so this pr has removed the previously added `ignore` identifier to restore testing. ### Why are the changes needed? Resume testing of `AlterTableRenamePartitionSuite` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? should monitor ci Closes apache#41647 from LuciferYang/SPARK-44089. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Max Gekk <max.gekk@gmail.com>

LuciferYang marked this pull request as draft June 9, 2023 08:48

github-actions bot added CONNECT INFRA SQL labels Jun 9, 2023

LuciferYang commented Jun 9, 2023

View reviewed changes

connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/application/ReplE2ESuite.scala Outdated Show resolved Hide resolved

add system gc in PruneFileSourcePartitionsSuite

6704065

LuciferYang force-pushed the GA-test branch from bee762c to 6704065 Compare June 11, 2023 07:36

github-actions bot removed INFRA CONNECT labels Jun 11, 2023

LuciferYang changed the title ~~[DRAFT] Test remove script~~ [DRAFT] Test add System.gc to PruneFileSourcePartitionsSuite Jun 11, 2023

LuciferYang added 2 commits June 11, 2023 15:38

Merge branch 'apache:master' into GA-test

4bd78fd

for ga

cb95fe4

LuciferYang commented Jun 11, 2023

View reviewed changes

Merge branch 'apache:master' into GA-test

5e207c7

zhengruifeng approved these changes Jun 12, 2023

View reviewed changes

LuciferYang marked this pull request as ready for review June 12, 2023 03:01

LuciferYang changed the title ~~[DRAFT] Test add System.gc to PruneFileSourcePartitionsSuite~~ [SPARK-44023][SQL][TESTS] Test add System.gc to PruneFileSourcePartitionsSuite Jun 12, 2023

LuciferYang changed the title ~~[SPARK-44023][SQL][TESTS] Test add System.gc to PruneFileSourcePartitionsSuite~~ [SPARK-44023][SQL][TESTS] Add System.gc at beforeEach in PruneFileSourcePartitionsSuite Jun 12, 2023

for ga

0ba8e6a

HyukjinKwon approved these changes Jun 12, 2023

View reviewed changes

ignore AlterTableRenamePartitionSuite

4a2cd26

LuciferYang commented Jun 12, 2023

View reviewed changes

Merge branch 'apache:master' into GA-test

66a2b49

HyukjinKwon closed this in 933dfd9 Jun 13, 2023

dongjoon-hyun reviewed Jun 13, 2023

View reviewed changes

dongjoon-hyun mentioned this pull request Jun 13, 2023

[SPARK-44045][SQL][TESTS] Mark WholeStageCodegenSparkSubmitSuite as ExtendedSQLTest #41579

Closed

This was referenced Jun 18, 2023

[SPARK-44034][TESTS] Add a new test group for sql module #41638

Closed

[SPARK-44089][SQL][TESTS] Remove the @ignore identifier from AlterTableRenamePartitionSuite #41647

Closed


		class PruneFileSourcePartitionsSuite extends PrunePartitionSuiteBase with SharedSparkSession {

		override def beforeEach(): Unit = {

Conversation

LuciferYang commented Jun 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Uh oh!

LuciferYang Jun 11, 2023

Choose a reason for hiding this comment

Uh oh!

LuciferYang commented Jun 12, 2023

Uh oh!

zhengruifeng commented Jun 12, 2023

Uh oh!

LuciferYang commented Jun 12, 2023

Uh oh!

LuciferYang commented Jun 12, 2023

Uh oh!

LuciferYang commented Jun 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LuciferYang commented Jun 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LuciferYang Jun 12, 2023

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon Jun 13, 2023

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Jun 13, 2023

Uh oh!

panbingkun commented Jun 13, 2023

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Comments

LuciferYang commented Jun 9, 2023 •

edited

Loading

LuciferYang commented Jun 12, 2023 •

edited

Loading

LuciferYang commented Jun 12, 2023 •

edited

Loading