[SPARK-35991][SQL] Add PlanStability suite for TPCH by AngersZhuuuu · Pull Request #33736 · apache/spark

AngersZhuuuu · 2021-08-13T08:49:28Z

What changes were proposed in this pull request?

Add PlanStability suite for TPCH

Why are the changes needed?

Add PlanStability suite for TPCH

Does this PR introduce any user-facing change?

No

How was this patch tested?

Added UT

AngersZhuuuu · 2021-08-13T09:17:09Z

FYI @wangyum @cloud-fan
Also ping @maropu What should I do with statistic data， I see you add tpcds's statistic data.

SparkQA · 2021-08-13T10:09:51Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46930/

SparkQA · 2021-08-13T10:49:13Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46930/

SparkQA · 2021-08-13T12:52:16Z

Test build #142424 has finished for PR 33736 at commit ad6585e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-08-14T02:56:27Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46967/

SparkQA · 2021-08-14T04:25:27Z

Test build #142461 has finished for PR 33736 at commit 7e54bf8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class BlockPushNonFatalFailure extends RuntimeException
public class TimerWithCustomTimeUnit extends Timer
class BlockFetchErrorHandler implements ErrorHandler
public class NoOpMergedShuffleFileManager implements MergedShuffleFileManager
public static class AppShuffleMergePartitionsInfo
public static class AppShuffleInfo
public class RetryingBlockTransferor
public class ShuffleChecksumHelper
public class BlockPushReturnCode extends BlockTransferMessage
public class CorruptionCause extends BlockTransferMessage
public class DiagnoseCorruption extends BlockTransferMessage
public final class SparkOutOfMemoryError extends OutOfMemoryError implements SparkThrowable
final class ShuffleExternalSorter extends MemoryConsumer implements ShuffleChecksumSupport
throw new IllegalArgumentException(s\"Cannot find error class '$errorClass'\"))
class SparkArithmeticException(errorClass: String, messageParameters: Array[String])
class MutableCheckedOutputStream(out: OutputStream) extends OutputStream
case class ShuffleChecksumBlockId(shuffleId: Int, mapId: Long, reduceId: Int) extends BlockId
case class ShufflePushBlockId(
case class ShuffleMergedBlockId(
case class ShuffleMergedDataBlockId(
class RemoteBlockDownloadFileManager(
class BlockSavedOnDecommissionedBlockManagerException(blockId: BlockId)
public final class JavaStructuredComplexSessionization
public static class Sessions
public static class SessionEvent implements Serializable
public static class SessionAcc implements Serializable
public static class Session implements Serializable
case class SessionEvent(
case class SessionAcc(events: List[SessionEvent])
case class Session(
case class AvroMatchedField(
public final class Aggregation implements Serializable
public final class Count implements AggregateFunc
public final class CountStar implements AggregateFunc
public final class Max implements AggregateFunc
public final class Min implements AggregateFunc
public final class Sum implements AggregateFunc
case class UnresolvedFieldPosition(position: ColumnPosition) extends FieldPosition
case class ExpressionEquals(e: Expression)
case class ExpressionStats(expr: Expression)(var useCount: Int = 1)
case class SessionWindow(timeColumn: Expression, gapDuration: Long) extends UnaryExpression
case class Average(
case class Sum(
case class SubExprEliminationState(eval: ExprCode, children: Seq[SubExprEliminationState])
case class LocalTimestamp(timeZoneId: Option[String] = None) extends LeafExpression
case class GetTimestamp(
case class ParseToTimestampLTZ(
case class ParseToTimestamp(
case class MakeTimestampNTZ(
case class MakeTimestampLTZ(
case class DomainJoin(
trait AlterTableCommand extends UnaryCommand
case class CommentOnTable(table: LogicalPlan, comment: String) extends AlterTableCommand
case class SetTableLocation(
case class SetTableProperties(
case class UnsetTableProperties(
case class AddColumns(
case class ReplaceColumns(
case class DropColumns(
case class RenameColumn(
case class AlterColumn(
.doc(\"The custom cost evaluator class to be used for adaptive execution. If not being set,\" +
static class IntegerUpdater implements ParquetVectorUpdater
class Observation(name: String)
case class CoalescedMapperPartitionSpec(
trait AQEShuffleReadRule extends Rule[SparkPlan]
case class CoalesceShufflePartitions(session: SparkSession) extends AQEShuffleReadRule
class BasicWriteTaskStatsTracker(
protected abstract class ConnectionProviderBase extends Logging
case class ScanBuilderHolder(
case class EnsureRequirements(optimizeOutRepartition: Boolean = true) extends Rule[SparkPlan]
class MergingSortWithSessionWindowStateIterator(
class ContinuousWriteRDD(var prev: RDD[InternalRow], writerFactory: StreamingDataWriterFactory,
case class WriteToContinuousDataSource(write: StreamingWrite, query: LogicalPlan,
case class WriteToContinuousDataSourceExec(write: StreamingWrite, query: SparkPlan,
trait HDFSBackedStateStoreMap
class NoPrefixHDFSBackedStateStoreMap extends HDFSBackedStateStoreMap
class PrefixScannableHDFSBackedStateStoreMap(
class HDFSBackedReadStateStore(val version: Long, map: HDFSBackedStateStoreMap)
class HDFSBackedStateStore(val version: Long, mapToUpdate: HDFSBackedStateStoreMap)
case class RocksDBMetrics(
case class RocksDBNativeHistogram(
case class RocksDBFileManagerMetrics(
sealed trait RocksDBStateEncoder
class PrefixKeyScanStateEncoder(
class NoPrefixKeyStateEncoder(keySchema: StructType, valueSchema: StructType)
class RocksDBStateStore(lastVersion: Long) extends StateStore
sealed trait StreamingSessionWindowStateManager extends Serializable
class StreamingSessionWindowStateManagerImplV1(
class StreamingSessionWindowHelper(sessionExpression: Attribute, inputSchema: Seq[Attribute])
case class SessionWindowStateStoreRestoreExec(
case class SessionWindowStateStoreSaveExec(

SparkQA · 2021-08-14T04:57:27Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46969/

SparkQA · 2021-08-14T07:27:02Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46971/

SparkQA · 2021-08-14T07:44:42Z

Test build #142463 has finished for PR 33736 at commit 5e3f257.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-08-14T08:05:12Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46971/

SparkQA · 2021-08-14T10:26:18Z

Test build #142464 has finished for PR 33736 at commit b4151d5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-08-14T16:00:34Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46975/

SparkQA · 2021-08-14T16:38:42Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46975/

SparkQA · 2021-08-14T19:11:19Z

Test build #142468 has finished for PR 33736 at commit 04bcc51.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-08-16T03:09:09Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46984/

SparkQA · 2021-08-16T03:17:15Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46985/

SparkQA · 2021-08-16T03:45:54Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46984/

SparkQA · 2021-08-16T03:54:00Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46985/

SparkQA · 2021-08-16T06:00:34Z

Test build #142476 has finished for PR 33736 at commit 5d0fedf.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class SessionWindow(timeColumn: Expression, gapDuration: Expression) extends Expression

SparkQA · 2021-08-17T11:08:46Z

Test build #142539 has finished for PR 33736 at commit 25a3b60.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-08-17T14:01:14Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47060/

SparkQA · 2021-08-17T21:22:06Z

Test build #142558 has finished for PR 33736 at commit 21539ff.

This patch fails from timeout after a configured wait of 500m.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait PlanStabilitySuite extends DisableAdaptiveExecutionSuite
class TPCDSV1_4_PlanStabilitySuite extends PlanStabilitySuite with TPCDSBase
class TPCDSV1_4_PlanStabilityWithStatsSuite extends PlanStabilitySuite with TPCDSBase
class TPCDSV2_7_PlanStabilitySuite extends PlanStabilitySuite with TPCDSBase
class TPCDSV2_7_PlanStabilityWithStatsSuite extends PlanStabilitySuite with TPCDSBase
class TPCDSModifiedPlanStabilitySuite extends PlanStabilitySuite with TPCDSBase
class TPCDSModifiedPlanStabilityWithStatsSuite extends PlanStabilitySuite with TPCDSBase
class TPCHPlanStabilitySuite extends PlanStabilitySuite with TPCHBase
trait PlanSuiteBase extends SharedSparkSession
trait TPCDSBase extends PlanSuiteBase with TPCDSSchema
trait TPCHBase extends PlanSuiteBase

AngersZhuuuu · 2021-08-18T02:23:53Z

retest this please

SparkQA · 2021-08-18T03:31:57Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47076/

SparkQA · 2021-08-18T04:11:33Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47076/

SparkQA · 2021-08-18T05:21:24Z

Test build #142575 has finished for PR 33736 at commit 21539ff.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait PlanStabilitySuite extends DisableAdaptiveExecutionSuite
class TPCDSV1_4_PlanStabilitySuite extends PlanStabilitySuite with TPCDSBase
class TPCDSV1_4_PlanStabilityWithStatsSuite extends PlanStabilitySuite with TPCDSBase
class TPCDSV2_7_PlanStabilitySuite extends PlanStabilitySuite with TPCDSBase
class TPCDSV2_7_PlanStabilityWithStatsSuite extends PlanStabilitySuite with TPCDSBase
class TPCDSModifiedPlanStabilitySuite extends PlanStabilitySuite with TPCDSBase
class TPCDSModifiedPlanStabilityWithStatsSuite extends PlanStabilitySuite with TPCDSBase
class TPCHPlanStabilitySuite extends PlanStabilitySuite with TPCHBase
trait PlanSuiteBase extends SharedSparkSession
trait TPCDSBase extends PlanSuiteBase with TPCDSSchema
trait TPCHBase extends PlanSuiteBase

SparkQA · 2021-08-18T10:38:35Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47105/

SparkQA · 2021-08-18T14:16:59Z

Test build #142605 has finished for PR 33736 at commit 989c823.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sql/core/src/test/scala/org/apache/spark/sql/PlanSuiteBase.scala

sql/core/src/test/scala/org/apache/spark/sql/TPCDSBase.scala

Co-authored-by: Wenchen Fan <cloud0fan@gmail.com>

…nto SPARK-35991

SparkQA · 2021-08-18T15:43:16Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47115/

SparkQA · 2021-08-18T16:05:19Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47116/

SparkQA · 2021-08-18T17:02:43Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47116/

SparkQA · 2021-08-18T17:24:30Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47120/

SparkQA · 2021-08-18T19:37:05Z

Test build #142615 has finished for PR 33736 at commit ec79d4b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-08-18T21:09:42Z

Test build #142619 has finished for PR 33736 at commit c2940dc.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait TPCBase extends SharedSparkSession
trait TPCDSBase extends TPCBase with TPCDSSchema
trait TPCHBase extends TPCBase

cloud-fan · 2021-08-19T03:40:30Z

thanks, merging to master!

cloud-fan · 2021-08-23T15:00:57Z

sql/core/src/test/scala/org/apache/spark/sql/TPCBase.scala

+
+  protected def injectStats: Boolean = false
+
+  override def sparkConf: SparkConf = {


nit: we should not remove the protected modifier here. It forces all the traits/classes extending this trait to also remove the protected modifier, which is an unnecessary breaking.

See https://github.com/apache/spark/pull/33736/files#diff-67f98139e0b98db9e0e896ffbfe9d4a761bd579c8047c304bdca12906c476384L76

[SPARK-35991][SQL] Add PlanStability suite for TPCH

e588949

github-actions bot added the SQL label Aug 13, 2021

Update PlanStabilitySuite.scala

ad6585e

Merge branch 'master' into SPARK-35991

7e54bf8

AngersZhuuuu added 2 commits August 14, 2021 11:46

Merge branch 'master' into SPARK-35991

2cf70ee

Update explain.txt

5e3f257

ii

b4151d5

Merge branch 'master' into SPARK-35991

04bcc51

AngersZhuuuu added 2 commits August 16, 2021 10:19

Merge branch 'master' into SPARK-35991

5d0fedf

Update PlanStabilitySuite.scala

5c3805f

Update PlanStabilitySuite.scala

d2f0cfa

follow comment

21539ff

update

989c823

cloud-fan reviewed Aug 18, 2021

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/PlanSuiteBase.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Aug 18, 2021

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/TPCDSBase.scala Show resolved Hide resolved

cloud-fan approved these changes Aug 18, 2021

View reviewed changes

AngersZhuuuu and others added 5 commits August 18, 2021 22:50

Update sql/core/src/test/scala/org/apache/spark/sql/PlanSuiteBase.scala

adeca5c

Co-authored-by: Wenchen Fan <cloud0fan@gmail.com>

Update PlanSuiteBase.scala

d74b3b4

Merge branch 'SPARK-35991' of https://github.com/AngersZhuuuu/spark i…

072f116

…nto SPARK-35991

Update PlanStabilitySuite.scala

ec79d4b

follow comment

3d38e24

follow comment

c2940dc

cloud-fan approved these changes Aug 18, 2021

View reviewed changes

cloud-fan closed this in 559fe96 Aug 19, 2021

cloud-fan reviewed Aug 23, 2021

View reviewed changes


		protected def injectStats: Boolean = false

		override def sparkConf: SparkConf = {

Conversation

AngersZhuuuu commented Aug 13, 2021

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

AngersZhuuuu commented Aug 13, 2021

Uh oh!

SparkQA commented Aug 13, 2021

Uh oh!

SparkQA commented Aug 13, 2021

Uh oh!

SparkQA commented Aug 13, 2021

Uh oh!

SparkQA commented Aug 14, 2021

Uh oh!

SparkQA commented Aug 14, 2021

Uh oh!

SparkQA commented Aug 14, 2021

Uh oh!

SparkQA commented Aug 14, 2021

Uh oh!

SparkQA commented Aug 14, 2021

Uh oh!

SparkQA commented Aug 14, 2021

Uh oh!

SparkQA commented Aug 14, 2021

Uh oh!

SparkQA commented Aug 14, 2021

Uh oh!

SparkQA commented Aug 14, 2021

Uh oh!

SparkQA commented Aug 14, 2021

Uh oh!

SparkQA commented Aug 16, 2021

Uh oh!

SparkQA commented Aug 16, 2021

Uh oh!

SparkQA commented Aug 16, 2021

Uh oh!

SparkQA commented Aug 16, 2021

Uh oh!

SparkQA commented Aug 16, 2021

Uh oh!

SparkQA commented Aug 17, 2021

Uh oh!

SparkQA commented Aug 17, 2021

Uh oh!

SparkQA commented Aug 17, 2021

Uh oh!

AngersZhuuuu commented Aug 18, 2021

Uh oh!

SparkQA commented Aug 18, 2021

Uh oh!

SparkQA commented Aug 18, 2021

Uh oh!

SparkQA commented Aug 18, 2021

Uh oh!

SparkQA commented Aug 18, 2021

Uh oh!

SparkQA commented Aug 18, 2021

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Aug 18, 2021

Uh oh!

SparkQA commented Aug 18, 2021

Uh oh!

SparkQA commented Aug 18, 2021

Uh oh!

SparkQA commented Aug 18, 2021

Uh oh!

SparkQA commented Aug 18, 2021

Uh oh!

SparkQA commented Aug 18, 2021

Uh oh!

cloud-fan commented Aug 19, 2021

Uh oh!

cloud-fan Aug 23, 2021