[SPARK-42599][CONNECT][INFRA] Introduce `dev/connect-jvm-client-mima-check` instead of `CompatibilitySuite` #40191

LuciferYang · 2023-02-27T09:33:31Z

What changes were proposed in this pull request?

The main changes of this pr as follows:

Refactor CompatibilitySuite as a new tool CheckConnectJvmClientCompatibility and move it into tools module
Add a new shell script dev/connect-jvm-client-mima-check, it will use CheckConnectJvmClientCompatibility to check the mima compatibility of connect-jvm-client module.
Add dev/connect-jvm-client-mima-check to github task

Why are the changes needed?

For fix test error report in [VOTE] Release Apache Spark 3.4.0 (RC1) mail list.

Testing CompatibilitySuite with maven requires some pre-work:

build/mvn clean install -DskipTests -pl sql/core -am
build/mvn clean install -DskipTests -pl connector/connect/client/jvm -am

So if we run build/mvn package test to test whole project as before, CompatibilitySuite will failed as follows:

CompatibilitySuite:
- compatibility MiMa tests *** FAILED ***
  java.lang.AssertionError: assertion failed: Failed to find the jar inside folder: /home/bjorn/spark-3.4.0/connector/connect/client/jvm/target
  at scala.Predef$.assert(Predef.scala:223)
  at org.apache.spark.sql.connect.client.util.IntegrationTestUtils$.findJar(IntegrationTestUtils.scala:67)
  at org.apache.spark.sql.connect.client.CompatibilitySuite.clientJar$lzycompute(CompatibilitySuite.scala:57)
  at org.apache.spark.sql.connect.client.CompatibilitySuite.clientJar(CompatibilitySuite.scala:53)
  at org.apache.spark.sql.connect.client.CompatibilitySuite.$anonfun$new$1(CompatibilitySuite.scala:69)
  at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
  at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
  at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
  at org.scalatest.Transformer.apply(Transformer.scala:22)
  at org.scalatest.Transformer.apply(Transformer.scala:20)
  ...
- compatibility API tests: Dataset *** FAILED ***
  java.lang.AssertionError: assertion failed: Failed to find the jar inside folder: /home/bjorn/spark-3.4.0/connector/connect/client/jvm/target
  at scala.Predef$.assert(Predef.scala:223)
  at org.apache.spark.sql.connect.client.util.IntegrationTestUtils$.findJar(IntegrationTestUtils.scala:67)
  at org.apache.spark.sql.connect.client.CompatibilitySuite.clientJar$lzycompute(CompatibilitySuite.scala:57)
  at org.apache.spark.sql.connect.client.CompatibilitySuite.clientJar(CompatibilitySuite.scala:53)
  at org.apache.spark.sql.connect.client.CompatibilitySuite.$anonfun$new$7(CompatibilitySuite.scala:110)
  at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
  at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
  at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
  at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
  at org.scalatest.Transformer.apply(Transformer.scala:22)

So we need to fix this problem for developers.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Pass GitHub Actions

HyukjinKwon · 2023-02-27T12:57:05Z

cc @Yikun FYI

LuciferYang · 2023-02-27T15:06:53Z

tools/src/main/scala/org/apache/spark/tools/CheckConnectJvmClientCompatibility.scala

-      fail(
-        s"\nComparing client jar: $clientJar\nand sql jar: $sqlJar\n" +
-          problems.map(p => p.description("client")).mkString("\n"))
+  private def checkDatasetApiCompatibility(clientJar: File, sqlJar: File): Array[String] = {


The mima check failed now due to there are 4 incompatible Apis between Dataset in sql module and Dataset in connect-client-jvm module with current comparison way, this is the right result.

zhenlineo · 2023-02-27T19:28:03Z

dev/connect-jvm-client-mima-check

+if test ! -z "$ERRORS"; then
+  cat .connect-mima-check-result
+  echo -e "connect-client-jvm module mima check failed."
+  echo -e "Exceptions to binary compatibility can be added in tools/CheckConnectJvmClientCompatibility.scala"


Can we keep the file CheckConnectJvmClientCompatibility.scala inside client/jvm and make the tools depends on the client test-jar? Then we do not need to copy past the findJar and recursiveListFiles methods. And also we can keep the code closer to the client?

According to the previous habit, the tools module is independent, and it does not rely on any other Spark module. I do not want to break this.

In addition, the compatibility API tests: Dataset related issues I mentioned in another pr, put it inside the client module may cause unexpected class loading.

If we are sure we don't need to check compatibility API tests: Dataset or there are other ways to solve it, I can try to put CheckConnectJvmClientCompatibility back into the client module.

Another pr #40213 keep CheckConnectJvmClientCompatibility in client/jvm, if we all recommend keeping CheckConnectJvmClientCompatibility in the client/jvm module, we can use that one

Hi, I vote for change #40213 Thanks!

LuciferYang · 2023-02-28T02:34:33Z

This pr often conflicts....

HyukjinKwon · 2023-02-28T03:50:30Z

cc @hvanhovell too FYI

hvanhovell · 2023-03-01T16:15:16Z

connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Dataset.scala

@@ -2739,7 +2739,7 @@ class Dataset[T] private[sql] (
    sparkSession.analyze(plan, proto.AnalyzePlanRequest.AnalyzeCase.SCHEMA)
  }

-  def collectResult(): SparkResult[T] = sparkSession.execute(plan, encoder)


Why make this private[sql]? For advanced use cases this is a better way of interacting with results.

hvanhovell · 2023-03-01T16:23:34Z

@LuciferYang can we close this one in favor of #40213?

LuciferYang · 2023-03-02T00:17:14Z

@LuciferYang can we close this one in favor of #40213?

OK, let me close this one and focus on #40213

LuciferYang added 2 commits February 27, 2023 17:30

add a tool check connect-jvm-client mima

9d99b39

Merge branch 'upmaster' into connect-client-compatibility

adbd5df

github-actions bot added BUILD INFRA labels Feb 27, 2023

LuciferYang marked this pull request as draft February 27, 2023 09:39

LuciferYang added 2 commits February 27, 2023 17:50

skip branch 3.2 & 3.3

373fe6a

remove CompatibilitySuite

7e0b96a

github-actions bot added the CONNECT label Feb 27, 2023

LuciferYang added 3 commits February 27, 2023 17:58

fix

ed49ca0

remove unused import

7a9e26b

fix compile

4b93767

LuciferYang changed the title ~~[SPARK-42554][CONNECT] Add dev/connect-jvm-client-mima-check tool to instead of CompatibilitySuite~~ [SPARK-42554][CONNECT][INFRA] Add dev/connect-jvm-client-mima-check tool to instead of CompatibilitySuite Feb 27, 2023

LuciferYang changed the title ~~[SPARK-42554][CONNECT][INFRA] Add dev/connect-jvm-client-mima-check tool to instead of CompatibilitySuite~~ [SPARK-42554][CONNECT][INFRA] Add dev/connect-jvm-client-mima-check instead of CompatibilitySuite Feb 27, 2023

LuciferYang changed the title ~~[SPARK-42554][CONNECT][INFRA] Add dev/connect-jvm-client-mima-check instead of CompatibilitySuite~~ [SPARK-42554][CONNECT][INFRA] Introduce dev/connect-jvm-client-mima-check instead of CompatibilitySuite Feb 27, 2023

LuciferYang changed the title ~~[SPARK-42554][CONNECT][INFRA] Introduce dev/connect-jvm-client-mima-check instead of CompatibilitySuite~~ [SPARK-42599][CONNECT][INFRA] Introduce dev/connect-jvm-client-mima-check instead of CompatibilitySuite Feb 27, 2023

Merge branch 'upmaster' into connect-client-compatibility

5c14dae

LuciferYang commented Feb 27, 2023

View reviewed changes

comments checkDatasetApiCompatibility

b913c9a

github-actions bot added the SQL label Feb 27, 2023

LuciferYang added 2 commits February 27, 2023 23:35

revert

ef029b1

revert

ddac49f

github-actions bot removed the SQL label Feb 27, 2023

LuciferYang added 3 commits February 27, 2023 23:37

revert

ad78165

change to use scala api

4e63fb9

fix compatibility

2888902

github-actions bot added the SQL label Feb 27, 2023

shell

f389143

LuciferYang marked this pull request as ready for review February 27, 2023 16:48

rm before test

067167c

LuciferYang added 2 commits February 28, 2023 00:56

rm before test

a5a193a

rm before test

0cfd327

zhenlineo reviewed Feb 27, 2023

View reviewed changes

Merge branch 'upmaster' into connect-client-compatibility

e25ec88

LuciferYang added 3 commits February 28, 2023 14:09

Merge branch 'upmaster' into connect-client-compatibility

af06d40

check wc

dd35cf6

Merge branch 'upmaster' into connect-client-compatibility

66db38c

hvanhovell reviewed Mar 1, 2023

View reviewed changes

hvanhovell mentioned this pull request Mar 1, 2023

[SPARK-42599][CONNECT][INFRA] Introduce dev/connect-jvm-client-mima-check instead of CompatibilitySuite #40213

Closed

LuciferYang closed this Mar 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-42599][CONNECT][INFRA] Introduce `dev/connect-jvm-client-mima-check` instead of `CompatibilitySuite` #40191

[SPARK-42599][CONNECT][INFRA] Introduce `dev/connect-jvm-client-mima-check` instead of `CompatibilitySuite` #40191

LuciferYang commented Feb 27, 2023 •

edited

HyukjinKwon commented Feb 27, 2023

LuciferYang Feb 27, 2023

zhenlineo Feb 27, 2023

LuciferYang Feb 28, 2023

LuciferYang Feb 28, 2023 •

edited

zhenlineo Feb 28, 2023

LuciferYang commented Feb 28, 2023

HyukjinKwon commented Feb 28, 2023

hvanhovell Mar 1, 2023

hvanhovell commented Mar 1, 2023

LuciferYang commented Mar 2, 2023

[SPARK-42599][CONNECT][INFRA] Introduce dev/connect-jvm-client-mima-check instead of CompatibilitySuite #40191

[SPARK-42599][CONNECT][INFRA] Introduce dev/connect-jvm-client-mima-check instead of CompatibilitySuite #40191

Conversation

LuciferYang commented Feb 27, 2023 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

HyukjinKwon commented Feb 27, 2023

LuciferYang Feb 27, 2023

Choose a reason for hiding this comment

zhenlineo Feb 27, 2023

Choose a reason for hiding this comment

LuciferYang Feb 28, 2023

Choose a reason for hiding this comment

LuciferYang Feb 28, 2023 • edited

Choose a reason for hiding this comment

zhenlineo Feb 28, 2023

Choose a reason for hiding this comment

LuciferYang commented Feb 28, 2023

HyukjinKwon commented Feb 28, 2023

hvanhovell Mar 1, 2023

Choose a reason for hiding this comment

hvanhovell commented Mar 1, 2023

LuciferYang commented Mar 2, 2023

[SPARK-42599][CONNECT][INFRA] Introduce `dev/connect-jvm-client-mima-check` instead of `CompatibilitySuite` #40191

[SPARK-42599][CONNECT][INFRA] Introduce `dev/connect-jvm-client-mima-check` instead of `CompatibilitySuite` #40191

LuciferYang commented Feb 27, 2023 •

edited

LuciferYang Feb 28, 2023 •

edited