[SPARK-46302][TESTS] Skip maven daily testing as ivy uses some corrupted cache jar files #44208

panbingkun · 2023-12-06T11:12:35Z

What changes were proposed in this pull request?

The pr aims to skip maven daily testing as ivy uses some corrupted cache jar files.
This is a temporary workaround solution.
After SPARK-46400 and applying it to the already released Spark version, we should remove this logic.

Why are the changes needed?

Fix maven daily testing GA.
In our Maven daily testing, some UTs failed due to some corrupt jars in maven repo:
https://github.com/apache/spark/actions/runs/7019155617/job/19095991788#step:9:27263

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass GA.
Manually check.

Was this patch authored or co-authored using generative AI tooling?

No.

panbingkun · 2023-12-07T07:24:18Z

As seen above, missing hive-exec-2.3.9.jar and slf4j-api-1.7.30.jar.

panbingkun · 2023-12-19T06:23:03Z

How about just add a filter at

spark/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala

Lines 256 to 266 in 8cd4661

val testingVersions: Seq[String] = if (isPythonVersionAvailable &&

SystemUtils.isJavaVersionAtMost(JavaVersion.JAVA_17)) {

import scala.io.Source

try Utils.tryWithResource(

Source.fromURL(s"$releaseMirror/spark")) { source =>

source.mkString

.split("\n")

.filter(_.contains("""<a href="spark-"""))

.filterNot(_.contains("preview"))

.map("""<a href="spark-(\d.\d.\d)/">""".r.findFirstMatchIn(_).get.group(1))

.filter(_ < org.apache.spark.SPARK_VERSION).toImmutableArraySeq

like:
val skipReleaseVersions =
    sys.env.getOrElse("SKIP_SPARK_RELEASE_VERSIONS", "").split(",").toSet
...

source.mkString
        .split("\n")
        .filter(_.contains("""<a href="spark-"""))
        .filterNot(_.contains("preview"))
        .map("""<a href="spark-(\d.\d.\d)/">""".r.findFirstMatchIn(_).get.group(1))
        .filter(_ < org.apache.spark.SPARK_VERSION)
        .filterNot(skipReleaseVersions.contains).toImmutableArraySeq

Done.

LuciferYang · 2023-12-19T06:40:40Z

.github/workflows/maven_test.yml

        required: false
        type: string
-        default: '{}'
+        default: '{"SKIP_SPARK_RELEASE_VERSIONS": "3.3.3,3.4.2,3.5.0,master"}'


No need for master

and default should be '{}' or '', we should pass it from build_maven.yml

LuciferYang · 2023-12-19T06:47:14Z

.github/workflows/maven_test.yml

      HIVE_PROFILE: ${{ matrix.hive }}
      SPARK_LOCAL_IP: localhost
      GITHUB_PREV_SHA: ${{ github.event.before }}
+      SKIP_SPARK_RELEASE_VERSIONS: ${{ fromJSON(inputs.envs).SKIP_SPARK_RELEASE_VERSIONS }}


This option should not be needed, the Run tests section has already been deserialized for env

spark/.github/workflows/maven_test.yml

Lines 172 to 173 in 0b8e099

- name: Run tests

env: ${{ fromJSON(inputs.envs) }}

LuciferYang · 2023-12-19T07:02:22Z

.github/workflows/build_maven.yml

+    with:
+      envs: >-
+        {
+          "SKIP_SPARK_RELEASE_VERSIONS": "3.3.3,3.4.2,3.5.0"


Suggested change

"SKIP_SPARK_RELEASE_VERSIONS": "3.3.3,3.4.2,3.5.0"

"SKIP_SPARK_RELEASE_VERSIONS": "3.3.4,3.4.2,3.5.0"

LuciferYang · 2023-12-19T08:13:23Z

I think we should make #44343 as TODO of this one, and explicitly add TODO in the code. On the other hand, please update the PR title and PR description to better match the change of this PR.

…ted cache jar files

panbingkun · 2023-12-19T11:03:31Z

I think we should make #44343 as TODO of this one, and explicitly add TODO in the code. On the other hand, please update the PR title and PR description to better match the change of this PR.

Done.

LuciferYang · 2023-12-19T11:22:07Z

.github/workflows/build_and_test.yml

          npm install --save-dev
          node --experimental-vm-modules node_modules/.bin/jest
+
+  maven-test:


fine to me, let's revert this change after test pass

I will revert this change, because the maven test has passed and the verification is successful.

LuciferYang · 2023-12-19T11:30:24Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala

 }

 object PROCESS_TABLES extends QueryTest with SQLTestUtils {
+  // TODO After SPARK-46400 and applying it to the already released Spark version,


hmm... I think maybe we can describe it like

In SPARK-46302, the env SKIP_SPARK_RELEASE_VERSIONS has been added to allow Maven tests to skip problematic release versions. Related issues will be fixed in SPARK-46400, and testing will be resumed after the fixed Spark 3.x version is released.

and I think just add comments here, no need to add them in line 247

Okay, let's run this test first.

in addition, before we release 3. x, should we test test("backward compatibility") after the SPARK-46400 feature is fixed on the branch master?

~~I don't think it has ever tested master...~~
I don't think it's necessary at the moment.

Before the fixed version released, Maven's daily tests can skip this case first.

…ted cache jar files

LuciferYang

+1, LGTM

…ted cache jar files

LuciferYang · 2023-12-20T05:51:44Z

Merged into master. Thanks @panbingkun

Let's monitor next maven daily test ~

…maven repo, skip this cache and try again ### What changes were proposed in this pull request? The pr aims to - fix potential bug(ie: #44208) and enhance user experience. - make the code more compliant with standards ### Why are the changes needed? We use the local maven repo as the first-level cache in ivy. The original intention was to reduce the time required to parse and obtain the ar, but when there are corrupted files in the local maven repo,The above mechanism will be directly interrupted and the prompt is very unfriendly, which will greatly confuse the user. Based on the original intention, we should skip the cache directly in similar situations. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manually test. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #44343 from panbingkun/SPARK-46400. Authored-by: panbingkun <pbk1982@gmail.com> Signed-off-by: Sean Owen <srowen@gmail.com>

…ocal maven repo, skip this cache and try again ### What changes were proposed in this pull request? The pr aims to - fix potential bug(ie: #44208) and enhance user experience. - make the code more compliant with standards Backport above to branch 3.5. Master branch pr: #44343 ### Why are the changes needed? We use the local maven repo as the first-level cache in ivy. The original intention was to reduce the time required to parse and obtain the ar, but when there are corrupted files in the local maven repo,The above mechanism will be directly interrupted and the prompt is very unfriendly, which will greatly confuse the user. Based on the original intention, we should skip the cache directly in similar situations. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manually test. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #45017 from panbingkun/branch-3.5_SPARK-46400. Authored-by: panbingkun <panbingkun@baidu.com> Signed-off-by: yangjie01 <yangjie01@baidu.com>

…ocal maven repo, skip this cache and try again ### What changes were proposed in this pull request? The pr aims to - fix potential bug(ie: #44208) and enhance user experience. - make the code more compliant with standards Backport above to branch 3.4. Master branch pr: #44343 ### Why are the changes needed? We use the local maven repo as the first-level cache in ivy. The original intention was to reduce the time required to parse and obtain the ar, but when there are corrupted files in the local maven repo,The above mechanism will be directly interrupted and the prompt is very unfriendly, which will greatly confuse the user. Based on the original intention, we should skip the cache directly in similar situations. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manually test. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #45018 from panbingkun/branch-3.4_SPARK-46400. Authored-by: panbingkun <panbingkun@baidu.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

…ocal maven repo, skip this cache and try again ### What changes were proposed in this pull request? The pr aims to - fix potential bug(ie: apache#44208) and enhance user experience. - make the code more compliant with standards Backport above to branch 3.4. Master branch pr: apache#44343 ### Why are the changes needed? We use the local maven repo as the first-level cache in ivy. The original intention was to reduce the time required to parse and obtain the ar, but when there are corrupted files in the local maven repo,The above mechanism will be directly interrupted and the prompt is very unfriendly, which will greatly confuse the user. Based on the original intention, we should skip the cache directly in similar situations. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manually test. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#45018 from panbingkun/branch-3.4_SPARK-46400. Authored-by: panbingkun <panbingkun@baidu.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

try fix maven test

f0f7035

github-actions bot added the INFRA label Dec 6, 2023

panbingkun changed the title ~~try fix maven test~~ [Don't merge & review] try fix maven test Dec 6, 2023

panbingkun added 5 commits December 6, 2023 21:28

try

0b75a0f

try

08e8859

try

5fca2b7

try

db9b07e

try

742e29b

github-actions bot added the SQL label Dec 6, 2023

panbingkun added 9 commits December 7, 2023 09:28

try

953bc7e

try

e3c6e8c

try

4377bfc

try

c128af6

try

a7e1b12

try

a0b9450

fix maven tests

77e94e9

Merge branch 'master' into try_fix_maven_test

224084e

fix maven tests

436c244

github-actions bot removed the SQL label Dec 7, 2023

panbingkun added 2 commits December 7, 2023 14:07

fix maven tests

a9e146a

fix maven tests

b7ae5cb

panbingkun changed the title ~~[Don't merge & review] try fix maven test~~ [Don't merge & review] [SPARK-46302][BUILD] Fix maven daily testing Dec 7, 2023

panbingkun added 3 commits December 7, 2023 16:03

try

a57c512

try

2fe75bd

try

5a971b4

github-actions bot added SQL BUILD labels Dec 7, 2023

panbingkun added 2 commits December 7, 2023 18:36

try

dbcbd64

try

b2fd3ee

LuciferYang reviewed Dec 19, 2023

View reviewed changes

[SPARK-46302][BUILD] Fix maven daily testing

ab06613

LuciferYang reviewed Dec 19, 2023

View reviewed changes

[SPARK-46302][BUILD] Fix maven daily testing

aa37791

panbingkun requested a review from LuciferYang December 19, 2023 07:35

panbingkun changed the title ~~[SPARK-46302][BUILD] Fix maven daily testing~~ [SPARK-46302][BUILD] Make maven's daily testing skip some released spark versions, where ivy uses corrupted cache jars Dec 19, 2023

panbingkun changed the title ~~[SPARK-46302][BUILD] Make maven's daily testing skip some released spark versions, where ivy uses corrupted cache jars~~ [SPARK-46302][TEST] Make maven's daily testing skip some released spark versions, where ivy uses corrupted cache jars Dec 19, 2023

panbingkun changed the title ~~[SPARK-46302][TEST] Make maven's daily testing skip some released spark versions, where ivy uses corrupted cache jars~~ [SPARK-46302][TESTS] Make maven's daily testing skip some released spark versions, where ivy uses corrupted cache jars Dec 19, 2023

panbingkun changed the title ~~[SPARK-46302][TESTS] Make maven's daily testing skip some released spark versions, where ivy uses corrupted cache jars~~ [SPARK-46302][TESTS] Skip maven daily testing as ivy uses some corrupted cache jar files Dec 19, 2023

[SPARK-46302][TESTS] Skip maven daily testing as ivy uses some corrup…

9eb03f8

…ted cache jar files

LuciferYang reviewed Dec 19, 2023

View reviewed changes

panbingkun added 2 commits December 20, 2023 08:31

[SPARK-46302][TESTS] Skip maven daily testing as ivy uses some corrup…

a70bf61

…ted cache jar files

Merge branch 'master' into try_fix_maven_test

45aaecb

panbingkun requested a review from LuciferYang December 20, 2023 03:26

LuciferYang approved these changes Dec 20, 2023

View reviewed changes

[SPARK-46302][TESTS] Skip maven daily testing as ivy uses some corrup…

941ab8e

…ted cache jar files

LuciferYang closed this in 6fe6828 Dec 20, 2023

This was referenced Feb 4, 2024

[SPARK-46400][CORE][SQL][3.5] When there are corrupted files in the local maven repo, skip this cache and try again #45017

Closed

[SPARK-46400][CORE][SQL][3.4] When there are corrupted files in the local maven repo, skip this cache and try again #45018

Closed

	"SKIP_SPARK_RELEASE_VERSIONS": "3.3.3,3.4.2,3.5.0"
	"SKIP_SPARK_RELEASE_VERSIONS": "3.3.4,3.4.2,3.5.0"

[SPARK-46302][TESTS] Skip maven daily testing as ivy uses some corrupted cache jar files #44208

[SPARK-46302][TESTS] Skip maven daily testing as ivy uses some corrupted cache jar files #44208

Uh oh!

Conversation

panbingkun commented Dec 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

panbingkun commented Dec 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

panbingkun commented Dec 19, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LuciferYang Dec 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LuciferYang commented Dec 19, 2023

Uh oh!

panbingkun commented Dec 19, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LuciferYang Dec 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LuciferYang Dec 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LuciferYang left a comment

Choose a reason for hiding this comment

Uh oh!

LuciferYang commented Dec 20, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

panbingkun commented Dec 6, 2023 •

edited

Loading

panbingkun commented Dec 7, 2023 •

edited

Loading

LuciferYang Dec 19, 2023 •

edited

Loading

LuciferYang Dec 19, 2023 •

edited

Loading

LuciferYang Dec 19, 2023 •

edited

Loading