Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-37086][R][ML][TESTS] Fix the R test of FPGrowthModel for Scala 2.13 #34357

Closed
wants to merge 5 commits into from

Conversation

sarutak
Copy link
Member

@sarutak sarutak commented Oct 21, 2021

What changes were proposed in this pull request?

This PR fixes an issue that the R test of FPGrowthModel fails with Scala 2.13.
Similar to the issue filed in SPARK-37059 (#34330), the R test of FPGrowthModel assumes that the result records returned by FPGrowthModel.freqItemsets are sorted by a certain kind of order but it's wrong.
As a result, the test fails with Scala 2.13.

 ══ Failed ══════════════════════════════════════════════════════════════════════
── 1. Failure (test_mllib_fpm.R:42:3): spark.fpGrowth ──────────────────────────
`expected_itemsets` not equivalent to `itemsets`.
Component “items”: Component 1: Component 1: 1 string mismatch
Component “items”: Component 2: Length mismatch: comparison on first 1 components
Component “items”: Component 2: Component 1: 1 string mismatch
Component “items”: Component 3: Length mismatch: comparison on first 1 components
Component “items”: Component 4: Length mismatch: comparison on first 1 components
Component “items”: Component 4: Component 1: 1 string mismatch
Component “items”: Component 5: Length mismatch: comparison on first 1 components
Component “items”: Component 5: Component 1: 1 string mismatch
Component “freq”: Mean relative difference: 0.5454545

Why are the changes needed?

For test stability.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

All the R tests passed on GA with a modified GA config to run with Scala 2.13 (set scala2.13 to the SPARK_PROFILE environment variable).
https://github.com/sarutak/spark/runs/3964081433?check_suite_focus=true#step:7:380

Test for Scala 2.12 should be done on the regular GA.

@SparkQA
Copy link

SparkQA commented Oct 21, 2021

Test build #144509 has finished for PR 34357 at commit 0a8d5e1.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Oct 21, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48981/

@SparkQA
Copy link

SparkQA commented Oct 21, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48981/

@dongjoon-hyun
Copy link
Member

Could you check AppVeyor failure?

@sarutak
Copy link
Member Author

sarutak commented Oct 21, 2021

Hmm, recently, AppVeyor seems to fail almost every time.
https://ci.appveyor.com/project/ApacheSoftwareFoundation/spark/history

But I'll re-trigger it.

@SparkQA
Copy link

SparkQA commented Oct 21, 2021

Test build #144512 has finished for PR 34357 at commit 2a061de.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Oct 21, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48984/

@SparkQA
Copy link

SparkQA commented Oct 21, 2021

Test build #144513 has finished for PR 34357 at commit b070ec8.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Oct 21, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48984/

@SparkQA
Copy link

SparkQA commented Oct 21, 2021

Test build #144514 has finished for PR 34357 at commit 7bde2a6.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Oct 21, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48985/

@SparkQA
Copy link

SparkQA commented Oct 21, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48985/

@HyukjinKwon
Copy link
Member

AppVeyor is very flaky .. I tried so hard to fix but no luck 😢

Copy link
Member

@HyukjinKwon HyukjinKwon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@HyukjinKwon
Copy link
Member

Merged to master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
4 participants