Skip to content

[SPARK-36345][INFRA] Update PySpark GitHubAction docker image to 20210730 #33595

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
dongjoon-hyun wants to merge 1 commit into from
Closed

[SPARK-36345][INFRA] Update PySpark GitHubAction docker image to 20210730 #33595

dongjoon-hyun wants to merge 1 commit into from

Conversation

@dongjoon-hyun
Copy link
Member

@dongjoon-hyun dongjoon-hyun commented Jul 30, 2021 •
edited
Loading

What changes were proposed in this pull request?

This PR aims to upgrade PySpark GitHub Action job to use the latest docker image 20210730 having sklearn and mlflow additionally.

$ docker run -it --rm dongjoon/apache-spark-github-action-image:20210730 python3.9 -m pip list | grep mlflow
mlflow                    1.19.0

$ docker run -it --rm dongjoon/apache-spark-github-action-image:20210730 python3.9 -m pip list | grep sklearn
sklearn                   0.0

Why are the changes needed?

This will save the installation time.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the GitHub Action PySpark jobs.

@github-actions github-actions bot added the INFRA label Jul 30, 2021
@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Jul 30, 2021

cc @HyukjinKwon

@SparkQA
Copy link

SparkQA commented Jul 30, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46417/

@HyukjinKwon
Copy link
Member

HyukjinKwon commented Jul 30, 2021

Merged to master and branch-3.2.

HyukjinKwon pushed a commit that referenced this pull request Jul 30, 2021
@dongjoon-hyun @HyukjinKwon
[SPARK-36345][INFRA] Update PySpark GitHubAction docker image to 2021…
586f898 
…0730

### What changes were proposed in this pull request?

This PR aims to upgrade PySpark GitHub Action job to use the latest docker image `20210730` having `sklearn` and `mlflow` additionally.
- dongjoon-hyun/ApacheSparkGitHubActionImage@5ca9445

```
$ docker run -it --rm dongjoon/apache-spark-github-action-image:20210730 python3.9 -m pip list | grep mlflow
mlflow                    1.19.0

$ docker run -it --rm dongjoon/apache-spark-github-action-image:20210730 python3.9 -m pip list | grep sklearn
sklearn                   0.0
```

### Why are the changes needed?

This will save the installation time.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the GitHub Action PySpark jobs.

Closes #33595 from dongjoon-hyun/SPARK-36345.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit 0e65ed5)
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
@HyukjinKwon
Copy link
Member

HyukjinKwon commented Jul 30, 2021

the test failure shouldn't be related. it's from my mistake #33596

@ueshin
Copy link
Member

ueshin commented Jul 30, 2021

Oh, wait. Actually pandas in the docker images was upgraded to 1.3.1. I guess it caused the test failures.

pandas                    1.3.1

@SparkQA
Copy link

SparkQA commented Jul 30, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46417/

@HyukjinKwon
Copy link
Member

HyukjinKwon commented Jul 30, 2021

Oh I a bit rushed to merge. @ueshin would you mind taking a quick look please for pandas 1.3.1 upgrade? I can't work on it at this moment. Reverting it for now is fine to me too.

@ueshin
Copy link
Member

ueshin commented Jul 30, 2021

Or we can disable the tests for now (#33598).

@ueshin ueshin mentioned this pull request Jul 31, 2021
Closed
@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Jul 31, 2021

This is reverted.

@ueshin
Copy link
Member

ueshin commented Jul 31, 2021

@dongjoon-hyun This was also merged to 3.2. Could you revert it as well? Thanks!

@HyukjinKwon
Copy link
Member

HyukjinKwon commented Jul 31, 2021

Thanks guys!!

@SparkQA
Copy link

SparkQA commented Jul 31, 2021

Test build #141908 has finished for PR 33595 at commit 5367960.

  • This patch fails from timeout after a configured wait of 500m.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member

HyukjinKwon commented Aug 1, 2021

@dongjoon-hyun This was also merged to 3.2. Could you revert it as well? Thanks!

Done 👍

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Aug 1, 2021 •
edited
Loading

Oh. Thanks, @HyukjinKwon . I missed the ping here. Sorry, @ueshin .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@HyukjinKwon HyukjinKwon HyukjinKwon approved these changes

Assignees

No one assigned

Labels

INFRA

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@dongjoon-hyun @SparkQA @HyukjinKwon @ueshin