[SPARK-35048][INFRA] Distribute GitHub Actions workflows to fork repositories to share the resources #32092

HyukjinKwon · 2021-04-08T09:11:27Z

What changes were proposed in this pull request?

This PR proposes to leverage the GitHub Actions resources from the forked repositories instead of using the resources in ASF organisation at GitHub.

This is how it works:

"Build and test" (build_and_test.yml) triggers a build on any commit on any branch (except branch-*.*), which roughly means:
- The original repository will trigger the build on any commits in master branch
- The forked repository will trigger the build on any commit in any branch.
The build triggered in the forked repository will checkout the original repository's master branch locally, and merge the branch from the forked repository into the original repository's master branch locally.
Therefore, the tests in the forked repository will run after being sync'ed with the original repository's master branch.
In the original repository, it triggers a workflow that detects the workflow triggered in the forked repository, and add a comment, to the PR, pointing out the workflow in forked repository.

In short, please see this example HyukjinKwon#34

You create a PR and your repository triggers the workflow. Your PR uses the resources allocated to you for testing.
Apache Spark repository finds your workflow, and links it in a comment in your PR

NOTE that we will still run the tests in the original repository for each commit pushed to master branch. This distributes the workflows only in PRs.

Why are the changes needed?

ASF shares the resources across all the ASF projects, which makes the development slow down.
Please see also:

Discussion in the builds@a.o mailing list: https://lists.apache.org/x/thread.html/r48d079eeff292254db22705c8ef8618f87ff7adc68d56c4e5d0b4105@%3Cbuilds.apache.org%3E
Infra ticket: https://issues.apache.org/jira/browse/INFRA-21646

By distributing the workflows to use author's resources, we can get around this issue.

Does this PR introduce any user-facing change?

No, this is a dev-only change.

How was this patch tested?

Manually tested at HyukjinKwon#34 and HyukjinKwon#33.

HyukjinKwon · 2021-04-08T09:14:12Z

cc @dongjoon-hyun, @MaxGekk, @maropu, @srowen, @gengliangwang FYI

cc @potiuk too FYI who might be interested in this approach.

SparkQA · 2021-04-08T10:10:16Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41644/

SparkQA · 2021-04-08T10:10:17Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41644/

srowen

Sure, seems like a good way to reduce load if it works!

SparkQA · 2021-04-08T11:45:44Z

Test build #137066 has finished for PR 32092 at commit 6dabe60.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2021-04-08T13:55:11Z

I was surprised that there is such a solution for the resource-shortage issue, nice.

gengliangwang · 2021-04-08T14:00:18Z

@HyukjinKwon Thanks for the great effort in testing infra!
+1 for this proposal. I just checked https://github.com/pricing and found that GitHub action is totally free for public repositories. Nice!

.github/workflows/notify_test_workflow.yml

dongjoon-hyun · 2021-04-08T15:29:31Z

It's great. Thank you, @HyukjinKwon !

potiuk · 2021-04-08T17:27:56Z

This is an excellent idea ! I ❤️ it. Are all the pieces already working :) ?

potiuk · 2021-04-08T17:44:24Z

One thing that is missing is how to get back the information about the status of the build in the forked repository. I do not think with the permissions model of GH it is possible to do directly, but I can think of another 'scheduled' workflow that could run in the original repository that could read status from all the PRs that had "forked" builds running.

That would be quite possible. We could even try to utilize "GitHub Check" for that https://docs.github.com/en/rest/reference/checks - I am already using it in Airflow to notify status betwen workflows (I have a 'build image` workflow that notifies the PR that runs a CI workflow this way.

This looks like the regular "yellow/green/red" build status indicator, and it impacts the "mergable" status of the PR.

This way we could have a complete solution, when the PR would become "green" when the forked build succeeds.

potiuk · 2021-04-08T17:45:58Z

One thing that is missing is how to get back the information about the status of the build in the forked repository. I do not think with the permissions model of GH it is possible to do directly, but I can think of another 'scheduled' workflow that could run in the original repository that could read status from all the PRs that had "forked" builds running.

I think it needs a bit complex logic, but I could possibly write a custom GitHub Action for that similar to what I've done with https://github.com/potiuk/cancel-workflow-runs

HyukjinKwon · 2021-04-09T00:53:17Z

Thanks @potiuk. Yeah, I think it makes sense to do that seamlessly with updating the status checks too. Borrowing the idea from cancel workflow is a good idea. If we're going this way, I will take a separate look for that.

FWIW, I do think this is still a workaround that disables GitHub Actions to work out of the box. In the mailing list, the permanent solution is being discussed, which I think makes sense and reasonable.

potiuk · 2021-04-09T01:11:11Z

Agree it's a nasty hack and should be out-of-the-box built in GitHub.

potiuk · 2021-04-12T08:31:15Z

I did not have time to take a look yet, and during the week I have a bit limited time but in a week or so I hope to have some POC with the checks in place.

tgravescs · 2021-04-12T15:09:24Z

Definitely seems like a possible way to distribute the load and cost.

Are there limitations on the personal repo GitHub actions?
looking at the free pricing for instance: (https://github.com/pricing)

2,000 Actions minutes/month
500MB of Packages storage

How many action minutes would an average spark pr take? the test build took 1 hours 27 minutes duration but is that action minutes (https://github.com/MaxGekk/spark/actions/runs/728916852) or do you have to add up each of the individual build modules, java 11, etc?

Similar with storage are log files still available, how long or is there any others differences to deal with?

just wondering some of these because say a user runs out of action minutes or there are problems, then what do we do?

potiuk · 2021-04-12T18:16:06Z

@tgravescs : Github Actions and storage are FREE for public repositories. If you look for the smaller footprint in the pricing page (bold is mine):

Unlimited public/private repositories
Unlimited collaborators
Included 2,000 Actions minutes/month
Free for public repositories
Included 500MB of GitHub Packages storage
Free for public repositories
IncludedCommunity Support

There are some limitations when it comes to number of runners available (so if you have more builds they will queue in your personal queue)

HyukjinKwon · 2021-04-13T07:41:25Z

This is ready for a review. I will take a look for #32092 (comment) separately.

SparkQA · 2021-04-13T09:48:44Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41866/

SparkQA · 2021-04-13T09:48:45Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41866/

SparkQA · 2021-04-13T10:30:40Z

Test build #137286 has finished for PR 32092 at commit 74a7674.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2021-04-14T02:53:11Z

Thanks @dongjoon-hyun!

gengliangwang

Amazing work! LGTM

HyukjinKwon · 2021-04-14T04:14:16Z

Thanks @gengliangwang! (I just fixed a indentation nit)

HyukjinKwon · 2021-04-14T04:25:57Z

dev/run-tests.py

@@ -656,16 +656,10 @@ def main():
        # If we're running the tests in GitHub Actions, attempt to detect and test
        # only the affected modules.
        if test_env == "github_actions":
-            if os.environ["GITHUB_INPUT_BRANCH"] != "":


@maropu FYI. I think we should update https://spark.apache.org/developer-tools.html because now we always run the tests on each commit in each branch in forked repositories.. I will take a look and fix it soon.

HyukjinKwon · 2021-04-14T04:26:42Z

I am merging it into master.

It's a daytime in my time .. so would likely be able to track it if anything goes wrong.

Currently contribution guide does not reflect actual flow to raise a new PR and hence it's not clear (for a new contributors) what exactly needs to be done to make a PR for Spark repository and test it as per expectation. This PR addresses that by following: - It describes in the Pull request section of the Contributing page the actual procedure and takes a contributor through a step by step process. - It removes optional "Running tests in your forked repository" section on Developer Tools page which is obsolete now and doesn't reflect reality anymore i.e. it says we can test by clicking “Run workflow” button which is not available anymore as workflow does not use "workflow_dispatch" event trigger anymore and was removed in apache/spark#32092 - Instead it documents the new procedure that above PR introduced i.e. contributors needs to use their own GitHub free workflow credits to test new changes they are purposing and a Spark Actions workflow will expect that to be completed before marking PR to be ready for a review. - Some general wording was copied from "Running tests in your forked repository" section on Developer Tools page but main content was rewritten to meet objective - Also fixed URL to developer-tools.html to be resolved by parser (that converted it into relative URI) instead of using hard coded absolute URL. Tested imperically with `bundle exec jekyll serve` and static files were generated with `bundle exec jekyll build` commands This closes https://issues.apache.org/jira/browse/SPARK-37996 Author: khalidmammadov <xmamedov@hotmail.com> Closes #378 from khalidmammadov/fix_contribution_workflow_guide.

HyukjinKwon marked this pull request as draft April 8, 2021 09:11

github-actions bot added BUILD INFRA labels Apr 8, 2021

srowen reviewed Apr 8, 2021

View reviewed changes

gengliangwang reviewed Apr 8, 2021

View reviewed changes

.github/workflows/notify_test_workflow.yml Show resolved Hide resolved

HyukjinKwon changed the title ~~[DO-NOT-MERGE][POC] Distribute GitHub Actions workflows to fork repositories to share the resources~~ [SPARK-35048][INFRA] Distribute GitHub Actions workflows to fork repositories to share the resources Apr 13, 2021

HyukjinKwon marked this pull request as ready for review April 13, 2021 07:38

HyukjinKwon added 2 commits April 13, 2021 16:39

Distribute workflows to fork repositories

1fe87d9

Trigger the sync with the latest upstream only in forked repositories

74a7674

HyukjinKwon force-pushed the poc-fork-resources branch from 6dabe60 to 74a7674 Compare April 13, 2021 07:41

dongjoon-hyun approved these changes Apr 14, 2021

View reviewed changes

gengliangwang approved these changes Apr 14, 2021

View reviewed changes

Fix indentation

dbb12b7

HyukjinKwon commented Apr 14, 2021

View reviewed changes

HyukjinKwon closed this in 2974b70 Apr 14, 2021

gengliangwang mentioned this pull request Apr 14, 2021

[DO NOT MERGE] Try the new Github action process #32158

Closed

HyukjinKwon deleted the poc-fork-resources branch January 4, 2022 00:54

khalidmammadov mentioned this pull request Jan 31, 2022

Contribution guide to document actual guide for pull requests apache/spark-website#378

Closed

singhpk234 mentioned this pull request Jul 9, 2022

Reduce CI Workload by Removing Some Spark Variants and Using Callable Workflows for Github Actions apache/iceberg#5153

Open

Hisoka-X mentioned this pull request Aug 25, 2023

[CI][Improve] Move ci to running on fork repository container apache/seatunnel#4759

Closed

3 tasks

Hisoka-X mentioned this pull request Sep 14, 2023

[INFRA] Move CI to running on fork repository container apache/seatunnel#5495

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-35048][INFRA] Distribute GitHub Actions workflows to fork repositories to share the resources #32092

[SPARK-35048][INFRA] Distribute GitHub Actions workflows to fork repositories to share the resources #32092

HyukjinKwon commented Apr 8, 2021 •

edited

Loading

HyukjinKwon commented Apr 8, 2021

SparkQA commented Apr 8, 2021

SparkQA commented Apr 8, 2021

srowen left a comment

SparkQA commented Apr 8, 2021

maropu commented Apr 8, 2021

gengliangwang commented Apr 8, 2021

dongjoon-hyun commented Apr 8, 2021

potiuk commented Apr 8, 2021

potiuk commented Apr 8, 2021

potiuk commented Apr 8, 2021

HyukjinKwon commented Apr 9, 2021

potiuk commented Apr 9, 2021

potiuk commented Apr 12, 2021

tgravescs commented Apr 12, 2021

potiuk commented Apr 12, 2021 •

edited

Loading

HyukjinKwon commented Apr 13, 2021

SparkQA commented Apr 13, 2021

SparkQA commented Apr 13, 2021

SparkQA commented Apr 13, 2021

HyukjinKwon commented Apr 14, 2021

gengliangwang left a comment

HyukjinKwon commented Apr 14, 2021

HyukjinKwon Apr 14, 2021

HyukjinKwon commented Apr 14, 2021

[SPARK-35048][INFRA] Distribute GitHub Actions workflows to fork repositories to share the resources #32092

[SPARK-35048][INFRA] Distribute GitHub Actions workflows to fork repositories to share the resources #32092

Conversation

HyukjinKwon commented Apr 8, 2021 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

HyukjinKwon commented Apr 8, 2021

SparkQA commented Apr 8, 2021

SparkQA commented Apr 8, 2021

srowen left a comment

Choose a reason for hiding this comment

SparkQA commented Apr 8, 2021

maropu commented Apr 8, 2021

gengliangwang commented Apr 8, 2021

dongjoon-hyun commented Apr 8, 2021

potiuk commented Apr 8, 2021

potiuk commented Apr 8, 2021

potiuk commented Apr 8, 2021

HyukjinKwon commented Apr 9, 2021

potiuk commented Apr 9, 2021

potiuk commented Apr 12, 2021

tgravescs commented Apr 12, 2021

potiuk commented Apr 12, 2021 • edited Loading

HyukjinKwon commented Apr 13, 2021

SparkQA commented Apr 13, 2021

SparkQA commented Apr 13, 2021

SparkQA commented Apr 13, 2021

HyukjinKwon commented Apr 14, 2021

gengliangwang left a comment

Choose a reason for hiding this comment

HyukjinKwon commented Apr 14, 2021

HyukjinKwon Apr 14, 2021

Choose a reason for hiding this comment

HyukjinKwon commented Apr 14, 2021

HyukjinKwon commented Apr 8, 2021 •

edited

Loading

potiuk commented Apr 12, 2021 •

edited

Loading