Patch for CI #2996

Sharpz7 · 2023-09-25T21:04:03Z

Strange Temp Note

Sometimes (Like on commit ad47bf5), the cache will be hit, but not used. I have no idea what causes this.

Partially mentioned in #2987

I tested it by merging it into my forks main and:

Checking that the release still passes
Checking that the cache was created
Created another PR on my fork to verify that the cache works, and the required check system functions: Update README.md Sharpz7/armada#17
Check the CI runs on my master fork okay: https://github.com/Sharpz7/armada/actions/runs/6591442058

The Integration Tests on master create the cache, and that is shared everywhere, including with new PR's (i.e no more long waits on your first push)

No Cache: https://photos.app.goo.gl/MGSWW51YY3BSwvPG9 (Integration Tests: 17m)
Old Cache: https://photos.app.goo.gl/CtuqvmgoKLfWNJpc6 (Integration Tests: 14m)
New Cache: https://photos.app.goo.gl/qyWcsKxUtN6RYdFk8 (Integration Tests: 7m) Half!!!

IMPORTANT (Taken from G-Research/fasttrackml#458 (comment))

This PR improves how we create the required check "All required checks succeeded" in the following ways:

We now use a GitHub app to create the check instead of using the token provided by GitHub Actions. This allows us to create the check run with its own check suite instead of attaching it to the first GHA check suite, which changes when a pull request is closed and reopened, or when a new run attempt is made. See https://github.com/orgs/community/discussions/24616#discussioncomment-6088422 for more details on this very annoying GitHub limitation.
A check is created and set as queued or in_progress when the corresponding workflow run's status changes.
This is needed so that the check is reset when a new run attempt is made or a PR is reopened.
Logging is improved.
Start/Completion times are added.

The following tasks need to be completed before merging this PR:

create and install a new GitHub app with the check:write permissions
add its credentials to a new create-check environment that can only be accessed by the main branch
The following tasks need to be completed after merging this PR:

update branch protection (or ruleset, if G-Research/fasttrackml#452 has been merged in the meantime) to enforce the required check to originate from the new GitHub app

.github/workflows/build.yml

pavlovic-ivan · 2023-10-19T13:42:39Z

.github/workflows/check-required.yml

+        with:
+          github-token: ${{ steps.app-token.outputs.token }}
+          script: |
+            const ghaAppId = 15368;


Should this value be a secret? Also, const names are not concise. Would be good to rename to something more descriptive (ghaAppId, ghaName ,myAppId, myName)

I don't think so. and since we use this check in other repos, I think we should keep it consistent between them in terms of naming

@jgiannuzzi do you know more about how to pick this ID?

oh this is the public app ID of GitHub Actions 😄

I don't think so. and since we use this check in other repos, I think we should keep it consistent between them in terms of naming

If this is the case, then the best thing would be for this workflow to exist in a dedicated "reusable" repo, and then called (if such a thing is acceptable for Armada). What happens when we have 10 repos all having the same check-required.yml workflow, and something changes in one of them? Are we going to track down the 9 other repos that don't have the latest change and create a PR?

@jgiannuzzi @Sharpz7 feel free to tell me i'm nitpicking

that's a great point @pavlovic-ivan

if you agree, i think we should create an action out of this

Sure! You may want to first prototype it in some repo and have your forks of armada and fml use it. If it works well, we can probably create a GR/actions repo to collect all of our shared actions/pipelines, starting with this one.

You are reading my mind (with GR/actions)!

pavlovic-ivan · 2023-10-19T13:46:41Z

I don't quite understand the purpose of the checks-required workflow, together with the change made to the all-required-checks-done in ci.yml. Can you elaborate please? The echo in that job can be misleading since the checks are done in the other workflow, that can fail, but CI will always print out "All required checks done"

Sharpz7 · 2023-10-20T16:52:08Z

@pavlovic-ivan This is something jonathan created that is used in other GR-OSS repos. Basically, the echo you are refering to is not the one you use. You make your required job the one that prints "All required checks succeeded". To the best of my knowledge, this is a work around for Github Actions treating skipped jobs in a way we do not want.

…mada into patch-required-checks

Sharpz7 · 2023-10-20T18:29:32Z

.github/workflows/test.yml

+      # cache-primary-key - Cache primary key passed in the input to use in subsequent steps of the workflow.
+      # cache-matched-key - Key of the cache that was restored, it could either be the primary key on cache-hit or a partial/complete match of one of the restore keys.
+      - uses: actions/cache/save@v3
+        if: steps.cache-restore.outputs.cache-hit == 'false' || steps.cache-restore.outputs.cache-primary-key != steps.cache-restore.outputs.cache-matched-key


This is to try and save the 20-90s we lose from the cache compressing, and then checking if the key exists. That should really happen the other way round.

Sharpz7 · 2023-10-20T18:37:19Z

Currently Broken:

Cannot get release to work
GitHub Pages Failure
check-required.yml not working
The time the build/prepare step takes varies massively.

No idea how to fix any of them currently, I will investigate.

jgiannuzzi · 2023-10-20T18:46:46Z

@Sharpz7 please add information in your PR description about what needs to happen before and after your PR is merged for check-required.yml to work. See G-Research/fasttrackml#458 (comment) for more info.

jgiannuzzi · 2023-10-20T18:49:17Z

and for anyone looking for more context about the "new" required checks, see the following PRs:

Create "All required checks succeeded" check run when "All required checks done" CI job is actually successful G-Research/fasttrackml#241
Improve required checks creation in CI G-Research/fasttrackml#458
Do not run "Check required jobs" on forks G-Research/fasttrackml#470
Rewrite the required checks process to look for check runs of the commit G-Research/fasttrackml#473
Create a pending required check as early as possible G-Research/fasttrackml#481

richscott · 2023-11-13T20:31:42Z

.github/workflows/airflow-operator.yml

@@ -90,15 +90,27 @@ jobs:
      - uses: actions/checkout@v3.3.0
      - run: docker buildx create --name ${DOCKER_BUILDX_BUILDER} --driver docker-container --use
      - run: docker buildx install
-      - uses: actions/setup-go@v4
+
+      - name: Set up Go


In the test, lint, and release workflows, we use the magnetikonline/action-galng-cache plugin, which functionally replaces the functionality of actions/setup-go and actions/cache/restore together - can that work here?

Yes - Now that I deleted the extra folder we cache, this can potentially change. But I do think having separate save and cache steps (Which is not possible with magnetikonline/action-galng-cache) is better?

I don't see the advantage of "separate save and cache steps" with using actions/cache/restore - I believe the magnetikonline cache plugin automatically sets up the build and module cache paths as needed before, and persists those when the job is done.

richscott · 2023-11-13T20:36:03Z

.github/workflows/release.yml

@@ -54,10 +54,23 @@ jobs:
      - name: Fetch Git tags
        run: git fetch --force --tags

-      - name: Setup Golang with Cache


And here we have the inverse - does magnetikonline/action-golang-cache not work, is buggy, or...?

See above comment

jgiannuzzi · 2023-11-14T12:18:36Z

Have a look at the work done in G-Research/fasttrackml#552 (and its sibling PRs: G-Research/fasttrackml#553 and G-Research/fasttrackml#554).

This should show you a rationale for what to save/restore from the cache in an optimised way.

pavlovic-ivan · 2023-11-16T14:07:20Z

FYI @dave-gantenbein @Sharpz7 @jgiannuzzi @richscott : i will be creating several PRs from now on, for this entire thing, split up into smaller pieces, and refer co-authors in those PRs (depending of course on the change itself). So i guess this PR can be closed, as others will contain changes from this one, + other necessary changes.

Sharpz7 · 2023-11-16T16:45:25Z

Closing. Good luck @pavlovic-ivan !

Change Added

7f38b32

Sharpz7 added the PR waiting for approval PR that is finished, but needs an approval label Sep 25, 2023

Virtual CI check addition

96519db

Sharpz7 added PR CI issues PR is waiting for CI fixes. PR waiting for approval PR that is finished, but needs an approval and removed PR waiting for approval PR that is finished, but needs an approval PR CI issues PR is waiting for CI fixes. labels Sep 26, 2023

Sharpz7 self-assigned this Oct 3, 2023

Sharpz7 mentioned this pull request Oct 13, 2023

Normalize workflow naming and mage usage [WIP] #2987

Closed

Sharpz7 and others added 9 commits October 13, 2023 03:03

Adding debug

ecef7cf

More debugging

bf65641

More testing

341dc13

Added Docker Compose Logs

83800b9

Fix

a819fb2

More changes

52bd0db

Extending Pulsar wait time

1001c7b

Merge branch 'master' into patch-required-checks

4cfbd65

Try this.

17846b2

Sharpz7 force-pushed the patch-required-checks branch 2 times, most recently from 965d2c8 to 17846b2 Compare October 16, 2023 04:12

Sharpz7 and others added 10 commits October 15, 2023 22:40

Update test.yml

2917ece

Update test.yml

508da8e

Update test.yml

b3afffd

Debug checks

a897304

More debugging

ce91066

Added debugg

32562cd

Merge branch 'master' into patch-required-checks

8a7268c

Trying fixes

7e1be9d

Removed url

cc1f033

Update ci.go

9069c43

Sharpz7 added do-not-merge Do not merge this PR and removed PR waiting for approval PR that is finished, but needs an approval labels Oct 18, 2023

dave-gantenbein reviewed Oct 19, 2023

View reviewed changes

.github/workflows/build.yml Outdated Show resolved Hide resolved

dave-gantenbein requested a review from pavlovic-ivan October 19, 2023 13:15

pavlovic-ivan reviewed Oct 19, 2023

View reviewed changes

Trying new change

85d3be1

Sharpz7 and others added 6 commits October 20, 2023 11:05

Merge branch 'master' into patch-required-checks

9f62931

Added more changes

d393304

Merge branch 'patch-required-checks' of https://github.com/Sharpz7/ar…

22a9c7d

…mada into patch-required-checks

Our changes

ab7ff8d

Added some docs changes and a change to python

40a7618

Added airflow fix

d68fc9f

Sharpz7 requested review from pavlovic-ivan and dave-gantenbein October 20, 2023 18:02

Sharpz7 commented Oct 20, 2023

View reviewed changes

Sharpz7 and others added 3 commits October 26, 2023 22:16

Added doc change

fd689eb

Merge branch 'master' into patch-required-checks

1123a43

Added debug step

8a20780

Sharpz7 added PR waiting for approval PR that is finished, but needs an approval and removed do-not-merge Do not merge this PR labels Oct 27, 2023

richscott reviewed Nov 13, 2023

View reviewed changes

Sharpz7 closed this Nov 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Patch for CI #2996

Patch for CI #2996

Sharpz7 commented Sep 25, 2023 •

edited

pavlovic-ivan Oct 19, 2023 •

edited

Sharpz7 Oct 20, 2023

Sharpz7 Oct 20, 2023

jgiannuzzi Oct 20, 2023

pavlovic-ivan Oct 23, 2023

jgiannuzzi Nov 13, 2023

pavlovic-ivan Nov 13, 2023

jgiannuzzi Nov 14, 2023

pavlovic-ivan Nov 14, 2023

pavlovic-ivan commented Oct 19, 2023

Sharpz7 commented Oct 20, 2023

Sharpz7 Oct 20, 2023

Sharpz7 commented Oct 20, 2023 •

edited

jgiannuzzi commented Oct 20, 2023

jgiannuzzi commented Oct 20, 2023

richscott Nov 13, 2023

Sharpz7 Nov 13, 2023

richscott Nov 13, 2023

richscott Nov 13, 2023

Sharpz7 Nov 13, 2023

jgiannuzzi commented Nov 14, 2023

pavlovic-ivan commented Nov 16, 2023 •

edited

Sharpz7 commented Nov 16, 2023

Patch for CI #2996

Patch for CI #2996

Conversation

Sharpz7 commented Sep 25, 2023 • edited

Strange Temp Note

IMPORTANT (Taken from G-Research/fasttrackml#458 (comment))

pavlovic-ivan Oct 19, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pavlovic-ivan commented Oct 19, 2023

Sharpz7 commented Oct 20, 2023

Choose a reason for hiding this comment

Sharpz7 commented Oct 20, 2023 • edited

jgiannuzzi commented Oct 20, 2023

jgiannuzzi commented Oct 20, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jgiannuzzi commented Nov 14, 2023

pavlovic-ivan commented Nov 16, 2023 • edited

Sharpz7 commented Nov 16, 2023

Sharpz7 commented Sep 25, 2023 •

edited

pavlovic-ivan Oct 19, 2023 •

edited

Sharpz7 commented Oct 20, 2023 •

edited

pavlovic-ivan commented Nov 16, 2023 •

edited