feat: cli option to enable the new action cache #1954

ChristopherHX · 2023-08-08T18:56:04Z

This is my initial code to switch the action cache over to my new implementation

Until all tests are migrated this will be kept as draft to avoid mergify spam.

I would be careful to test this change, to detect known issues of my previous PR's and this change.

Checklist

Test symlinked action.yml file in remote actions
Test symlinked action.yml file in local actions
Test remote docker actions
Test using images with a non root container (I expect we still need to fix permissions)
Test error messages like
- action.yml not found
- Clone failed
Make Tests Green (If I add a Feature Toggle for this change the test don't need to change)

Do you prefer to add a Feature Toggle for this change? Then we can merge this while iron out remaining bugs later and make it possible to make use of this change for people affected by the current problems.

ChristopherHX · 2023-08-08T19:41:52Z

pkg/container/docker_run.go

+	// Mkdir
+	buf := &bytes.Buffer{}
+	tw := tar.NewWriter(buf)
+	_ = tw.WriteHeader(&tar.Header{
+		Name:     destPath,
+		Mode:     777,
+		Typeflag: tar.TypeDir,
+	})
+	tw.Close()
+	err := cr.cli.CopyToContainer(ctx, cr.id, "/", buf, types.CopyToContainerOptions{})
+	if err != nil {
+		return fmt.Errorf("failed to mkdir to copy content to container: %w", err)
+	}


The second CopyToContainer would fail with destPath doesn't exists.
Not using mkdir, works in containers without that tool and use plain docker api.

ChristopherHX · 2023-08-08T19:42:17Z

pkg/container/docker_run.go

+	// If this fails, then folders have wrong permissions on non root container
+	if cr.UID != 0 || cr.GID != 0 {
+		_ = cr.Exec([]string{"chown", "-R", fmt.Sprintf("%d:%d", cr.UID, cr.GID), destPath}, nil, "0", "")(ctx)
+	}


As before we need to fix folder ownership

ChristopherHX · 2023-08-08T19:42:52Z

pkg/runner/action.go

 	rc := step.getRunContext()
 	stepModel := step.getStepModel()

-	if stepModel.Type() != model.StepTypeUsesActionRemote {
+	if stepModel.Type() != model.StepTypeUsesActionRemote || rc.ActionCache == nil {


This is added to avoid a panic in a Test

ChristopherHX · 2023-08-08T19:44:45Z

pkg/runner/run_context.go

@@ -47,6 +47,7 @@ type RunContext struct {
 	Masks               []string
 	cleanUpJobContainer common.Executor
 	caller              *caller // job calling this RunContext (reusable workflows)
+	ActionCache         ActionCache


Move to runner.Config?

ChristopherHX · 2023-08-08T19:45:30Z

pkg/runner/step_action_remote.go

+		sar.cacheDir = fmt.Sprintf("%s/%s", sar.remoteAction.Org, sar.remoteAction.Repo)
+		sar.resolvedSha, err = cache.Fetch(ctx, sar.cacheDir, sar.remoteAction.URL+"/"+sar.cacheDir, sar.remoteAction.Ref, github.Token)


resolve the action and cache access variables

codecov · 2023-08-16T15:34:10Z

Codecov Report

Attention: 1013 lines in your changes are missing coverage. Please review.

Comparison is base (4989f44) 61.22% compared to head (cc6ca22) 60.72%.
Report is 298 commits behind head on master.

Files	Patch %	Lines
pkg/artifactcache/handler.go	65.46%	102 Missing and 42 partials ⚠️
pkg/runner/run_context.go	73.37%	75 Missing and 19 partials ⚠️
pkg/runner/expression.go	55.17%	66 Missing and 12 partials ⚠️
pkg/runner/action_cache.go	50.74%	49 Missing and 17 partials ⚠️
pkg/container/docker_run.go	1.51%	64 Missing and 1 partial ⚠️
pkg/container/docker_network.go	0.00%	56 Missing ⚠️
pkg/model/planner.go	28.57%	53 Missing and 2 partials ⚠️
pkg/runner/reusable_workflow.go	52.00%	42 Missing and 6 partials ⚠️
pkg/model/workflow.go	43.37%	40 Missing and 7 partials ⚠️
pkg/common/outbound_ip.go	0.00%	44 Missing ⚠️
... and 27 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1954      +/-   ##
==========================================
- Coverage   61.22%   60.72%   -0.51%     
==========================================
  Files          46       53       +7     
  Lines        7141     8959    +1818     
==========================================
+ Hits         4372     5440    +1068     
- Misses       2462     3081     +619     
- Partials      307      438     +131

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

ChristopherHX

This is a feature preview, a cli flag controls if this feature is on.

Next steps are cli flags for local repository replacements instead of cloning the action.

E.g. developing an actions pre step in act.

ChristopherHX · 2023-08-16T15:03:32Z

cmd/root.go

@@ -612,6 +613,11 @@ func newRunCommand(ctx context.Context, input *Input) func(*cobra.Command, []str
 			ReplaceGheActionTokenWithGithubCom: input.replaceGheActionTokenWithGithubCom,
 			Matrix:                             matrixes,
 		}
+		if input.useNewActionCache {
+			config.ActionCache = &runner.GoGitActionCache{


Enabled behind feature cli flag: --use-new-action-cache, old code has been restored.

ChristopherHX · 2023-08-16T16:56:15Z

See ChristopherHX#34 for what is possible with substitution.

The filecollector should be moved out of the container package and made public.

cmd/root.go

Co-authored-by: Björn Brauer <zaubernerd@zaubernerd.de>

mergify · 2023-08-21T12:34:57Z

@ChristopherHX this pull request has failed checks 🛠

ChristopherHX · 2023-09-23T21:15:01Z

Just added support for remote reusable workflows, should be possible to be redirected them to a local folder via https://github.com/nektos/act/tree/local-repository-action-cache

./act --github-instance ubuntu.fritz.box:3042 --use-new-action-cache -W w.yml -P ubuntu-latest=ubuntu.fritz.box:2000/runner-images:ubuntu22-runner-large-latest --local-repository https://ubuntu.fritz.box:3042/Christopher/re-complex@main=$PWD

w.yml

on: push
jobs:
  caller:
    uses: Christopher/re-complex/.github/workflows/callable.yml@main
    with:
      os: ubuntu-latest
    secrets:
      API_TOKEN: ${{ tojson(github.sha) }}

.github/workflows/callable.yml

on:
  workflow_call:
    inputs:
      os:
        required: true
        type: string
    secrets:
      API_TOKEN:
        required: true
jobs:
  test-scripts:
    runs-on: ${{ inputs.os }}
    steps:
    - run: echo ${{ secrets.API_TOKEN }}

mergify · 2023-10-23T03:26:47Z

@ChristopherHX this pull request is now in conflict 😩

ChristopherHX · 2024-01-08T19:18:36Z

Seems like we decided to no longer try to fix concurrent action / reusable workflow updates.

Am I allowed to continue here, or what are your concerns?

ChristopherHX · 2024-01-08T19:22:43Z

pkg/runner/step.go

+func symlinkJoin(filename, sym, parent string) (string, error) {
+	dir := path.Dir(filename)
+	dest := path.Join(dir, sym)
+	prefix := path.Clean(parent) + "/"
+	if strings.HasPrefix(dest, prefix) || prefix == "./" {
+		return dest, nil
+	}
+	return "", fmt.Errorf("symlink tries to access file '%s' outside of '%s'", strings.ReplaceAll(dest, "'", "''"), strings.ReplaceAll(parent, "'", "''"))
+}


I forget that \ are technically allowed here, path wouldn't handle them...
Don't think it is a new problem

cplee

tested locally 👍

* Enable the new action cache * fix * fix: CopyTarStream (Docker) * suppress panic in test * add a cli option for opt in * fixups * add package * fix * rc.Config nil in test??? * add feature flag * patch * Fix respect --action-cache-path Co-authored-by: Björn Brauer <zaubernerd@zaubernerd.de> * add remote reusable workflow to ActionCache * fixup --------- Co-authored-by: Björn Brauer <zaubernerd@zaubernerd.de> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

Enable the new action cache

d620ca7

pull-request-size bot added the size/L label Aug 8, 2023

ChristopherHX added 3 commits August 8, 2023 20:58

fix

b8a5410

fix: CopyTarStream (Docker)

64df8ae

suppress panic in test

be5d3a4

ChristopherHX commented Aug 8, 2023

View reviewed changes

ChristopherHX added 7 commits August 16, 2023 16:28

add a cli option for opt in

b497266

fixups

7e7f4b2

add package

13ffce7

fix

5c1c3b7

rc.Config nil in test???

509df53

add feature flag

05c91d6

patch

48364c6

ChristopherHX changed the title ~~Enable the new action cache~~ feat: cli option to enable the new action cache Aug 16, 2023

ChristopherHX commented Aug 16, 2023

View reviewed changes

ChristopherHX marked this pull request as ready for review August 16, 2023 15:37

ChristopherHX requested a review from a team as a code owner August 16, 2023 15:37

Merge branch 'master' into enable-new-action-cache

d1ada52

ChristopherHX mentioned this pull request Aug 16, 2023

Allow to work with local actions outside of the repository we run the workflows from #1579

Open

ZauberNerd reviewed Aug 21, 2023

View reviewed changes

cmd/root.go Outdated Show resolved Hide resolved

mergify bot requested a review from a team August 21, 2023 11:53

Fix respect --action-cache-path

0cee15c

Co-authored-by: Björn Brauer <zaubernerd@zaubernerd.de>

mergify bot added the needs-work Extra attention is needed label Aug 21, 2023

Merge branch 'master' into enable-new-action-cache

354d9d0

mergify bot removed the needs-work Extra attention is needed label Aug 21, 2023

ChristopherHX added 2 commits September 23, 2023 22:31

Merge branch 'master' of github.com:nektos/act into HEAD

a1a5869

add remote reusable workflow to ActionCache

139998c

fixup

ff9dd6a

mergify bot added the conflict PR has conflicts label Oct 23, 2023

Merge branch 'master' into enable-new-action-cache

f8226cd

mergify bot removed the conflict PR has conflicts label Jan 8, 2024

ChristopherHX commented Jan 8, 2024

View reviewed changes

This was referenced Jan 13, 2024

act fails to update cached action clones #1785

Open

feat: Test writing remote Actions locally #2155

Merged

cplee approved these changes Jan 19, 2024

View reviewed changes

Merge branch 'master' into enable-new-action-cache

cc6ca22

mergify bot merged commit f7a846d into nektos:master Jan 19, 2024
10 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: cli option to enable the new action cache #1954

feat: cli option to enable the new action cache #1954

ChristopherHX commented Aug 8, 2023 •

edited

ChristopherHX Aug 8, 2023

ChristopherHX Aug 8, 2023

ChristopherHX Aug 8, 2023

ChristopherHX Aug 8, 2023

ChristopherHX Aug 8, 2023

codecov bot commented Aug 16, 2023 •

edited

ChristopherHX left a comment

ChristopherHX Aug 16, 2023

ChristopherHX commented Aug 16, 2023

mergify bot commented Aug 21, 2023

ChristopherHX commented Sep 23, 2023 •

edited

mergify bot commented Oct 23, 2023

ChristopherHX commented Jan 8, 2024

ChristopherHX Jan 8, 2024 •

edited

cplee left a comment

		sar.cacheDir = fmt.Sprintf("%s/%s", sar.remoteAction.Org, sar.remoteAction.Repo)
		sar.resolvedSha, err = cache.Fetch(ctx, sar.cacheDir, sar.remoteAction.URL+"/"+sar.cacheDir, sar.remoteAction.Ref, github.Token)

feat: cli option to enable the new action cache #1954

feat: cli option to enable the new action cache #1954

Conversation

ChristopherHX commented Aug 8, 2023 • edited

ChristopherHX Aug 8, 2023

Choose a reason for hiding this comment

ChristopherHX Aug 8, 2023

Choose a reason for hiding this comment

ChristopherHX Aug 8, 2023

Choose a reason for hiding this comment

ChristopherHX Aug 8, 2023

Choose a reason for hiding this comment

ChristopherHX Aug 8, 2023

Choose a reason for hiding this comment

codecov bot commented Aug 16, 2023 • edited

Codecov Report

ChristopherHX left a comment

Choose a reason for hiding this comment

ChristopherHX Aug 16, 2023

Choose a reason for hiding this comment

ChristopherHX commented Aug 16, 2023

mergify bot commented Aug 21, 2023

ChristopherHX commented Sep 23, 2023 • edited

mergify bot commented Oct 23, 2023

ChristopherHX commented Jan 8, 2024

ChristopherHX Jan 8, 2024 • edited

Choose a reason for hiding this comment

cplee left a comment

Choose a reason for hiding this comment

ChristopherHX commented Aug 8, 2023 •

edited

codecov bot commented Aug 16, 2023 •

edited

ChristopherHX commented Sep 23, 2023 •

edited

ChristopherHX Jan 8, 2024 •

edited