Golang caching does not cache dependencies on action builds

[ben-of-codecraft]

If not using a versioned request Golang will not cache. This creates a longer builds as dependencies have to be downloaded every time. This is a bug with how composite actions and setup-go work together that has been filed.

I found an alternate action that does effectively the same thing without using the built-in setup-go caching.
https://github.com/magnetikonline/action-golang-cache/blob/main/action.yaml

Unexpected Result:

Updated action code:

    - name: Setup OpenTofu
      uses: opentofu/setup-opentofu@v1.0.3
      with:
        tofu_version: ${{ inputs.opentofu-version }}
      if: inputs.setup-opentofu == 'true'


    - name: Setup Checkov
      run: |
        python3 -m venv .venv
        source .venv/bin/activate
        pip3 install --upgrade pip
        pip3 install --upgrade setuptools
        pip3 install -U checkov==${{ inputs.checkov-version }}
      shell: bash
      if: inputs.setup-checkov == 'true'

    - name: Setup go with cache  
      uses: magnetikonline/action-golang-cache@v5
      with:
        go-version-file: ${{ github.action_path }}/cli/go.mod          
      if: ${{ !startsWith(github.action_ref, 'v') && github.action_ref != 'latest'}}

With the changes above result:

Overall this took repeat builds from ~2-3 min down to 30-40s

[motatoes]

Hey! Thanks for filing this. For everyone's context we currently have to ways of consuming the digger action. The first is through downloading prebuilt binaries and invoking digger. The second is to build from source and its useful for testing from a branch that you are actively working on (development).

We build digger binaries on release and we have a convention that release names start with "v". So that is what we check for in the action as well. If it starts with a "v" then we download the available binary, otherwise we clone the repo and build from source.

If you wish to use digger with always latest we have a release called "vLatest" which is always a release pointing to the latest release branch ..

Now to your issue, which is about slow builds from cache due to no caching. I haven't investigated much but I did realise how slow the builds take when you are working and testing a feature. Usually I sometimes create a temp "v-xyz" release just to make my tests quickly but I know its not great. I'm not sure about using a third party action for this because I have some considerations there, especially with something critical like the toolchain itself. Would much rather have the solution built into native GHA step or another thing that we custom build somehow ..

Thoughts ?

[ben-of-codecraft]

I hear you about adding another custom action from an unknown source, I am not a fan of that either for production areas. I was thinking more that since the action is relatively simply to port the steps either as a composite in digger itself or just applying the steps into the workflow directly.

Adjacent to the issue above is how pushing versions to workflows are handled:
Your repo has an existing workflow script that updates a "latest" tag automatically on release. What I have been doing, since we do an internal build with custom changes, is have our template builder workflow tag the digger_workflow.yml at '@latest' instead of a tagged release version. This causes the build to happen instead of leveraging the pre-compiled binary. This is also why I noticed the golang caching problem.

Why do it this way?
This is the simplest way for us to push changes to all the repos / workflows that will be using the workflow action. I want project owners to opt-out of the "latest" changes and revert to an earlier version if a build problem occurs. I know our user base they this will lead to better engagement than asking everyone to update versions as we release.

So how are we going about this inside the action:
Because we are using an internal version instead of a public one, we have additional hoops to jump through, I assume this would be the case for any larger organization self-hosting in a regulated environment.

First I addressed the golang build caching. Though this really only addresses branches after the change in "second"
Second I added in changes to the action to download "latest" binaries if latest tag is specified. This ensures the latest tagged binary is download and used.

This is what the code looks like (I redacted our info)

    - name: run digger
      if: ${{ startsWith(github.action_ref, 'v') || github.action_ref == 'latest' }}
      env:
        actionref: ${{ github.action_ref }}
        PLAN_UPLOAD_DESTINATION: ${{ inputs.upload-plan-destination }}
        GOOGLE_STORAGE_LOCK_BUCKET: ${{ inputs.google-lock-bucket }}                        
        GOOGLE_STORAGE_PLAN_ARTEFACT_BUCKET: ${{ inputs.upload-plan-destination-gcp-bucket }}
        AWS_S3_BUCKET: ${{ inputs.upload-plan-destination-s3-bucket }}
        ACTIVATE_VENV: ${{ inputs.setup-checkov == 'true' }}
        DISABLE_LOCKING: ${{ inputs.disable-locking == 'true' }}
        DIGGER_TOKEN: ${{ inputs.digger-token }}
        DIGGER_ORGANISATION: ${{ inputs.digger-organisation }}
        DIGGER_HOSTNAME: ${{ inputs.digger-hostname }}
        DIGGER_FILENAME: ${{ inputs.digger-filename }}
        ACCUMULATE_PLANS: ${{ inputs.post-plans-as-one-comment == 'true' }}
        REPORTING_STRATEGY: ${{ inputs.reporting-strategy }}
        INPUT_DIGGER_PROJECT: ${{ inputs.project }}
        INPUT_DIGGER_MODE: ${{ inputs.mode }}
        INPUT_DIGGER_COMMAND: ${{ inputs.command }}
        INPUT_DRIFT_DETECTION_SLACK_NOTIFICATION_URL: ${{ inputs.drift-detection-slack-notification-url }}
        NO_BACKEND: ${{ inputs.no-backend }}
        TF_PLUGIN_CACHE_DIR: ${{ github.workspace }}/cache
        TERRAGRUNT_PROVIDER_CACHE: ${{ inputs.cache-dependencies == 'true' && 1 || 0 }}
        TERRAGRUNT_PROVIDER_CACHE_DIR: ${{ github.workspace }}/cache        
      id: digger
      shell: bash
      run: |                
        if [[ ${actionref} -eq "latest" ]]; then
          RELEASE_URL=$(curl -s -H "Authorization: token $(REDACTED)_GITHUB_TOKEN" \
          https://api.github.com/repos/our_org/our_repo/releases/latest | \
          jq -r '.assets[] | select(.name | contains("digger-cli-${{ runner.os  }}-${{ runner.arch }}")) | .url')        
          curl -sL -H "Authorization: token $(REDACTED)_GITHUB_TOKEN" -H "Accept: application/octet-stream" -H "X-GitHub-Api-Version: 2022-11-28" $RELEASE_URL -o digger
        else          
          curl -sL -H "Authorization: token $(REDACTED)_GITHUB_TOKEN" https://github.com/our_org/our_repo/releases/download/${actionref}/digger-cli-${{ runner.os  }}-${{ runner.arch }} -o digger
        fi
        
        chmod +x digger
        PATH=$PATH:$(pwd)
        cd $GITHUB_WORKSPACE
        digger

[ben-of-codecraft]

This does require an extra step to creating an org secret that has "Read Access" to the internal digger fork and passing that as a new env variable in the digger_workflow.yaml

[motatoes]

I have an alternative idea for your case since you want to keep some kind of "latest" release always up to date what if you created an automation action for releasing that:

• releases the new vX.Y.Z tag
• performs another release called "latest" for the same commit. It can also be called something different like version-latest or so

Now instead of releasing via github releases you can instead invoke the "releaser action"

In this way your teams can be consuming latest by simply using "latest" or "v-latest" and you have a stable release process based on github too. I saw this done in opentofu and I'm thinking to do the same here as well. Currently we have a release called "vLatest" which I make sure to update on every release but planning to automate that part as well

[ben-of-codecraft]

@motatoes that's what we do, but you already have this working in your repo. Oddly, you already had this built in as automated in 0.4.32 then you removed it in 0.4.33? It did exactly what I wanted in creating a latest tag.

https://github.com/diggerhq/digger/blob/v0.4.32/.github/workflows/cli_latest.yml

That does what you suggest, the issue we had was in the action it did a build always instead of just downloading that binary, which is what I fixed in the action.

[ben-of-codecraft]

I did update it some to use modern output directives in GHA, and to handle private repos:

updated .github/workflows/ci_latest.yml

name: Update latest tag for every new latest release

on:
  release:
    types:
      - released

permissions:
  contents: write
  actions: read
  pull-requests: write
  issues: read
  discussions: read

jobs:
  update_latest_tag:
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository
        uses: actions/checkout@v4

      - name: Check if the latest release
        id: check_latest_release
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          latest_release=$(curl -s -H "Authorization: token $GITHUB_TOKEN" https://api.github.com/repos/${{ github.repository }}/releases/latest | jq -r '.tag_name')
          echo "Latest release: $latest_release"
          echo "Current release: ${{ github.ref }}"
          if [[ "refs/tags/$latest_release" == "${{ github.ref }}" ]]; then            
            echo "is_latest=true" >> $GITHUB_OUTPUT            
          else
            echo "is_latest=false" >> $GITHUB_OUTPUT
          fi

      - name: Update latest tag
        if: steps.check_latest_release.outputs.is_latest == 'true'
        uses: EndBug/latest-tag@latest
        with:
          ref: latest
          description: Latest tag
          force-branch: false

      - name: Upload release asset
        if: steps.check_latest_release.outputs.is_latest == 'true'
        uses: softprops/action-gh-release@v2
        with:
          files: LICENSE
          token: ${{ secrets.GITHUB_TOKEN }}

[motatoes]

@ben-of-codecraft this is cool! We removed it because I thought nobody was using the latest tag and we were planning to do that thing with "vLatest" release, but clearly we are mistaken.

We can bring it back again using what you have up there if of interest.

As for caching, if we manage to extract what we need from that third party action and have some native logic in our action.yml I'm all for it too.

Contributions welcomed!

[ben-of-codecraft]

I'll see what I can do to help out with the caching part, and for the latest tag, I'd recommend that approach, as it's more prevalent in package management systems and package registries, making it easy for most programmers.

[ben-of-codecraft]

Also, please bring back latest tag action 🙏