Generating dependency report fails on release

[richardcase]

/kind bug

What steps did you take and what happened:

When trying to do recent releases of v2.3.x & v2.4.0-beta.0 we encountered the following problem when generating the dependency report for the release notes:

ERRO unable to run cmd: go list -mod=readonly -m all, workdir: /tmp/go-modiff3838785554, stdout: , stderr: go: updates to go.mod needed, disabled by -mod=readonly; to update it:
       go mod tidy
, error: exit status 1  file="modiff/modiff.go:276"
FATA generating dependency report: getting dependency changes: unable to run cmd: go list -mod=readonly -m all, workdir: /tmp/go-modiff3838785554, stdout: , stderr: go: updates to go.mod needed, disabled by -mod=readonly; to update it:
       go mod tidy
, error: exit status 1  file="release-notes/main.go:193

Digging into it (and looking at the temporary directory created) its requiring changes to go.mod including:

toolchain go1.21.5

However, adding this causes the staging image build to fail with:

/workspace/hack/tools/go.mod:5: unknown directive: toolchain
go: errors parsing go.mod:

What did you expect to happen:

The build of the staging image and release notes should succeed.

Anything else you would like to add:

Short term we can disable the dependency report generation. Then for the future we can look to do:

• add the `toolchain1 back into go mod
• Upgrade the gcb image

Environment:

• Cluster-api-provider-aws version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[richardcase]

/triage accepted
/lifecycle active

[richardcase]

There is an issue with release-notes. It turns out if the previous version (i.e specified in --start-sha) was for a version of go prior to 1.21 and the current version (i.e. specified in --end-sha) is now v1.21 then when it checks out the starting sha there is no toolchain directive and so when go list -mod=readonly -m all is run it tries to update go.mod to add toolchain and fails.

[richardcase]

We will probably have to disable the dependency report generation short term using --dependencies=false and then re-enable it again in the next release.

I will also investigate if we can influence this behaviour via env vars.