Nightly CI is failing integration tests

[jsturtevant]

Looks like nightly CI started failing last week: https://github.com/containerd/rust-extensions/actions with the first failure in https://github.com/containerd/rust-extensions/actions/runs/5273246763

Originally posted by @jsturtevant in #139 (comment)

Interestingly it was passing and is now failing on the same commit: 35a97f1 so something else in the dependency chain must have changed.

[jsturtevant]

The tests failing are consistent:

2023-06-15T00:17:12.6609993Z --- FAIL: TestContainerAttachProcess (0.16s)
2023-06-15T00:17:14.0144827Z --- FAIL: TestContainerAttach (0.10s)

Comparing https://github.com/containerd/rust-extensions/actions/runs/5261500598/jobs/9509676956 (passing) and https://github.com/containerd/rust-extensions/actions/runs/5273246763/jobs/9536435763 (failing)

The rust version is the same

info: latest update on 2023-01-10, rust version 1.66.1 (90743e729 2023-01-10)

The containerd commit is the same for refs/tags/v1.6.21

3dce8eb055cbb6872793272b4f20ed16117344f8

[jsturtevant]

go version is different:

Failing;

go version go1.20.5 linux/amd64

passing:

go version go1.20.4 linux/amd64

[jsturtevant]

based on #148 it does appear this was related to a golang patch update 😬

[jsturtevant]

Containerd 1.6.21 is built with golang 1.19.9
https://github.com/containerd/containerd/blob/v1.6.21/.github/workflows/release.yml

Is there a reason to build a released version vs downloading the released binary?

[jsturtevant]

#148 unblocks CI for PR's.

@mxpv Do we want to put a longer term fix in, such as downloading the released binary or run with the correct version of golang? We still need to checkout the containerd code to run the tests.

[mxpv]

@jsturtevant Yeah, downloading the release binary is fine too (and easier).

[jsturtevant]

Similar but different issue now causing CI to fail nightly and blocking PR's

https://github.com/containerd/rust-extensions/actions/workflows/ci.yml?query=event%3Aschedule

go: github.com/AdaLogics/go-fuzz-headers@v0.0.0-20230106234847-43070de90fa1 (replaced by 
github.com/AdamKorcz/go-fuzz-headers-1@v0.0.0-20230111232327-1f10f66a31bf): version "v0.0.0-20230111232327-1f10f66a31bf" invalid: unknown revision 1f10f66a31bf

[jsturtevant]

go get on the replaced revision is working for me:

go get github.com/AdamKorcz/go-fuzz-headers-1@v0.0.0-20230111232327-1f10f66a31bf
go: downloading github.com/AdamKorcz/go-fuzz-headers-1 v0.0.0-20230111232327-1f10f66a31bf
go: added github.com/AdamKorcz/go-fuzz-headers-1 v0.0.0-20230111232327-1f10f66a31bf
go: added github.com/cyphar/filepath-securejoin v0.2.3

[jsturtevant]

I reporduced with

go clean -modcache
go env -w GOPROXY=direct
go get  github.com/AdamKorcz/go-fuzz-headers-1@v0.0.0-20230111232327-1f10f66a31bf
go: github.com/AdamKorcz/go-fuzz-headers-1@v0.0.0-20230111232327-1f10f66a31bf: invalid version: unknown revision 1f10f66a31bf

and fixed with the proxy:

GOPROXY=https://proxy.golang.org  go get  github.com/AdamKorcz/go-fuzz-headers-1@v0.0.0-20230111232327-1f10f66a31bf                                     
 go: downloading github.com/AdamKorcz/go-fuzz-headers-1 v0.0.0-20230111232327-1f10f66a31bf   
 go: downloading github.com/cyphar/filepath-securejoin v0.2.3 

[jsturtevant]

so it looks like this is stemming from a transient dependencies through

https://github.com/containerd/imgcrypt/blob/v1.1.7/go.mod#L9

which brings in

https://github.com/containerd/containerd/blob/9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6/integration/client/go.sum#L37

[jsturtevant]

Noticed that github.com/AdamKorcz/go-fuzz-headers-1 is different from github.com/AdaLogics/go-fuzz-headers and that came from containerd/containerd#7957

The CI started failing on Aug 11 when AdaLogics/go-fuzz-headers#113 merged

[jsturtevant]

The following fixes that error:

go mod edit -dropreplace github.com/AdaLogics/go-fuzz-headers
go get -u github.com/AdaLogics/go-fuzz-headers