[release/1.7] Add API go module #10189

dmcgowan · 2024-05-08T04:02:27Z

Add api go module to 1.7. The plan is we do not need to update 1.7 to the 1.8 api go module if we separate it into its like we have done with main. The changes needed are minimal.

This helps the upgrade path from 1.7 to 2.0. After updating to 1.7, it should be easier to update to 2.0 since the api module can be just be upgraded to 1.8 and the latest containerd 1.7 will use the api package from the module.

For those not upgrading to 2.0 right away, the change should be invisible beside an extra dependency showing up in go module. We shouldn't need to do this for 1.6 since this only effects importers who are upgrading. In cases where an import is trying to stay on 1.6 but a dependency uses the latest version of 1.7, go mod would already forces the upgrade anyway.

k8s-ci-robot · 2024-05-08T04:02:30Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

dmcgowan · 2024-05-17T03:24:16Z

Let's get this one into the next 1.7 release

thaJeztah · 2024-05-28T16:02:26Z

Trying this PR in moby/moby (through a replace rule), but looks like it's not happy. Perhaps I need to force a version for the API module, or something else (or the API package must have aliases in it; or removed, not sure;

hack/vendor.sh
+ /Users/thajeztah/go/src/github.com/docker/docker/hack/with-go-mod.sh go mod tidy -modfile vendor.mod -compat 1.18
with-go-mod.sh: WARN: go.mod exists in the repository root!
with-go-mod.sh: WARN: Using your go.mod instead of our generated version -- this may misbehave!
go: downloading github.com/dmcgowan/containerd v1.7.16-0.20240528144541-4a2ca38f6508
go: downloading github.com/containerd/containerd/api v0.0.0
go: github.com/docker/docker/libcontainerd/remote imports
	github.com/containerd/containerd/api/events: reading github.com/containerd/containerd/api/go.mod at revision api/v0.0.0: unknown revision api/v0.0.0
go: github.com/docker/docker/libcontainerd/remote imports
	github.com/containerd/containerd/api/types: reading github.com/containerd/containerd/api/go.mod at revision api/v0.0.0: unknown revision api/v0.0.0
go: github.com/docker/docker/builder/builder-next imports
	github.com/containerd/containerd imports
	github.com/containerd/containerd/api/services/containers/v1: reading github.com/containerd/containerd/api/go.mod at revision api/v0.0.0: unknown revision api/v0.0.0
go: github.com/docker/docker/builder/builder-next imports
	github.com/containerd/containerd imports
	github.com/containerd/containerd/api/services/content/v1: reading github.com/containerd/containerd/api/go.mod at revision api/v0.0.0: unknown revision api/v0.0.0
go: github.com/docker/docker/builder/builder-next imports
	github.com/containerd/containerd imports
	github.com/containerd/containerd/api/services/diff/v1: reading github.com/containerd/containerd/api/go.mod at revision api/v0.0.0: unknown revision api/v0.0.0
go: github.com/docker/docker/builder/builder-next imports
	github.com/containerd/containerd imports
	github.com/containerd/containerd/api/services/events/v1: reading github.com/containerd/containerd/api/go.mod at revision api/v0.0.0: unknown revision api/v0.0.0
go: github.com/docker/docker/builder/builder-next imports
	github.com/containerd/containerd imports
	github.com/containerd/containerd/api/services/images/v1: reading github.com/containerd/containerd/api/go.mod at revision api/v0.0.0: unknown revision api/v0.0.0
go: github.com/docker/docker/builder/builder-next imports
	github.com/containerd/containerd imports
	github.com/containerd/containerd/api/services/introspection/v1: reading github.com/containerd/containerd/api/go.mod at revision api/v0.0.0: unknown revision api/v0.0.0
go: github.com/docker/docker/builder/builder-next imports
	github.com/containerd/containerd imports
	github.com/containerd/containerd/api/services/leases/v1: reading github.com/containerd/containerd/api/go.mod at revision api/v0.0.0: unknown revision api/v0.0.0
go: github.com/docker/docker/builder/builder-next imports
	github.com/containerd/containerd imports
	github.com/containerd/containerd/api/services/namespaces/v1: reading github.com/containerd/containerd/api/go.mod at revision api/v0.0.0: unknown revision api/v0.0.0

thaJeztah · 2024-05-28T16:04:52Z

OH! Maybe it works; but it's just version resolution; I manually added the API module, and that ... seems to go better.

Only now it enforces go1.22, because that's what the API module defines?

containerd/api/go.mod

Line 3 in 114ef75

go 1.22.0

go 1.22.0

toolchain go1.22.2

dmcgowan · 2024-05-28T20:11:05Z

@thaJeztah there is going to be no way to actually use this new API submodule until this PR gets merged. I could possibly push the dev branch into this repo so we can use the hash, however, it seems easier to just get this merged knowing there will be follows ups with an updated tag.

For the 1.22, that is currently on the api 1.8 which we could update, but it is the version we are going to import in 1.7 right now.

dmcgowan · 2024-05-29T22:36:04Z

Added one more commit to so 1.7 is compatible with the 1.7 and 1.8 api. This PR is testing with 1.7 and #10278 shows it working with the latest 1.8 rc.

This should be ready to merge now.

thaJeztah · 2024-06-03T17:41:01Z

I updated my PR in moby/moby to use both the containerd module, and the API module from this branch, and it looks happy; moby/moby#47142

I can give it one more run, but with the API module from main (to be "rc.3" - with #10279 included) if you think we should check that.

thaJeztah · 2024-06-13T11:06:12Z

@dmcgowan can you rebase this one?

Signed-off-by: Akhil Mohan <akhilerm@gmail.com> (cherry picked from commit b16e357) Signed-off-by: Derek McGowan <derek@mcg.dev>

Signed-off-by: Derek McGowan <derek@mcg.dev>

Allows the api version to be imported and upgraded separately from the main module. Signed-off-by: Derek McGowan <derek@mcg.dev>

Signed-off-by: Derek McGowan <derek@mcg.dev> (cherry picked from commit e69efd5) Signed-off-by: Derek McGowan <derek@mcg.dev>

Signed-off-by: Derek McGowan <derek@mcg.dev>

thaJeztah · 2024-06-13T13:42:33Z

testing in BuildKit (first run, with API v1.8.0-rc.2); [WIP] update to containerd 1.7 with separate API module moby/buildkit#5036

dims · 2024-06-17T20:26:17Z

+1 from me!

thaJeztah

LGTM

the "vendoring itself" is a bit awkward, but I guess that's transitional

thaJeztah · 2024-06-19T14:37:31Z

api/go.mod

+require (
+	github.com/containerd/ttrpc v1.2.3
+	github.com/containerd/typeurl/v2 v2.1.1
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20230822172742-b8732ec3820d
+	google.golang.org/grpc v1.59.0
+	google.golang.org/protobuf v1.33.0
+)
+
+require (
+	github.com/gogo/protobuf v1.3.2 // indirect
+	github.com/golang/protobuf v1.5.3 // indirect
+	github.com/sirupsen/logrus v1.8.1 // indirect
+	github.com/stretchr/testify v1.7.0 // indirect
+	golang.org/x/net v0.21.0 // indirect
+	golang.org/x/sys v0.17.0 // indirect
+	golang.org/x/text v0.14.0 // indirect
+	gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b // indirect
+)


One thing I'm looking at, and to some extent it's "moot", because this module will be used with containerd 1.7, so dependencies will already be updated based on that, but I noticed some of these are somewhat "behind" on containerd 1.7 itself, and at least one of them (gopkg.in/yaml.v3) is an untagged release, which has a vulnerability (so the api module may get flagged by over-enthusiastic scanners)

go mod graph | grep ' gopkg.in/yaml.v3' github.com/containerd/containerd/api gopkg.in/yaml.v3@v3.0.0-20210107192922-496545a6307b github.com/stretchr/testify@v1.7.0 gopkg.in/yaml.v3@v3.0.0-20200313102051-9f266ea9e77c go mod graph | grep ' github.com/sirupsen/logrus' github.com/containerd/containerd/api github.com/sirupsen/logrus@v1.8.1 github.com/containerd/ttrpc@v1.2.3 github.com/sirupsen/logrus@v1.8.1

Looking at the release/1.7 branch itself (so containerd/containerd v1.7);

containerd/go.mod

Line 24 in 0a9ac76

github.com/containerd/ttrpc v1.2.4

containerd/go.mod

Line 143 in 0a9ac76

gopkg.in/yaml.v3 v3.0.1 // indirect

containerd/go.mod

Lines 58 to 59 in 0a9ac76

github.com/sirupsen/logrus v1.9.3

github.com/stretchr/testify v1.8.4

Updating ttrpc to v1.2.4 would at least address one vulnerability detected by govulncheck (as it would update golang.org/x/net); containerd/ttrpc@v1.2.3...v1.2.4

govulncheck ./... Scanning your code and 251 packages across 13 dependent modules for known vulnerabilities... === Symbol Results === Vulnerability #1: GO-2024-2687 HTTP/2 CONTINUATION flood in net/http More info: https://pkg.go.dev/vuln/GO-2024-2687 Module: golang.org/x/net Found in: golang.org/x/net@v0.21.0 Fixed in: golang.org/x/net@v0.23.0 Example traces found: #1: events/task_fieldpath.pb.go:85:20: events.TaskIO.Field calls fmt.Sprint, which eventually calls http2.ConnectionError.Error #2: events/task_fieldpath.pb.go:85:20: events.TaskIO.Field calls fmt.Sprint, which eventually calls http2.ErrCode.String #3: events/task_fieldpath.pb.go:85:20: events.TaskIO.Field calls fmt.Sprint, which eventually calls http2.FrameHeader.String #4: events/task_fieldpath.pb.go:85:20: events.TaskIO.Field calls fmt.Sprint, which eventually calls http2.FrameType.String #5: events/task_fieldpath.pb.go:85:20: events.TaskIO.Field calls fmt.Sprint, which eventually calls http2.Setting.String #6: events/task_fieldpath.pb.go:85:20: events.TaskIO.Field calls fmt.Sprint, which eventually calls http2.SettingID.String #7: events/task_fieldpath.pb.go:85:20: events.TaskIO.Field calls fmt.Sprint, which eventually calls http2.StreamError.Error #8: services/version/v1/version_grpc.pb.go:13:2: version.init calls status.init, which eventually calls http2.chunkWriter.Write #9: events/task_fieldpath.pb.go:85:20: events.TaskIO.Field calls fmt.Sprint, which eventually calls http2.connError.Error #10: events/task_fieldpath.pb.go:85:20: events.TaskIO.Field calls fmt.Sprint, which eventually calls http2.duplicatePseudoHeaderError.Error #11: events/task_fieldpath.pb.go:85:20: events.TaskIO.Field calls fmt.Sprint, which eventually calls http2.headerFieldNameError.Error #12: events/task_fieldpath.pb.go:85:20: events.TaskIO.Field calls fmt.Sprint, which eventually calls http2.headerFieldValueError.Error #13: events/task_fieldpath.pb.go:85:20: events.TaskIO.Field calls fmt.Sprint, which eventually calls http2.pseudoHeaderError.Error #14: events/task_fieldpath.pb.go:85:20: events.TaskIO.Field calls fmt.Sprint, which eventually calls http2.writeData.String Your code is affected by 1 vulnerability from 1 module. This scan also found 0 vulnerabilities in packages you import and 3 vulnerabilities in modules you require, but your code doesn't appear to call these vulnerabilities. Use '-show verbose' for more details.

We could update testify to a newer version, to get the untagged gopkg.in/yaml.v3@v3.0.0-20210107192922-496545a6307b out of the list, and to prevent any warnings about CVE-2022-28948; see f6bc986

Lastly, it's probably fine to update logrus to latest (which we should do in ttrpc as well).

OH LOL; just got merged.

It was fine for a follow-up either way; I can open a PR for consideration 😄

thaJeztah · 2024-06-19T14:40:55Z

@dmcgowan I just noticed that make vendor doesn't like this; is there some trick to make it work?

make vendor
🇩 vendor
go: finding module for package github.com/containerd/containerd/v2/pkg/tracing
go: downloading github.com/containerd/containerd/v2 v2.0.0-rc.3
go: found github.com/containerd/containerd/v2/pkg/tracing in github.com/containerd/containerd/v2 v2.0.0-rc.3
go: downloading github.com/onsi/ginkgo v1.11.0
go: finding module for package github.com/containerd/containerd/api/types/runc/options
go: github.com/containerd/containerd/pkg/cri/sbserver imports
	github.com/containerd/imgcrypt/images/encryption imports
	github.com/containerd/containerd/v2/client imports
	github.com/containerd/containerd/api/types/runc/options: module github.com/containerd/containerd/api@latest found (v1.8.0-rc.2, replaced by ./api), but does not contain package github.com/containerd/containerd/api/types/runc/options
make: *** [vendor] Error 1

dmcgowan · 2024-06-19T14:44:23Z

@thaJeztah We need a tagged release and dependencies to bump 1.7 dependencies. Which make vendor do you see that in?

thaJeztah · 2024-06-19T14:47:39Z

OH! Nevermind ; ignore me; looks like I had a stray file in my code that imported v2 🙈

thaJeztah · 2024-06-19T14:57:05Z

opened [release/1.7]: api: update dependencies to be aligned with containerd 1.7 module #10364

k8s-ci-robot added do-not-merge/work-in-progress size/XXL labels May 8, 2024

dmcgowan force-pushed the 1.7-api-go-module branch 2 times, most recently from 41cb738 to 7ee92ff Compare May 8, 2024 04:39

k8s-ci-robot added the needs-rebase label May 14, 2024

dmcgowan force-pushed the 1.7-api-go-module branch from 7ee92ff to ddc2a76 Compare May 14, 2024 18:26

k8s-ci-robot removed the needs-rebase label May 15, 2024

dmcgowan force-pushed the 1.7-api-go-module branch from ddc2a76 to 943a70a Compare May 15, 2024 04:33

dmcgowan added the impact/changelog label May 15, 2024

dmcgowan marked this pull request as ready for review May 17, 2024 03:23

k8s-ci-robot removed the do-not-merge/work-in-progress label May 17, 2024

k8s-ci-robot added the needs-rebase label May 27, 2024

dmcgowan force-pushed the 1.7-api-go-module branch from 943a70a to 5be6019 Compare May 27, 2024 04:57

k8s-ci-robot removed the needs-rebase label May 27, 2024

dmcgowan force-pushed the 1.7-api-go-module branch from 5be6019 to 4a2ca38 Compare May 28, 2024 14:46

thaJeztah mentioned this pull request May 28, 2024

[WIP] Migrate to github.com/containerd/platforms module moby/moby#47142

Draft

9 tasks

akhilerm and others added 5 commits June 13, 2024 06:22

refactor: move plugin/fieldpath to api/types/

4b82470

Signed-off-by: Akhil Mohan <akhilerm@gmail.com> (cherry picked from commit b16e357) Signed-off-by: Derek McGowan <derek@mcg.dev>

Alias protobuf plugin to new api types package

61b3e22

Signed-off-by: Derek McGowan <derek@mcg.dev>

Add api go submodule

a3a7431

Allows the api version to be imported and upgraded separately from the main module. Signed-off-by: Derek McGowan <derek@mcg.dev>

Add go mod replace when proto changes happen

5b87eb5

Signed-off-by: Derek McGowan <derek@mcg.dev> (cherry picked from commit e69efd5) Signed-off-by: Derek McGowan <derek@mcg.dev>

Add support for 1.8 interfaces

3be919f

Signed-off-by: Derek McGowan <derek@mcg.dev>

dmcgowan force-pushed the 1.7-api-go-module branch from 9df456d to 3be919f Compare June 13, 2024 13:22

thaJeztah mentioned this pull request Jun 13, 2024

[WIP] update to containerd 1.7 with separate API module moby/buildkit#5036

Closed

thaJeztah approved these changes Jun 18, 2024

View reviewed changes

fuweid approved these changes Jun 19, 2024

View reviewed changes

dmcgowan merged commit 1fdc9a0 into containerd:release/1.7 Jun 19, 2024
57 checks passed

thaJeztah reviewed Jun 19, 2024

View reviewed changes

thaJeztah mentioned this pull request Jun 19, 2024

[release/1.7]: api: update dependencies to be aligned with containerd 1.7 module #10364

Open

thaJeztah mentioned this pull request Jun 19, 2024

[release/1.7] remove imports of errdefs package, and add depguard linter #10367

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[release/1.7] Add API go module #10189

[release/1.7] Add API go module #10189

dmcgowan commented May 8, 2024

k8s-ci-robot commented May 8, 2024

dmcgowan commented May 17, 2024

thaJeztah commented May 28, 2024

thaJeztah commented May 28, 2024 •

edited

dmcgowan commented May 28, 2024

dmcgowan commented May 29, 2024

thaJeztah commented Jun 3, 2024

thaJeztah commented Jun 13, 2024

thaJeztah commented Jun 13, 2024

dims commented Jun 17, 2024

thaJeztah left a comment

thaJeztah Jun 19, 2024

thaJeztah Jun 19, 2024

thaJeztah commented Jun 19, 2024

dmcgowan commented Jun 19, 2024

thaJeztah commented Jun 19, 2024

thaJeztah commented Jun 19, 2024

	github.com/sirupsen/logrus v1.9.3
	github.com/stretchr/testify v1.8.4

[release/1.7] Add API go module #10189

[release/1.7] Add API go module #10189

Conversation

dmcgowan commented May 8, 2024

k8s-ci-robot commented May 8, 2024

dmcgowan commented May 17, 2024

thaJeztah commented May 28, 2024

thaJeztah commented May 28, 2024 • edited

dmcgowan commented May 28, 2024

dmcgowan commented May 29, 2024

thaJeztah commented Jun 3, 2024

thaJeztah commented Jun 13, 2024

thaJeztah commented Jun 13, 2024

dims commented Jun 17, 2024

thaJeztah left a comment

Choose a reason for hiding this comment

thaJeztah Jun 19, 2024

Choose a reason for hiding this comment

thaJeztah Jun 19, 2024

Choose a reason for hiding this comment

thaJeztah commented Jun 19, 2024

dmcgowan commented Jun 19, 2024

thaJeztah commented Jun 19, 2024

thaJeztah commented Jun 19, 2024

thaJeztah commented May 28, 2024 •

edited