Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[0.11] vendor github.com/containerd/containerd v1.6.20 #3736

Merged
merged 5 commits into from
Apr 13, 2023

Conversation

thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Mar 23, 2023

[0.11] vendor: github.com/Microsoft/hcsshim v0.9.8

  • wcow: support graceful termination of servercore containers

full diff: microsoft/hcsshim@v0.9.6...v0.9.8

[0.11] vendor: github.com/containerd/ttrpc v1.1.1

  • server: Fix connection leak when receiving ECONNRESET

full diff: containerd/ttrpc@v1.1.0...v1.1.1

[0.11] vendor: github.com/opencontainers/runc v1.1.5

full diff: opencontainers/runc@v1.1.3...v1.1.5

[0.11] vendor: github.com/opencontainers/image-spec v1.1.0-rc2.0.20221005185240-3a7f492d3f1b

full diff: opencontainers/image-spec@02efb9a...3a7f492

[0.11] vendor: github.com/containerd/containerd v1.6.20

full diff: containerd/containerd@v1.6.18...v1.6.20

@tonistiigi
Copy link
Member

@AkihiroSuda Could you take a look at what change is breaking the digest in the CI test. This shouldn't really break. If we test this then these digests should be actually stable between releases (unless we make an exception for security issues or obvious bugs etc.). If there is some more fundamental issue, then we need to reduce the test to only cover the stability that we can actually guarantee. If (arbitrary example), for example, containerd creates a different tar or different diff for the old parameters, then we need to fix this in containerd so that at least we can pass some parameters to get the previous behavior.

I know that there is a differ issue with hardlinks between different differs giving a different result that we discussed in another thread. I think we need to consider this as a bug and track it down, not declare that it is normal for the digest to only be stable for a specific snapshotter.

@thaJeztah thaJeztah force-pushed the 0.11_containerd_1.6.20 branch 3 times, most recently from dba0ed0 to fbd74a3 Compare March 25, 2023 14:52
Comment on lines -6565 to +6568
const expectedDigest = "sha256:9e36395384d073e711102b13bd0ba4b779ef6afbaf5cadeb77fe77dba8967d1f"
// note that this digest differs from the one in master, due to
// commit a89f482dcb3428c0297f39474eebd7de15e4792a not being included
// in this branch.
const expectedDigest = "sha256:e26093cc8a7524089a1d0136457e6c09a34176e2b2efcf99ac471baa729c7dc9"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a comment here; we'll have to update this test if we were to decide to backport that commit;

@thaJeztah
Copy link
Member Author

thaJeztah commented Mar 25, 2023

One failure on containerd-rootless; 1713

--- FAIL: TestIntegration/TestDiffCircularSymlinks/worker=containerd-rootless (78.02s)
2023-03-25T15:39:29.4356578Z === CONT  TestIntegration/TestDiffCircularSymlinks/worker=containerd-rootless
2023-03-25T15:39:29.4356897Z     client_test.go:6372: 
2023-03-25T15:39:29.4357229Z         	Error Trace:	/src/client/client_test.go:6372
2023-03-25T15:39:29.4357525Z         	            				/src/client/mergediff_test.go:1426
2023-03-25T15:39:29.4357808Z         	            				/src/client/mergediff_test.go:1270
2023-03-25T15:39:29.4358069Z         	            				/src/client/run.go:196
2023-03-25T15:39:29.4358307Z         	Error:      	content still exists
2023-03-25T15:39:29.4358751Z         	Test:       	TestIntegration/TestDiffCircularSymlinks/worker=containerd-rootless
2023-03-25T15:39:29.4359553Z         	Messages:   	[{Digest:sha256:09d8086629746f4f5da0df8742febeea8c5dd8a1c16519be974eda666c14527e Size:104727 CreatedAt:2023-03-25 15:31:54.585726429 +0000 UTC UpdatedAt:0001-01-01 00:00:00 +0000 UTC Labels:map[]}]
2023-03-25T15:39:29.4359974Z     sandbox.go:274: stderr: /usr/bin/sudo
2023-03-25T15:39:29.4361749Z     sandbox.go:277: > startCmd 2023-03-25 15:31:36.366088542 +0000 UTC m=+103.566450379 [sudo -u #1000 -i -- exec nsenter -U --preserve-credentials -m -t 19955 buildkitd --oci-worker=false --containerd-worker-gc=false --containerd-worker=true --containerd-worker-addr /tmp/bktest_containerd838830590/containerd.sock --containerd-worker-labels=org.mobyproject.buildkit.worker.sandbox=true --containerd-worker-snapshotter=native --containerd-worker-snapshotter=native --config=/tmp/bktest_config3219341177/buildkitd.toml --root /tmp/bktest_buildkitd1126559310 --addr unix:///tmp/bktest_buildkitd1126559310/buildkitd.sock --debug]
2023-03-25T15:39:29.4362995Z     sandbox.go:277: time="2023-03-25T15:31:36Z" level=debug msg="running in rootless mode"
2023-03-25T15:39:29.4363671Z     sandbox.go:277: time="2023-03-25T15:31:36Z" level=debug msg="remote introspection plugin filters" filters="[type==io.containerd.runtime.v1 type==io.containerd.runtime.v2]"
2023-03-25T15:39:29.4365893Z     sandbox.go:277: time="2023-03-25T15:31:36Z" level=info msg="found worker \"lc052lq31zez62jqhgkk5vrkq\", labels=map[org.mobyproject.buildkit.worker.containerd.namespace:buildkit org.mobyproject.buildkit.worker.containerd.uuid:c535f5ec-4096-40c0-8c3e-575a5de04b68 org.mobyproject.buildkit.worker.executor:containerd org.mobyproject.buildkit.worker.hostname:3be0a3e91f48 org.mobyproject.buildkit.worker.network:host org.mobyproject.buildkit.worker.sandbox:true org.mobyproject.buildkit.worker.selinux.enabled:false org.mobyproject.buildkit.worker.snapshotter:native], platforms=[linux/amd64 linux/amd64/v2 linux/amd64/v3 linux/amd64/v4 linux/arm64 linux/riscv64 linux/ppc64le linux/s390x linux/386 linux/mips64le linux/mips64 linux/arm/v7 linux/arm/v6]"
2023-03-25T15:39:29.4367643Z     sandbox.go:277: time="2023-03-25T15:31:36Z" level=info msg="found 1 workers, default=\"lc052lq31zez62jqhgkk5vrkq\""
2023-03-25T15:39:29.4368212Z     sandbox.go:277: time="2023-03-25T15:31:36Z" level=warning msg="currently, only the default worker can be used."
2023-03-25T15:39:29.4368808Z     sandbox.go:277: time="2023-03-25T15:31:36Z" level=info msg="running server on /tmp/bktest_buildkitd1126559310/buildkitd.sock"
2023-03-25T15:39:29.4369490Z     sandbox.go:277: time="2023-03-25T15:31:36Z" level=debug msg="skipping invalid cache export mode: " spanID=7f9e8530ebacf46b traceID=d8474acb1d0b8f4efd90ffb610d1e63d
...
2023-03-25T15:39:29.4827871Z     sandbox.go:277: time="2023-03-25T15:31:42Z" level=debug msg="session finished: <nil>" spanID=94e4b5cb4d2a6cb5 traceID=a18cf2f26018d6ba89c6f48031fe6493
2023-03-25T15:39:29.4828518Z     sandbox.go:277: time="2023-03-25T15:31:44Z" level=error msg="failed to save trace for m6epqaxiipk0ic9737klgawgw: file does not exist"
2023-03-25T15:39:29.4829239Z     sandbox.go:277: time="2023-03-25T15:31:45Z" level=debug msg="session started" spanID=4bd3dad9d6752039 traceID=76e04a4032cb052596abad9506f3e05e
2023-03-25T15:39:29.4829902Z     sandbox.go:277: time="2023-03-25T15:31:45Z" level=debug msg=resolving host="localhost:46113"
...
...
2023-03-25T15:39:29.5631199Z     sandbox.go:277: time="2023-03-25T15:31:51Z" level=debug msg="session finished: <nil>" spanID=31545117a951505f traceID=e9110d3ac938420634586d513995037b
2023-03-25T15:39:29.5631596Z     sandbox.go:277: time="2023-03-25T15:31:53Z" level=error msg="failed to save trace for qrid7cn3qoedc5u3w6ql8jtv6: file does not exist"
2023-03-25T15:39:29.5631967Z     sandbox.go:277: time="2023-03-25T15:31:54Z" level=error msg="failed to save trace for fx12cuekp3wjsmb4dbkx5y64y: file does not exist"
2023-03-25T15:39:29.5632083Z     sandbox.go:274: stdout: /usr/bin/sudo
level=debug msg="skipping invalid cache export mode: " spanID=7f9e8530ebacf46b traceID=d8474acb1d0b8f4efd90ffb610d1e63d

That error / warning comes from

buildkit/control/control.go

Lines 380 to 384 in 252ae63

if exportMode, supported := parseCacheExportMode(e.Attrs["mode"]); !supported {
bklog.G(ctx).Debugf("skipping invalid cache export mode: %s", e.Attrs["mode"])
} else {
exp.CacheExportMode = exportMode
}

But looks like it's empty (?). Perhaps needs one of bcdad5d or e849b62 to ignore

@crazy-max
Copy link
Member

2023-03-25T15:39:29.4358307Z         	Error:      	content still exists

Related to #3401

@thaJeztah
Copy link
Member Author

2023-03-25T15:39:29.4358307Z         	Error:      	content still exists

Related to #3401

Ah, thanks!

Looks like it's green now. I'll update things once containerd 1.6.20 is released (and clean up the commit message etc)

laurazard added a commit to laurazard/moby that referenced this pull request Mar 29, 2023
(see moby/buildkit#3736)

Signed-off-by: Laura Brehm <laurabrehm@hey.com>
laurazard added a commit to laurazard/moby that referenced this pull request Mar 29, 2023
(see moby/buildkit#3736)

Signed-off-by: Laura Brehm <laurabrehm@hey.com>
(cherry picked from commit 5a1cc36)
Signed-off-by: Laura Brehm <laurabrehm@hey.com>
laurazard added a commit to laurazard/moby that referenced this pull request Mar 29, 2023
(see moby/buildkit#3736)

Signed-off-by: Laura Brehm <laurabrehm@hey.com>
laurazard added a commit to laurazard/moby that referenced this pull request Mar 30, 2023
(see moby/buildkit#3736)

Signed-off-by: Laura Brehm <laurabrehm@hey.com>
laurazard added a commit to laurazard/moby that referenced this pull request Mar 30, 2023
(see moby/buildkit#3736)

Signed-off-by: Laura Brehm <laurabrehm@hey.com>
- wcow: support graceful termination of servercore containers

full diff: microsoft/hcsshim@v0.9.6...v0.9.8

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- server: Fix connection leak when receiving ECONNRESET

full diff: containerd/ttrpc@v1.1.0...v1.1.1

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: opencontainers/runc@v1.1.3...v1.1.5

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
…1005185240-3a7f492d3f1b

full diff: opencontainers/image-spec@02efb9a...3a7f492

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: containerd/containerd@v1.6.18...v1.6.20

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah thaJeztah changed the title [0.11] vendor github.com/containerd/containerd v1.6.20-0.20230322235238-de33abf0547c [0.11] vendor github.com/containerd/containerd v1.6.20 Apr 1, 2023
@thaJeztah
Copy link
Member Author

Rebased this one, and updated to containerd 1.6.20

@thaJeztah thaJeztah marked this pull request as ready for review April 1, 2023 14:07
@thaJeztah
Copy link
Member Author

Looks like 3 tests failed on rootless; I think these may the same as last time #3736 (comment) but I'll give it another run to see if it was a flaky

https://pipelines.actions.githubusercontent.com/serviceHosts/2ff2ce03-99e5-4646-9775-5b110c91788d/_apis/pipelines/1/runs/11514/signedlogcontent/66?urlExpires=2023-04-01T13%3A10%3A29.7674467Z&urlSigningMethod=HMACV1&urlSignature=8LkUM8tNHTLY87X1AQbegHw4xjJ4FeH08xyHLgQj718%3D

2023-04-01T13:07:10.9567204Z --- FAIL: TestIntegration (3.93s)
...
2023-04-01T13:07:10.9580200Z     --- FAIL: TestIntegration/TestDiffCircularParentDirSymlinks/worker=containerd-rootless (46.40s)
...
2023-04-01T13:07:10.9581549Z     --- FAIL: TestIntegration/TestDiffSymlinkOverridesSymlink/worker=containerd-rootless (93.92s)
...
2023-04-01T13:07:10.9583081Z     --- FAIL: TestIntegration/TestDiffDirOverridesSymlink/worker=containerd-rootless (100.15s)
=== CONT  TestIntegration/TestDiffCircularParentDirSymlinks/worker=containerd-rootless
    client_test.go:6372: 
        	Error Trace:	/src/client/client_test.go:6372
        	            				/src/client/mergediff_test.go:1426
        	            				/src/client/mergediff_test.go:1270
        	            				/src/client/run.go:196
        	Error:      	content still exists
        	Test:       	TestIntegration/TestDiffCircularParentDirSymlinks/worker=containerd-rootless
        	Messages:   	[{Digest:sha256:87b0ec868f807b3b6ba95e6fcdad7dec43a3a6173bfd83ead04c52b003572185 Size:104739 CreatedAt:2023-04-01 12:58:59.427135244 +0000 UTC UpdatedAt:0001-01-01 00:00:00 +0000 UTC Labels:map[]}]
    sandbox.go:274: stdout: /usr/bin/sudo
    sandbox.go:274: stderr: /usr/bin/sudo
    sandbox.go:277: > startCmd 2023-04-01 12:58:49.336923129 +0000 UTC m=+80.186422101 [sudo -u #1000 -i -- exec nsenter -U --preserve-credentials -m -t 18350 buildkitd --oci-worker=false --containerd-worker-gc=false --containerd-worker=true --containerd-worker-addr /tmp/bktest_containerd3928964251/containerd.sock --containerd-worker-labels=org.mobyproject.buildkit.worker.sandbox=true --containerd-worker-snapshotter=native --containerd-worker-snapshotter=native --config=/tmp/bktest_config3855782646/buildkitd.toml --root /tmp/bktest_buildkitd1423300277 --addr unix:///tmp/bktest_buildkitd1423300277/buildkitd.sock --debug]
...
=== CONT  TestIntegration/TestDiffSymlinkOverridesSymlink/worker=containerd-rootless
    client_test.go:6372: 
        	Error Trace:	/src/client/client_test.go:6372
        	            				/src/client/mergediff_test.go:1426
        	            				/src/client/mergediff_test.go:1270
        	            				/src/client/run.go:196
        	Error:      	content still exists
        	Test:       	TestIntegration/TestDiffSymlinkOverridesSymlink/worker=containerd-rootless
        	Messages:   	[{Digest:sha256:aba4194e271c18f4b5fb2e1e7586ea071eb1cf46a626dfae0278925478d8da5d Size:104742 CreatedAt:2023-04-01 13:00:02.031429927 +0000 UTC UpdatedAt:0001-01-01 00:00:00 +0000 UTC Labels:map[]}]
    sandbox.go:274: stdout: /usr/bin/sudo
    sandbox.go:274: stderr: /usr/bin/sudo
    sandbox.go:277: > startCmd 2023-04-01 12:59:52.734449095 +0000 UTC m=+143.583948167 [sudo -u #1000 -i -- exec nsenter -U --preserve-credentials -m -t 21495 buildkitd --oci-worker=false --containerd-worker-gc=false --containerd-worker=true --containerd-worker-addr /tmp/bktest_containerd2033659447/containerd.sock --containerd-worker-labels=org.mobyproject.buildkit.worker.sandbox=true --containerd-worker-snapshotter=native --containerd-worker-snapshotter=native --config=/tmp/bktest_config3551577955/buildkitd.toml --root /tmp/bktest_buildkitd3571323943 --addr unix:///tmp/bktest_buildkitd3571323943/buildkitd.sock --debug]
...
=== CONT  TestIntegration/TestDiffDirOverridesSymlink/worker=containerd-rootless
    client_test.go:6372: 
        	Error Trace:	/src/client/client_test.go:6372
        	            				/src/client/mergediff_test.go:1426
        	            				/src/client/mergediff_test.go:1270
        	            				/src/client/run.go:196
        	Error:      	content still exists
        	Test:       	TestIntegration/TestDiffDirOverridesSymlink/worker=containerd-rootless
        	Messages:   	[{Digest:sha256:521bbd365092c7092328896635882359706391c59d1f9f8fe3eb349ce6e33eeb Size:104730 CreatedAt:2023-04-01 13:00:39.578488245 +0000 UTC UpdatedAt:0001-01-01 00:00:00 +0000 UTC Labels:map[]}]
    sandbox.go:274: stdout: /usr/bin/sudo
    sandbox.go:274: stderr: /usr/bin/sudo
    sandbox.go:277: > startCmd 2023-04-01 13:00:27.227680161 +0000 UTC m=+178.077179233 [sudo -u #1000 -i -- exec nsenter -U --preserve-credentials -m -t 23136 buildkitd --oci-worker=false --containerd-worker-gc=false --containerd-worker=true --containerd-worker-addr /tmp/bktest_containerd2242760103/containerd.sock --containerd-worker-labels=org.mobyproject.buildkit.worker.sandbox=true --containerd-worker-snapshotter=native --containerd-worker-snapshotter=native --config=/tmp/bktest_config2385712047/buildkitd.toml --root /tmp/bktest_buildkitd3610843316 --addr unix:///tmp/bktest_buildkitd3610843316/buildkitd.sock --debug]
...

@thaJeztah
Copy link
Member Author

Looks like it's green now; @AkihiroSuda @tonistiigi ptal

@tonistiigi
Copy link
Member

If this completely changes the output that differ produces, then I don't think we can pick anything like this for a patch release. Why would they backport such a huge behavior change?

@thaJeztah
Copy link
Member Author

Looking at;

I guess previously it would already produce a different (unpredictable) sha, depending on what's on the host. I guess that wouldn't reproduce in CI as long as CI always had the same users in /etc/password

@tonistiigi
Copy link
Member

tonistiigi commented Apr 3, 2023

That is true. Although for most of images, it would have just been root:root strings. Also, moby/buildkit:latest image always has the static /etc/passwd.

@thaJeztah
Copy link
Member Author

I guess a workaround for the "most common" case would be to hard-code "0:0" to "root:root" as only lookup, to "somewhat" keep the happy path the same. But for any other scenario it would already be horribly broken 😞

Copy link
Member

@tonistiigi tonistiigi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is way too big for my liking(especially because we already had a regression in a patch release) but I don't think we have another choice than to pick it.

@AkihiroSuda @jedevc Please verify as well.

Copy link
Member

@jedevc jedevc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The digest changes from containerd/containerd@063ad2f do worry me a little bit - ideally we wouldn't take this change for v0.11. We've already had minor version bumps break users in v0.11 already 😢

Is the reason for this change that we have a different version of containerd in moby? If the issue is just the tests giving different digests in those tests, could we simply disable those tests for the time being?

That said, glancing through changelogs, nothing springs out as otherwise dangerous - I'll give a tentative approval.

@tonistiigi
Copy link
Member

If the issue is just the tests giving different digests in those tests, could we simply disable those tests for the time being?

It is not just the tests, but if there are separate dependencies in moby and buildkitd it could lead to different behavior and fragmentation.

@thaJeztah
Copy link
Member Author

Opened moby/moby#45326 to test the latest changes in moby/moby

@tonistiigi
Copy link
Member

@thaJeztah We should update the binary as well for consistency, right?

@thaJeztah
Copy link
Member Author

Here in BuildKit CI? Yes, probably.

@thaJeztah
Copy link
Member Author

Ugh.. just checked the Moby PR with this change, and looks like it's still failing 😞 with yet another digest now? No clue now what's happening.

=== CONT  TestIntegration/TestReproSourceDateEpoch/worker=dockerd-containerd/frontend=builtin
    dockerfile_test.go:6540: SOURCE_DATE_EPOCH=1673354096
    dockerfile_test.go:6607: OCI archive digest="sha256:aa2d0a0f9a6560c267b0c2d41c758ca60386d6001cd687adf837365236507a0a"
    dockerfile_test.go:6608: The digest may change depending on the BuildKit version, the snapshotter configuration, etc.
    dockerfile_test.go:6609: 
        	Error Trace:	/src/frontend/dockerfile/dockerfile_test.go:6609
        	            				/src/frontend/dockerfile/run.go:87
        	            				/src/frontend/dockerfile/run.go:196
        	Error:      	Not equal: 
        	            	expected: "sha256:e26093cc8a7524089a1d0136457e6c09a34176e2b2efcf99ac471baa729c7dc9"
        	            	actual  : "sha256:aa2d0a0f9a6560c267b0c2d41c758ca60386d6001cd687adf837365236507a0a"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-sha256:e26093cc8a7524089a1d0136457e6c09a34176e2b2efcf99ac471baa729c7dc9
        	            	+sha256:aa2d0a0f9a6560c267b0c2d41c758ca60386d6001cd687adf837365236507a0a
        	Test:       	TestIntegration/TestReproSourceDateEpoch/worker=dockerd-containerd/frontend=builtin

@tonistiigi
Copy link
Member

@thaJeztah Did this test ever pass in moby?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants