Blockers for v1.2.0

[AkihiroSuda]

Blockers for rc.1 (#3963):

• libcontainer: dmz: fix "go get" builds #4101
• libcontainer: remove all mount logic from nsexec #3985
• configs: validate: add validation for bind-mount fsflags #3990
• runc-dmz: Inheritable capabilities are dropped when they previously weren't #4125
• Fix runc-dmz error printing #4172
• never send procError after the socket closed #4173

Blockers for GA:

• Tests broken in debian due to PR: rootfs: remove --no-mount-fallback and finally fix MS_REMOUNT #4093
• Release runtime-spec v1.1.1 v1.2.0 Release v1.2.0 runtime-spec#1242
• runc doesn't work with go1.22  #4233
• Revert "Set temporary single CPU affinity before cgroup cpuset transition" #4283 (comment)
• Verify compatibility with well-known downstreams
  • Dockerfile: update to runc v1.2.0-rc.2 moby/moby#47666
    • Fix failing moby tests in CI Moby CI is failing due to stdio issue introduced in f2f16213 init: close internal fds before execve #4384
  • update runc (1.2.0) containerd/nerdctl#3153
    • [v1.2 regression] SCMP_ACT_NOTIFY rule for fcntl causes runc to hang, before connecting to the seccomp listener agent #4328
    • [v1.2 regression] [cgroup v1 + rootless] nerdctl run -d --name=bar --pid=container:foo ; nerdctl rm -f bar hangs #4394
  • containerd works fine, CI scripts need fixes
    • PR to fix containerd's CI scripts: CI fixes for upcoming runc 1.2.0 containerd/containerd#10449
    • cri-tools PR backported for v1.30.1: [release-1.30] userns: Skip tests if the host doesn't support idmap mounts kubernetes-sigs/cri-tools#1492
• Fd resue race causes runc init can't start due to a go stdlib bug #4294

Needs discussion (probably deferrable to v1.2.1 or v1.3.0):

• Issue joining cgroups cpuset with kernel scheduler task "random" distribution #3922

@opencontainers/runc-maintainers Feel free to edit this issue

[AkihiroSuda]

We don't have timeline for v1.2.0, but we should try to release v1.2.0 by the end of the year.

[kolyshkin]

Well, we have the 1.2.0 milestone (which I am trying comb through regularly), and everything that's in there is kind of a blocker for 1.2.0.

Feel free to modify milestones for any issues/PRs, but at the very least we have one more blocker:

• regression: can't kill and delete the container with shared(host) pid ns when the init process has dead #4047 (which I'm actively working one, hope to come to a closure this week)
  (and I guess @cyphar can add more)

[AkihiroSuda]

Well, we have the 1.2.0 milestone (which I am trying comb through regularly), and everything that's in there is kind of a blocker for 1.2.0.

Not everything.
At least this one can be postponed to v1.3 or later

• Add mon groups for resctrl. #2523

[kolyshkin]

Not everything.

Well, strictly speaking, everything that is under that milestone needs to either be included into 1.2.0, or be reassigned to a different milestone. Whis is why I said

Feel free to modify milestones for any issues/PRs

[cyphar]

I still think #3985 is needed for 1.2.0. There are several issues with mount propagation (not to mention the serious restriction of idmapped mounts) that are fixed with #3985. We also need #3990 if we want to make sure people scream at us during the -rc1 if it breaks something.

Other than those, I don't have any strong opinions on any remaining PRs.

[h-vetinari]

Blockers for GA: Release runtime-spec v1.1.1

Given that it took well over 3 years to get 1.1.0 finished (incl. half a year from rc1 to GA), is it really necessary (resp. judicious) to block runc 1.2 on a spec-release?

[AkihiroSuda]

Blockers for GA: Release runtime-spec v1.1.1

Given that it took well over 3 years to get 1.1.0 finished (incl. half a year from rc1 to GA), is it really necessary (resp. judicious) to block runc 1.2 on a spec-release?

Necessary, because we have been depending on its main branch:

• features: Expose idmap support #3993

[utam0k]

This task has been done 👍

Release runtime-spec v1.1.1 v1.2.0 opencontainers/runtime-spec#1242

[testinfected]

Hi guys,

Eagerly waiting for that 1.2 release with fixes on mount options. What do you think is a realistic release date?

Thanks

[testinfected]

Ping

[AkihiroSuda]

I think we are ready to release rc.2, and release GA after verifying compatibility with Docker, containerd, BuildKit, etc.

cc @opencontainers/runc-maintainers

[AkihiroSuda]

Turned out that we need to make a decision on this one too

• Revert "Set temporary single CPU affinity before cgroup cpuset transition" #4283 (comment)

[MaxXor]

Any ETA when 1.2.0 will be released?

[AkihiroSuda]

Any ETA when 1.2.0 will be released?

As soon as the compatibility issues get resolved

• Dockerfile: update to runc v1.2.0-rc.2 moby/moby#47666
• update runc (1.2.0) containerd/nerdctl#3153

[h-vetinari]

#3922 is noted as a blocker here, and a recent comment notes

@kolyshkin: This is going to be implemented via opencontainers/runtime-spec#1253

Since opencontainers/runtime-spec@7017384 is not yet in a runtime-spec release, does that mean you'll need another spec release to unblock runc 1.2?

From the peanut gallery: Given that #3922 is a year old already (and the issue apparently exists back until runc 1.0.2 at least), I'm skeptical that this really has to block runc 1.2? After being 28 months behind the originally intended release date, perhaps it would be reasonable to stop the scope-creep and just focus on fixing any remaining hard regressions and then release?

[AkihiroSuda]

Let me move #3922 to the "probably deferrable to v1.2.1 or v1.3.0" list

[rata]

I've tested runc 1.2.0-rc.2 with containerd and opened a PR to fix the issues (some fixes belonged to the cri-tools repo, that are already merged and included in a patch release). All the fixes belong to the containerd/cri-tools repos, nothing to do here in runc: containerd/containerd#10449.

Can someone please take a look at the moby issue? moby/moby#47666

[kolyshkin]

Looking into moby/moby#47666, as well as other dmz issues, I'm thinking maybe we should remove it entirely from runc-1.2?

[rata]

@kolyshkin are you sure that is dmz related? That moby PR is compiling with nodmz, there might be something on the say we disable it, but it felt like another issue given that it fails in the same way with and without the nodmz buildtag

[kolyshkin]

@kolyshkin are you sure that is dmz related? That moby PR is compiling with nodmz, there might be something on the say we disable it, but it felt like another issue given that it fails in the same way with and without the nodmz buildtag

Yeah, probably not. I will take a look.

[rata]

It was indeed something on the way runc_nodmz is disabled, work-around here: #4345 (still need to fix cc_platform.mk).

I didn't manage to debug the failing tests now that docker compiles and need to leave now. Please someone else take a look :)

To use a patched runc, I used this moby PR to just compile my fork: moby/moby#48161. Compilation works fine here, but the test fail. You can use something like that to try fixing the tests on the CI.

[kolyshkin]

I figured out the runc compile issue in moby; see moby/moby#48160

[apostasie]

Hey folks.

Would it be possible to cut an RC3? Ideally that would allow closing containerd/nerdctl#3153

Thanks!

[AkihiroSuda]

If somebody can take a look at incompatibility issues with Moby (moby/moby#48336), that will be helpful for releasing the next RC (and GA)

[rata]

@AkihiroSuda I can't repro the issue locally (even without apparmor=0 in the kernel cmdline), I created a VM in Azure with a stock ubuntu (to see if there was something in that setup), but all test seem to pass just fine. I've tested CI without apparmor, etc, no differences (I'll update the moby PR with more details). @kolyshkin has been testing several things also and he can't repro either.

The issues in CI seem real to me, its' 100% reproducible that the new runc causes failures and the old doesn't. I guess there is something in the moby CI setup that we don't have in the other envs we are testing this.

@AkihiroSuda Can you have a look, please?
@thaJeztah can you have a look too, pretty please?

[AkihiroSuda]

f2f1621 init: close internal fds before execve seems closing stdout too earlier

• Moby CI is failing due to stdio issue introduced in f2f16213 init: close internal fds before execve #4384

[rata]

Okay, I think I understand now the issues with moby: #4384 (comment). They are on moby, though, and not runc. @thaJeztah it would be great if you can take a look though, as it is quite moby related.

IMHO we can tag a new rc now. @AkihiroSuda @kolyshkin @cyphar @lifubang @thaJeztah (or any other maintainer reading this) what do you think?

[lifubang]

IMHO we can tag a new rc now.

I agree, if there is no begin, I'll open the release PR this weekend.

[cyphar]

@lifubang I'll do the release this time.

[AkihiroSuda]

• update runc (1.2.0) containerd/nerdctl#3153

Tried to run nerdctl tests with runc v1.2.0-rc.3, but something seems hanging on Ubuntu 20.04 (cgroup v1):

• [v1.2 regression] [cgroup v1 + rootless] nerdctl run -d --name=bar --pid=container:foo ; nerdctl rm -f bar hangs #4394

[lifubang]

#4408

[rata]

@lifubang thanks! IMHO, we can cut rc4 when @cyphar fixes that issue

[cyphar]

I'll send a patch for it later today. (This will be our first logging output in filepath-securejoin, I guess I'll just use logrus...)

[rata]

IMHO we are ready to make an rc4 (securejoin fixes are in). @opencontainers/runc-maintainers if you agree, can someone do it? I'll be on PTO until next week :D

[cyphar]

I'm flying home today, but I can prepare the release PR and merge it when I get back home in ~30 hours.

[cyphar]

Actually @kolyshkin, do we want #4358 or #4418 for rc4? I'll review them when I get home regardless, but since we have #4367 already I guess we can punt on this until rc5/GA?

[kannon92]

Hello. We are very interested in v1.2 for Kubernetes. Can we get an update on this release? It seems most blockers have been resolved so I'm not entirely sure what is blocking this.

[AkihiroSuda]

Actually @kolyshkin, do we want #4358 or #4418 for rc4? I'll review them when I get home regardless, but since we have #4367 already I guess we can punt on this until rc5/GA?

Can be v1.3?

[h-vetinari]

Actually @kolyshkin, do we want #4358 or #4418 for rc4? I'll review them when I get home regardless, but since we have #4367 already I guess we can punt on this until rc5/GA?

Pleeeeeeeease stop the scope creep. Both of these are still in heavy discussion (aside from changing 100s of lines of code). They should not be included after rc3 anymore - the only things still going into 1.2.0 should be regression fixes and unavoidable infra work (like unbreaking CI).

This release started out with the ambition "we should switch to time-based releases, let's aim for 3 months"¹. How can it be that years later the urge to "but shouldn't we maybe still include that other thing?" is still unchecked? For some more context of why I'm arguing more forcefully than last time -- over a year ago you said

I don't really see any point to doing a 1.2 release and then doing a 1.3 release a few weeks later, when we can instead get a few remaining PRs in and then do a 1.2 release.

The point would have been that ~1.5 years of feature work until mid-2023 would have become available to users over a year earlier. Please have a read of https://wg21.link/p1000 (it may sound unrelated, but I promise it's applicable), in particular the elaboration after

There are two basic release target choices: Pick the features, or pick the release time, and whichever you pick means relinquishing control over determining the other. It is not possible to control both at once.

IMO the project should hold itself accountable for the goals it sets - either time-based releases (then the "should we include this?" discussions need to go away), or abandon any pretense of actually wanting to release anything within a certain timeframe, but then define a list of things you're planning to achieve "when it's done", and similarly stick to it until it's released. FWIW, I think time-based releases would be much more beneficial.

Footnotes

1. the 1.3 milestone still has its target release date set to Jun 13th 2022; 3 months after the original target date for 1.2 ↩

[cyphar]

@AkihiroSuda I was wondering what @kolyshkin's view on it is, given he worked on the rework of capability and knows more than me about the #4426. But yes, I would also be in favour of pushing it to 1.3 (afaics this is a long-standing issue).

[cyphar]

I went through the 1.2.0 issues again, and AFAICS the only thing left we actually need is #4452 (which fixes a regression related to the Go /proc/self/exe cloning we added for 1.2.0, and is a very easy-to-review one-line patch). I can prepare a 1.2.0 release this week once we get that merged.

[rata]

@cyphar SGTM 🚀 !

[kannon92]

Thank you for your followup. Really looking forward to the PSI features in runc so looking forward to 1.2!

[AkihiroSuda]

🎉
https://github.com/opencontainers/runc/releases/tag/v1.2.0

[jorydotcom]

@opencontainers/runc-maintainers Now that this has been released, would you like to do a blog post? Happy to assist with this and/or social posts!