[13.4-stable] pillar/containerd: drop bogus scheduler from debug container exec - #6248
Merged
rene merged 1 commit intoAug 3, 2026
Conversation
The process spec used for execs into the debug container sets Scheduler.Deadline, but leaves Scheduler.Policy empty. runc rejects that in ToSchedAttr(), so the container init fails before it can execute the requested program: OCI runtime exec failed: exec failed: unable to start container process: invalid scheduler policy:: unknown Both users of RunInDebugContainer() are affected: the collect-info run triggered through a local operator console (collectinfo agent) and the bpftrace endpoint of pillar's http-debug interface. The spec has looked like this since the bpftrace interface was added, but runc ignored process.scheduler on the exec path until v1.3.0-rc1 (runc#4585), so it stayed unnoticed until the rootfs moved from runc 1.1.12 to 1.3.3 in aa18688 ("bump runc to v3.3.0, containerd to v2.2.0; addresses critical CVEs"). Rather than completing the scheduler spec, drop it. Scheduler.Deadline is a SCHED_DEADLINE bandwidth parameter in nanoseconds, not a limit on how long a process may run, and it was assigned a unix timestamp in seconds, so it never did what it looks like it does - the kernel does not kill tasks for missing a deadline either. The timeout is enforced by RunInDebugContainer() itself, which kills the process once its timer fires. Without process.scheduler runc leaves the exec'ed process with the scheduling attributes it inherits from the debug container, which is what CtrExec() relies on as well. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Christoph Ostarek <christoph@zededa.com> (cherry picked from commit b7d4463) [ 13.4-stable: this branch has neither containerd/run.go nor the collectinfo agent - the same bogus spec sits inline in runInDebugContainer() in agentlog/http-debug.go, so the identical hunk was dropped there instead. Only the bpftrace endpoint of pillar's http-debug interface is affected on this branch. ]
This was referenced Jul 31, 2026
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## 13.4-stable #6248 +/- ##
===============================================
+ Coverage 24.78% 32.66% +7.88%
===============================================
Files 8 9 +1
Lines 1138 1090 -48
===============================================
+ Hits 282 356 +74
+ Misses 788 654 -134
- Partials 68 80 +12 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
christoph-zededa
marked this pull request as ready for review
August 3, 2026 10:45
rene
approved these changes
Aug 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Backport of #6231 to
13.4-stable.The process spec used for execs into the debug container sets
Scheduler.Deadline, but leavesScheduler.Policyempty. runc rejects that inToSchedAttr(), so the container init fails before it can execute the requestedprogram:
runc ignored
process.scheduleron the exec path until v1.3.0-rc1(opencontainers/runc#4585), so the bogus spec stayed unnoticed until the rootfs
moved from runc 1.1.12 to 1.3.3. This branch ships runc 1.3.3, so it is
affected.
Rather than completing the scheduler spec, this drops it.
Scheduler.Deadlineis a
SCHED_DEADLINEbandwidth parameter in nanoseconds, not a limit on howlong a process may run, and it was assigned a unix timestamp in seconds, so it
never did what it looks like it does — the kernel does not kill tasks for
missing a deadline either. The timeout that matters is enforced by the exec
wrapper itself, which kills the process once its timer fires. See #6231 for the
full rationale.
Adapted cherry-pick. Unlike the other three branches of this backport set,
13.4-stablehas neitherpkg/pillar/containerd/run.gonor the collectinfoagent — the same bogus spec sits inline in
runInDebugContainer()inpkg/pillar/agentlog/http-debug.go, so the identical three-line hunk wasdropped there instead. The
-xreference tob7d446377a5a69712d9c1f57427961b9bdc042aa(the squash-merge of #6231 onmaster) is preserved in the commit message, together with a note recording theadaptation. Consequences for this branch:
no local-operator-console collect-info path going through an OCI exec
How to test and validate this PR
Not covered by an automated test — the path needs a real containerd/runc, and
there is currently no unit or Eden coverage for execs into the debug container.
Run a bpftrace script through pillar's http-debug interface:
Without this PR the response is
Error happened: process start failed: … invalid scheduler policy:: unknown;with it the program runs and its JSON output is returned.
Note that
bpftrace-compiler run-via-sshdoes not exercise this path andworks either way: sshd runs inside the debug container itself, so the program
is started as a plain child of the ssh session, without an OCI exec at all.
Only
run-via-httpandrun-via-edgeviewPOST to/debug/bpftraceandtherefore go through
runInDebugContainer().Changelog notes
Fixed pillar's bpftrace debug endpoint, which failed to start with an
invalid scheduler policyerror from the container runtime.PR Backports
All four current LTS branches ship runc 1.3.3, so all of them are affected
(verified in #6231 by extracting
/usr/bin/runcfrom thelinuxkit/runcimageeach branch pins). This PR is one of a set of four:
Checklist
or config surface changes
equivalent change was verified on amd64 in pillar/containerd: drop bogus scheduler from debug container exec #6231. Since the hunk had to be
re-applied to a different file here, this one deserves an actual run of the
bpftrace reproduction above before it leaves draft
For backport PRs (remove it if it's not a backport):
And the last but not least:
check them.
🤖 Generated with Claude Code