Skip to content

pillar/containerd: drop bogus scheduler from debug container exec - #6231

Merged
rene merged 1 commit into
lf-edge:masterfrom
christoph-zededa:collectinfo_edgesync_fix_scheduler_policy
Jul 31, 2026
Merged

pillar/containerd: drop bogus scheduler from debug container exec#6231
rene merged 1 commit into
lf-edge:masterfrom
christoph-zededa:collectinfo_edgesync_fix_scheduler_policy

Conversation

@christoph-zededa

@christoph-zededa christoph-zededa commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Description

The process spec used for execs into the debug container sets
Scheduler.Deadline, but leaves Scheduler.Policy empty. runc rejects that in
ToSchedAttr(), so the container init fails before it can execute the requested
program:

OCI runtime exec failed: exec failed: unable to start container process: invalid scheduler policy:: unknown

Both users of RunInDebugContainer() are affected:

  • the collect-info run triggered through a local operator console (collectinfo
    agent), reported from the field on 17.0.0-rc5
  • the bpftrace endpoint of pillar's http-debug interface

The spec has looked like this since the bpftrace interface was added, but runc
ignored process.scheduler on the exec path until v1.3.0-rc1
(opencontainers/runc#4585), so it stayed unnoticed until the rootfs moved from
runc 1.1.12 to 1.3.3 in aa18688 ("bump runc to v3.3.0, containerd to
v2.2.0; addresses critical CVEs").

Rather than completing the scheduler spec, this drops it. Scheduler.Deadline
is a SCHED_DEADLINE bandwidth parameter in nanoseconds, not a limit on how
long a process may run, and it was assigned a unix timestamp in seconds, so it
never did what it looks like it does — the kernel does not kill tasks for
missing a deadline either. The timeout that matters is enforced by
RunInDebugContainer() itself, which kills the process once its timer fires.
Without process.scheduler runc leaves the exec'ed process with the scheduling
attributes it inherits from the debug container, which is what CtrExec()
relies on as well.

How to test and validate this PR

Not covered by an automated test — the path needs a real containerd/runc, and
there is currently no unit or Eden coverage for execs into the debug container.

Reproduction of the bug (fails without this PR on any build carrying runc 1.3.x,
i.e. 16.3 and later):

  1. Configure a collect-info datastore of type HTTP/HTTPS with auth enabled on
    the local operator console.
  2. Trigger collect info for the device.
  3. Without this PR, pillar logs
    running [/usr/bin/collect-info.sh -u http://…] failed: process start failed: … invalid scheduler policy:: unknown
    and nothing is uploaded.
  4. With this PR, collect-info.sh runs and its tarball appears on the
    datastore.

Please verify the tarball actually lands on the datastore rather than relying on
the pillar log line: RunInDebugContainer() reports failure whenever the script
writes anything to stderr (likely even on a successful run) and reports success
when the 15 min timeout kills the script, and collect-info.sh exits 0 even
when the upload itself fails. Those reporting bugs are pre-existing and out of
scope here.

A second, quicker check that exercises the same code path (no datastore
needed) — run a bpftrace script through pillar's http-debug interface:

# on the device
eve http-debug
# on the dev host - the debug server only listens on localhost, so tunnel it
ssh -p 2222 -L 6543:localhost:6543 root@127.1
bpftrace-compiler run-via-http 127.1:6543 examples/opensnoop.bt

Without this PR the response is
Error happened: process start failed: … invalid scheduler policy:: unknown;
with it the program runs and its JSON output is returned.

Note that bpftrace-compiler run-via-ssh does not exercise this path and
works either way: sshd runs inside the debug container itself, so the program
is started as a plain child of the ssh session, without an OCI exec at all.
Only run-via-http and run-via-edgeview POST to /debug/bpftrace and
therefore go through RunInDebugContainer().

Changelog notes

Fixed collect info triggered from a local operator console, and pillar's
bpftrace debug endpoint, which both failed to start with an
invalid scheduler policy error from the container runtime.

PR Backports

All four current stable branches ship runc 1.3.3, so all of them are affected
(verified by extracting /usr/bin/runc from the linuxkit/runc image each
branch pins):

  • 17.0-stable: To be backported. This is where the bug was reported
    (17.0.0-rc5); the commit cherry-picks cleanly.
  • 16.0-stable: To be backported. Same containerd/run.go hunk, cherry-picks
    cleanly.
  • 14.5-stable: To be backported. Same containerd/run.go hunk, cherry-picks
    cleanly.
  • 13.4-stable: Affected, but this commit does not apply — the branch has no
    containerd/run.go and no collectinfo agent; the same bogus spec sits inline
    in pkg/pillar/agentlog/http-debug.go, affecting the bpftrace endpoint only.
    Needs an equivalent one-hunk patch rather than a cherry-pick.

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR

And the last but not least:

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

Documentation: no user-facing behaviour or config surface changes, nothing to
document. Device testing: still pending, the fix is deliberately kept to
removing a spec field that the kernel ignored anyway — draft until it has been
verified on a device with the reproduction above.

@christoph-zededa christoph-zededa added the stable Should be backported to stable release(s) label Jul 27, 2026
@codecov

codecov Bot commented Jul 27, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 23.56%. Comparing base (8b33286) to head (a708299).
⚠️ Report is 4 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6231      +/-   ##
==========================================
+ Coverage   23.20%   23.56%   +0.36%     
==========================================
  Files         510      520      +10     
  Lines       93473    95186    +1713     
==========================================
+ Hits        21691    22433     +742     
- Misses      70031    70817     +786     
- Partials     1751     1936     +185     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@christoph-zededa christoph-zededa added the bug Something isn't working label Jul 28, 2026
@christoph-zededa
christoph-zededa marked this pull request as ready for review July 28, 2026 10:20
@christoph-zededa

Copy link
Copy Markdown
Contributor Author

/rerun red

The process spec used for execs into the debug container sets
Scheduler.Deadline, but leaves Scheduler.Policy empty. runc rejects
that in ToSchedAttr(), so the container init fails before it can
execute the requested program:

  OCI runtime exec failed: exec failed: unable to start container
  process: invalid scheduler policy:: unknown

Both users of RunInDebugContainer() are affected: the collect-info run
triggered through a local operator console (collectinfo agent) and the
bpftrace endpoint of pillar's http-debug interface.

The spec has looked like this since the bpftrace interface was added,
but runc ignored process.scheduler on the exec path until v1.3.0-rc1
(runc#4585), so it stayed unnoticed until the rootfs moved from runc
1.1.12 to 1.3.3 in aa18688 ("bump runc to v3.3.0, containerd to
v2.2.0; addresses critical CVEs").

Rather than completing the scheduler spec, drop it. Scheduler.Deadline
is a SCHED_DEADLINE bandwidth parameter in nanoseconds, not a limit on
how long a process may run, and it was assigned a unix timestamp in
seconds, so it never did what it looks like it does - the kernel does
not kill tasks for missing a deadline either. The timeout is enforced
by RunInDebugContainer() itself, which kills the process once its timer
fires. Without process.scheduler runc leaves the exec'ed process with
the scheduling attributes it inherits from the debug container, which
is what CtrExec() relies on as well.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Christoph Ostarek <christoph@zededa.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working stable Should be backported to stable release(s)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants