Log cgroup memory.events on exit code 137 as dmesg fallback

[danmoseley]

Note

This PR was authored with Copilot assistance.

When a test process exits with code 137 (SIGKILL/OOM) on Helix Linux containers, dmesg is already called but fails because CAP_SYSLOG is not available in the container. This adds a fallback that reads /sys/fs/cgroup/memory.events (cgroup v2), which requires no special privileges and confirms whether the OOM killer fired (the oom_kill counter).

The script tries two paths (first match wins):

1. /sys/fs/cgroup/<cgroup-path>/memory.events (process-specific cgroup, derived from /proc/self/cgroup)
2. /sys/fs/cgroup/memory.events (root cgroup fallback)

This won't replace dmesg — a separate issue (dotnet/dnceng#6481) tracks enabling CAP_SYSLOG for richer OOM diagnostics — but it gives immediate confirmation of OOM at zero infrastructure cost. Silent no-op on macOS, cgroup v1, or any system without /proc/self/cgroup.

Related issues:

• possible checked CLR inefficiency with very large arrays #98964 (System.Runtime.Tests exit code 137)
• [outerloop] System.Runtime.Numerics.Tests crash with exit code 137 in linux debug coreclr #104588 (System.Runtime.Numerics.Tests exit code 137)

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

[danmoseley]

Tested on Linux in a cgroup forced to go OOM

 === Running inside cgroup: 0::/system.slice/run-r9f8eabb6405f4917a55147e7977f0422.scope ===                   
                                                                                                                 
   === Child (PID 1069) exited with code 137 ===                                                                 
   Exit code is 137 (SIGKILL) — checking cgroup memory.events...                                                 
                                                                                                                 
   cgroup memory.events (/sys/fs/cgroup//system.slice/run-r9f8eabb6405f4917a55147e7977f0422.scope/memory.events):
   low 0                                                                                                         
   high 0                                                                                                        
   max 48                                                                                                        
   oom 1                                                                                                         
   oom_kill 1                                                                                                    
   oom_group_kill 0                                                                                              
                               

this says in this example:

• max 48 — 48 times the cgroup hit its memory limit
• oom 1 — 1 OOM condition triggered
• oom_kill 1 — 1 process was killed by the OOM killer

So it's not much but it's a clue that tells us exit 137 (which could be OOM killer, Helix timeout or something else) is OOM (or almost surely)

[Copilot]

Pull request overview

This PR improves Linux Helix test diagnostics for cases where test processes exit with code 137 (SIGKILL/OOM) by adding a non-privileged fallback to confirm OOM killer activity via cgroup v2 memory.events, since dmesg often fails in containers without CAP_SYSLOG.

Changes:

• On Helix work items with exit code 137, attempt to read and print memory.events from the root cgroup and then from the per-cgroup path derived from /proc/self/cgroup.
• Keep the existing dmesg | tail -50 behavior, adding the cgroup dump as additional context.

[danmoseley]

@copilot address feedback

[danmoseley]

Note

This comment was generated with Copilot.

Added readability comments (example of /proc/self/cgroup format, what the awk expression extracts, note that it's a silent no-op on macOS/cgroup v1).

Also fixed the loop order: the old code checked the root cgroup's memory.events first, but on cgroup v2 that file always exists, so break would prevent the more informative process-specific cgroup path from ever being checked. Reversed to check specific first, falling back to root.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.