Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

System.Runtime.Tests Outerloop tests failing with SIGKILL #66396

Closed
adamsitnik opened this issue Mar 9, 2022 · 2 comments
Closed

System.Runtime.Tests Outerloop tests failing with SIGKILL #66396

adamsitnik opened this issue Mar 9, 2022 · 2 comments
Labels
area-System.Runtime

Comments

@adamsitnik
Copy link
Member

Hit in #66387.

Sample log file: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-66387-merge-8f05b743b17c405b9f/System.Runtime.Tests/1/console.d0128aa7.log?sv=2019-07-07&se=2022-03-29T13%3A33%3A14Z&sr=c&sp=rl&sig=mDrKRAWhoS3nfIdvqjfHj9WPL5uoPC5Rw6Kk5r5E0Hg%3D

/root/helix/work/correlation/dotnet exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=IgnoreForCI -notrait category=failing 
popd
===========================================================================================================
/root/helix/work/workitem/e /root/helix/work/workitem/e
  Discovering: System.Runtime.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Runtime.Tests (found 28 of 6309 test cases)
  Starting:    System.Runtime.Tests (parallel test collections = on, max threads = 2)
./RunTests.sh: line 168:    25 Killed                  "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=IgnoreForCI -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Wed 09 Mar 2022 01:53:24 PM UTC ----- exit code 137 ----------------------------------------------------------
exit code 137 means SIGKILL Killed eg by kill
ulimit -c value: unlimited
[  317.229311] 0 pages hwpoisoned
[  317.229311] Tasks state (memory values in pages):
[  317.229312] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[  317.229316] [    451]     0   451    23710      198   184320        0             0 systemd-journal
[  317.229318] [    465]     0   465    24428       44    94208        0             0 lvmetad
[  317.229320] [    474]     0   474    11006      348   118784        0         -1000 systemd-udevd
[  317.229321] [    540]     0   540     3067      267    65536        0             0 hv_kvp_daemon
[  317.229323] [    854] 62583   854    35446      108   180224        0             0 systemd-timesyn
[  317.229325] [    945]   100   945    20009      163   167936        0             0 systemd-network
[  317.229326] [    969]   101   969    17654      150   184320        0             0 systemd-resolve
[  317.229328] [   1173]     0  1173    42708     1957   233472        0             0 networkd-dispat
[  317.229329] [   1176]     0  1176    15502      143   167936        0             0 systemd-logind
[  317.229331] [   1177]     0  1177    20059     3266   196608        0             0 python3
[  317.229333] [   1178]     0  1178    40270       33    81920        0             0 lxcfs
[  317.229334] [   1183]     0  1183     7964       78    98304        0             0 cron
[  317.229336] [   1186]     0  1186     7084       52    98304        0             0 atd
[  317.229337] [   1187]     0  1187    72000      215   200704        0             0 accounts-daemon
[  317.229339] [   1188]   103  1188    12539      179   147456        0          -900 dbus-daemon
[  317.229340] [   1200]     0  1200    27605       55   122880        0             0 irqbalance
[  317.229342] [   1202]     0  1202   337818     5223   294912        0          -999 containerd
[  317.229343] [   1221]     0  1221     4104       37    73728        0             0 agetty
[  317.229345] [   1230]     0  1230     3723       34    69632        0             0 agetty
[  317.229346] [   1232]     0  1232    72221      197   204800        0             0 polkitd
[  317.229348] [   1370]     0  1370     1128       17    53248        0             0 none
[  317.229349] [   1443]     0  1443   115283     4961   270336        0             0 python3
[  317.229351] [   1732]     0  1732    18076      183   180224        0         -1000 sshd
[  317.229353] [   1899]     0  1899   410936    21841   589824        0          -500 dockerd
[  317.229354] [   2242]  1000  2242     2899       64    61440        0             0 helix.sh
[  317.229356] [   2794]     0  2794    37139     2964   192512        0             0 python3
[  317.229357] [   2799]     0  2799   900949     7022   303104        0             0 amacoreagent
[  317.229358] [   2886]     0  2886   229350      299   151552        0             0 agentlauncher
[  317.229360] [   2932]     0  2932    79371       69   118784        0             0 mdsdmgr
[  317.229362] [   3296]  1000  3296    44344     6857   253952        0             0 python3
[  317.229363] [   3364]   106  3364     7150       45   102400        0             0 uuidd
[  317.229365] [   3398]  1000  3398    63562     7108   270336        0             0 python3
[  317.229366] [   3399]  1000  3399   105350    10700   323584        0             0 python3
[  317.229368] [   3473]   102  3473    67333      139   180224        0             0 rsyslogd
[  317.229369] [   3605]   102  3605   150857     2983   446464        0             0 mdsd
[  317.229371] [   3893]     0  3893   297958     3060   249856        0             0 auoms
[  317.229372] [   3921]     0  3921   215737     1698   188416        0             0 auomscollect
[  317.229374] [   4237]     0  4237   201897     4319   241664        0             0 azsecd
[  317.229375] [   4248]     0  4248   177083     3317   188416        0             0 azsecmond
[  317.229377] [   5040]     0  5040   178212     1696   114688        0          -998 containerd-shim
[  317.229378] [   5062]  1000  5062      596       17    40960        0             0 helix_docker_wo
[  317.229380] [   5133]  1000  5133      596       17    40960        0             0 execute.sh
[  317.229381] [   5135]  1000  5135     1714       86    49152        0             0 bash
[  317.229383] [   5145]  1000  5145  2904006  1869990 15400960        0             0 dotnet
[  317.229384] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=d6d83bd2640930892e9431ca04fa850dffadde62971abe155f3701da4bb9a1cc,mems_allowed=0,global_oom,task_memcg=/docker/d6d83bd2640930892e9431ca04fa850dffadde62971abe155f3701da4bb9a1cc,task=dotnet,pid=5145,uid=1000
[  317.229424] Out of memory: Killed process 5145 (dotnet) total-vm:11616024kB, anon-rss:7479960kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:15040kB oom_score_adj:0
[  317.359200] oom_reaper: reaped process 5145 (dotnet), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Waiting a few seconds for any dump to be written..
cat /proc/sys/kernel/core_pattern: /home/helixbot/dotnetbuild/dumps/core.%u.%p
cat /proc/sys/kernel/core_uses_pid: 0
cat: /proc/sys/kernel/coredump_filter: No such file or directory
cat /proc/sys/kernel/coredump_filter:
Looking around for any Linux dump..
... found no dump in /root/helix/work/workitem/e
@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Mar 9, 2022
@ghost
Copy link

ghost commented Mar 9, 2022

Tagging subscribers to this area: @dotnet/area-system-runtime
See info in area-owners.md if you want to be subscribed.

Issue Details

Hit in #66387.

Sample log file: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-66387-merge-8f05b743b17c405b9f/System.Runtime.Tests/1/console.d0128aa7.log?sv=2019-07-07&se=2022-03-29T13%3A33%3A14Z&sr=c&sp=rl&sig=mDrKRAWhoS3nfIdvqjfHj9WPL5uoPC5Rw6Kk5r5E0Hg%3D

/root/helix/work/correlation/dotnet exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=IgnoreForCI -notrait category=failing 
popd
===========================================================================================================
/root/helix/work/workitem/e /root/helix/work/workitem/e
  Discovering: System.Runtime.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Runtime.Tests (found 28 of 6309 test cases)
  Starting:    System.Runtime.Tests (parallel test collections = on, max threads = 2)
./RunTests.sh: line 168:    25 Killed                  "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=IgnoreForCI -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Wed 09 Mar 2022 01:53:24 PM UTC ----- exit code 137 ----------------------------------------------------------
exit code 137 means SIGKILL Killed eg by kill
ulimit -c value: unlimited
[  317.229311] 0 pages hwpoisoned
[  317.229311] Tasks state (memory values in pages):
[  317.229312] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[  317.229316] [    451]     0   451    23710      198   184320        0             0 systemd-journal
[  317.229318] [    465]     0   465    24428       44    94208        0             0 lvmetad
[  317.229320] [    474]     0   474    11006      348   118784        0         -1000 systemd-udevd
[  317.229321] [    540]     0   540     3067      267    65536        0             0 hv_kvp_daemon
[  317.229323] [    854] 62583   854    35446      108   180224        0             0 systemd-timesyn
[  317.229325] [    945]   100   945    20009      163   167936        0             0 systemd-network
[  317.229326] [    969]   101   969    17654      150   184320        0             0 systemd-resolve
[  317.229328] [   1173]     0  1173    42708     1957   233472        0             0 networkd-dispat
[  317.229329] [   1176]     0  1176    15502      143   167936        0             0 systemd-logind
[  317.229331] [   1177]     0  1177    20059     3266   196608        0             0 python3
[  317.229333] [   1178]     0  1178    40270       33    81920        0             0 lxcfs
[  317.229334] [   1183]     0  1183     7964       78    98304        0             0 cron
[  317.229336] [   1186]     0  1186     7084       52    98304        0             0 atd
[  317.229337] [   1187]     0  1187    72000      215   200704        0             0 accounts-daemon
[  317.229339] [   1188]   103  1188    12539      179   147456        0          -900 dbus-daemon
[  317.229340] [   1200]     0  1200    27605       55   122880        0             0 irqbalance
[  317.229342] [   1202]     0  1202   337818     5223   294912        0          -999 containerd
[  317.229343] [   1221]     0  1221     4104       37    73728        0             0 agetty
[  317.229345] [   1230]     0  1230     3723       34    69632        0             0 agetty
[  317.229346] [   1232]     0  1232    72221      197   204800        0             0 polkitd
[  317.229348] [   1370]     0  1370     1128       17    53248        0             0 none
[  317.229349] [   1443]     0  1443   115283     4961   270336        0             0 python3
[  317.229351] [   1732]     0  1732    18076      183   180224        0         -1000 sshd
[  317.229353] [   1899]     0  1899   410936    21841   589824        0          -500 dockerd
[  317.229354] [   2242]  1000  2242     2899       64    61440        0             0 helix.sh
[  317.229356] [   2794]     0  2794    37139     2964   192512        0             0 python3
[  317.229357] [   2799]     0  2799   900949     7022   303104        0             0 amacoreagent
[  317.229358] [   2886]     0  2886   229350      299   151552        0             0 agentlauncher
[  317.229360] [   2932]     0  2932    79371       69   118784        0             0 mdsdmgr
[  317.229362] [   3296]  1000  3296    44344     6857   253952        0             0 python3
[  317.229363] [   3364]   106  3364     7150       45   102400        0             0 uuidd
[  317.229365] [   3398]  1000  3398    63562     7108   270336        0             0 python3
[  317.229366] [   3399]  1000  3399   105350    10700   323584        0             0 python3
[  317.229368] [   3473]   102  3473    67333      139   180224        0             0 rsyslogd
[  317.229369] [   3605]   102  3605   150857     2983   446464        0             0 mdsd
[  317.229371] [   3893]     0  3893   297958     3060   249856        0             0 auoms
[  317.229372] [   3921]     0  3921   215737     1698   188416        0             0 auomscollect
[  317.229374] [   4237]     0  4237   201897     4319   241664        0             0 azsecd
[  317.229375] [   4248]     0  4248   177083     3317   188416        0             0 azsecmond
[  317.229377] [   5040]     0  5040   178212     1696   114688        0          -998 containerd-shim
[  317.229378] [   5062]  1000  5062      596       17    40960        0             0 helix_docker_wo
[  317.229380] [   5133]  1000  5133      596       17    40960        0             0 execute.sh
[  317.229381] [   5135]  1000  5135     1714       86    49152        0             0 bash
[  317.229383] [   5145]  1000  5145  2904006  1869990 15400960        0             0 dotnet
[  317.229384] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=d6d83bd2640930892e9431ca04fa850dffadde62971abe155f3701da4bb9a1cc,mems_allowed=0,global_oom,task_memcg=/docker/d6d83bd2640930892e9431ca04fa850dffadde62971abe155f3701da4bb9a1cc,task=dotnet,pid=5145,uid=1000
[  317.229424] Out of memory: Killed process 5145 (dotnet) total-vm:11616024kB, anon-rss:7479960kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:15040kB oom_score_adj:0
[  317.359200] oom_reaper: reaped process 5145 (dotnet), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Waiting a few seconds for any dump to be written..
cat /proc/sys/kernel/core_pattern: /home/helixbot/dotnetbuild/dumps/core.%u.%p
cat /proc/sys/kernel/core_uses_pid: 0
cat: /proc/sys/kernel/coredump_filter: No such file or directory
cat /proc/sys/kernel/coredump_filter:
Looking around for any Linux dump..
... found no dump in /root/helix/work/workitem/e
Author: adamsitnik
Assignees: -
Labels:

area-System.Runtime
Milestone: -

@adamsitnik adamsitnik mentioned this issue Mar 9, 2022
Merged
@adamsitnik
Copy link
Member Author

It's a duplicate of #58616

@ghost ghost locked as resolved and limited conversation to collaborators Apr 8, 2022
@jeffhandley jeffhandley removed the untriaged New issue has not been triaged by the area owner label May 6, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-System.Runtime
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jeffhandley @adamsitnik