x/build: corellium builders failing with signal: killed

[heschi]

#!watchflakes
post <- builder ~ `.*-corellium` && log ~ `signal: killed`

2022-09-01T15:27:07-b5b2cf5/android-arm-corellium
2022-09-01T02:24:14-91ef076/android-arm-corellium
2022-09-01T00:42:27-64b260d/android-arm-corellium
2022-09-01T00:42:27-64b260d/android-arm64-corellium
2022-08-31T23:52:00-ca634fa/android-arm-corellium
2022-08-31T23:52:00-ca634fa/android-arm64-corellium
2022-08-31T22:22:43-e4b624e/android-arm-corellium
2022-08-31T22:09:44-5f5c018/android-arm-corellium
2022-08-31T22:04:55-af7f067/android-arm-corellium
2022-08-31T21:10:41-52d9e7f/android-arm64-corellium
2022-08-31T20:05:46-239115c/android-arm-corellium
2022-08-31T20:05:06-9085ff5/android-arm64-corellium
2022-08-31T16:37:20-ebc75ac/android-arm64-corellium
2022-08-31T15:33:00-fa0e3bf/android-arm-corellium
2022-08-31T15:30:57-d01200e/android-arm-corellium
2022-08-31T15:30:26-a25a34a/android-arm64-corellium
2022-08-31T15:12:21-d3b35a4/android-arm64-corellium
2022-08-31T15:12:18-28e3885/android-arm64-corellium
2022-08-31T09:35:10-d2e0587/android-arm-corellium
2022-08-31T01:16:54-ee0e40a/android-arm-corellium

cc @golang/android

[changkun]

There was no manual intervention on the builder side to my knowledge. It is either caused by internal service changes on the Corellium side or the changes in the Go code base.

There is not much we can do for the first case, although I've communicated with them many times in the past.

For the second case, if the build process somehow changes to utilize more resources, such as memory, it may hit the VM limit set by Corellium, which could lead to the build failure.

[changkun]

cc @cuonglm as CL 421880 seems to be the first commit that causes: https://build.golang.org/log/b787fac76217446a7796bfe687e299e2313baf67

[cherrymui]

They look like OOM kill. How much memory does the builder have? Does the build run in parallel with other programs or containers?

[changkun]

These are the system info collected using adb shell dumpsys meminfo as of Sep 02, 2022.

Android Builder 1:

Total RAM: 1,984,500K (status moderate)
 Free RAM: 1,084,998K (  253,370K cached pss +   728,536K cached kernel +   103,092K free)
 Used RAM:   956,023K (  765,363K used pss +   190,660K kernel)
 Lost RAM:   -56,521K
   Tuning: 48 (large 192), oom   322,560K, restore limit   107,520K (high-end-gfx)

Android Builder 2:

Total RAM: 1,984,500K (status critical)
 Free RAM: 1,239,008K (        0K cached pss +   957,908K cached kernel +   281,100K free)
 Used RAM:   829,439K (  674,879K used pss +   154,560K kernel)
 Lost RAM:   -83,947K
   Tuning: 48 (large 192), oom   322,560K, restore limit   107,520K (high-end-gfx)

Android Builder 3:

Total RAM: 1,984,500K (status low)
 Free RAM: 1,237,810K (   91,994K cached pss +   185,712K cached kernel +   960,104K free)
 Used RAM:   834,669K (  671,777K used pss +   162,892K kernel)
 Lost RAM:   -87,979K
   Tuning: 48 (large 192), oom   322,560K, restore limit   107,520K (high-end-gfx)

[cuonglm]

cc @cuonglm as CL 421880 seems to be the first commit that causes: https://build.golang.org/log/b787fac76217446a7796bfe687e299e2313baf67

This is a real failed and was fixed at tip. The OOM killed seems different problem.

[changkun]

Based on some recent observations, it seems that the issue is gone. Perhaps now Corellium internal becomes more stable in managing their VMs and not killing OOMed builders anymore. At some point, we may simply close this issue without any further action.