Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: corellium builders failing with signal: killed #54819

Open
heschi opened this issue Sep 1, 2022 · 6 comments
Open

x/build: corellium builders failing with signal: killed #54819

heschi opened this issue Sep 1, 2022 · 6 comments
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@heschi heschi added Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done. labels Sep 1, 2022
@heschi heschi added this to the Go1.20 milestone Sep 1, 2022
@changkun
Copy link
Member

changkun commented Sep 1, 2022

There was no manual intervention on the builder side to my knowledge. It is either caused by internal service changes on the Corellium side or the changes in the Go code base.

There is not much we can do for the first case, although I've communicated with them many times in the past.

For the second case, if the build process somehow changes to utilize more resources, such as memory, it may hit the VM limit set by Corellium, which could lead to the build failure.

@changkun
Copy link
Member

changkun commented Sep 1, 2022

cc @cuonglm as CL 421880 seems to be the first commit that causes: https://build.golang.org/log/b787fac76217446a7796bfe687e299e2313baf67

@cherrymui
Copy link
Member

cherrymui commented Sep 1, 2022

They look like OOM kill. How much memory does the builder have? Does the build run in parallel with other programs or containers?

@changkun
Copy link
Member

changkun commented Sep 1, 2022

These are the system info collected using adb shell dumpsys meminfo as of Sep 02, 2022.

Android Builder 1:

Total RAM: 1,984,500K (status moderate)
 Free RAM: 1,084,998K (  253,370K cached pss +   728,536K cached kernel +   103,092K free)
 Used RAM:   956,023K (  765,363K used pss +   190,660K kernel)
 Lost RAM:   -56,521K
   Tuning: 48 (large 192), oom   322,560K, restore limit   107,520K (high-end-gfx)

Android Builder 2:

Total RAM: 1,984,500K (status critical)
 Free RAM: 1,239,008K (        0K cached pss +   957,908K cached kernel +   281,100K free)
 Used RAM:   829,439K (  674,879K used pss +   154,560K kernel)
 Lost RAM:   -83,947K
   Tuning: 48 (large 192), oom   322,560K, restore limit   107,520K (high-end-gfx)

Android Builder 3:

Total RAM: 1,984,500K (status low)
 Free RAM: 1,237,810K (   91,994K cached pss +   185,712K cached kernel +   960,104K free)
 Used RAM:   834,669K (  671,777K used pss +   162,892K kernel)
 Lost RAM:   -87,979K
   Tuning: 48 (large 192), oom   322,560K, restore limit   107,520K (high-end-gfx)

@cuonglm
Copy link
Member

cuonglm commented Sep 2, 2022

cc @cuonglm as CL 421880 seems to be the first commit that causes: https://build.golang.org/log/b787fac76217446a7796bfe687e299e2313baf67

This is a real failed and was fixed at tip. The OOM killed seems different problem.

@changkun
Copy link
Member

changkun commented Sep 3, 2022

Based on some recent observations, it seems that the issue is gone. Perhaps now Corellium internal becomes more stable in managing their VMs and not killing OOMed builders anymore. At some point, we may simply close this issue without any further action.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Development

No branches or pull requests

4 participants