Insufficient internal & external storages on ubuntu.1804.amd64.android.open.rt

[greenEkatherine]

About 800 test failed in last 4 days due to installation error. There was no enough space on SD as well as in internal storage

We should find why it happens and set a clean up.

[greenEkatherine]

I checked several machines and this issues is intermittent, so after it happens it is usually followed by passed jobs. However, I tried to check free space and I get:

Console log: 'Using a Payload' from job 827ac7ed-8b90-408a-9616-5d00e1a67b7a workitem a11278f3-b591-43d6-a46d-30989b56075c (ubuntu.1804.amd64.android.open.rt) executed on machine a000ZCV
List of devices attached
emulator-5554          device product:sdk_phone_x86 model:Android_SDK_built_for_x86 device:generic_x86 transport_id:1
emulator-5556          device product:sdk_phone_x86_64 model:Android_SDK_built_for_x86_64 device:generic_x86_64 transport_id:2

Filesystem       Size  Used Avail Use% Mounted on
/dev/block/dm-2  0.9G  956M   48M  96% /
tmpfs            1.4G  596K  1.4G   1% /dev
tmpfs            1.4G     0  1.4G   0% /mnt
tmpfs            1.4G     0  1.4G   0% /apex
/dev/block/dm-1   69M   69M     0 100% /vendor
/dev/block/vdc   775M   15M  728M   2% /data
/dev/block/loop0 232K   36K  192K  16% /apex/com.android.apex.cts.shim@1
/data/media      775M   15M  728M   2% /storage/emulated

My first suspicion was that apps were not properly uninstalled, but I checked previous jobs and didn't see any issues with uninstallation. I'm trying to connect to VM to find more details about what might fill all the space there.

[greenEkatherine]

PR with 2x storages to unblock tests https://dev.azure.com/dnceng/internal/_git/dotnet-helix-machines/pullrequest/16345

[greenEkatherine]

Meanwhile I continue investigation what takes all space (logs? test results?). I'm struggling to connect to VM via jump box, so use jobs with payload and it slows down work

[greenEkatherine]

PR merged and rolled out today, so new machines will have more space

[greenEkatherine]

@steveisok did it help? I see 10x less tests with exit code == 78 for last 24 hours and no one due to space issue. There are intermittent errors that might be solved by switching to another machine and it's out of scope for this issue.

[greenEkatherine]

I don't see this issue since the last update, closing