-
Notifications
You must be signed in to change notification settings - Fork 166
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
No space left on macOS host? #2917
Comments
🤷 And
|
hmm. https://ci.nodejs.org/job/node-test-commit-osx/44429/nodes=osx1015/console also ran out of space on the same machine, test-nearform-macos10.15-x64-3.
Again we have around 13G of free space.
The test-commit job does a |
@AshCripps do you know if it's normal for |
im going to say no, |
So far, all three CI runs on nodejs/node#42628 failed with |
FYI the builds really are using about 14GB while in flight:
After build (Workspace cleared up):
Some of the files in
and (probably obviously from the above) So we need to either get these expanded or find something else to delete. The disks are showing as around 51Gb at the moment. |
Note - the
|
Note these machines cant really be any bigger as the host they are on started hitting issues when the VMs were a bigger size. (This was the roughly the largest size that was stable) |
Is that disk size specifically? Any idea what the issues were? I could understand if RAM/CPU allocation was overdone it would lead to issues but disk seems a little odd. |
I think it was due to apples autoupdate feature meaning the host needed enough headroom other it crashes - im sure there was an issue on this in the build repo. |
Jenkins has automatically taken test-nearform-macos10.15-x64-3 offline due to low disk space. We're now also seeing disk space issues on test-orka-macos11-x64-2,
|
For test-orka-macos11-x64-2 I've removed |
test-orka-macos11-x64-2:
21G in
but that leaves ~50G unaccounted for 😞. |
you are probably best off not looking in |
🤷 |
Can you use the GUI somehow? Maybe macOS tools (for example the "Storage" tab, in "About This Mac") will be able to find what's taking the remaining space. |
I have no idea. The internal Build docs say the Orka machines have VNC disabled but then give a username/password for screen sharing. I do not have a mac device and only have access to Windows and Linux locally. |
If we are unable to find what's occupying the disk space, is it possible to just erase the disk and set up the machines from scratch again? |
Yeah the mac VNC client is a bit differnet from the others. I've been able to use Having said that I cannot establish a connection to the VNC server on those machines even when on the private network. |
By the time I logged into to have a look there seemed to be 16Gb of space free so it's probably ok now again. |
I think that's because #2917 (comment) freed up enough space for |
osx1015 is failing tests with |
So ive had a little poke around on The big problem with the nearform x64 machines is they are 2 vms hosted on a physical mac mini which means we need space for 3 os's all our tools and builds on only 200GB of disk space - there is around 40GB of head room on the host itself so we can give some leeway but if I wil try to find a solution for this file bloat in the meantime but if this keeps happening I will increase the size of the VMs as a stop gap. |
I don't know if it's only because of this issue, but at the moment, macOS CI is almost permanently red. |
Ive just logged into |
Maybe someone with a mac can verify? The test CI runs
and the release job doesn't run tests either so that doesn't account for extra space required by the tests such as |
My |
ah ok well that is a problem then. Also once again the private folder on the machine I rebooted a couple weeks back is over 7.6GB again so this needs some consideration. It might be worth putting in a weekly reboot into these machines (for reference my mac private folder is ~60% of that size and its turned off every day) but thats also only a stop gap. I think we need to have a think about what to do with these machines and what solutions we can put in place. |
|
I got the same error on today's build: #3071 |
I was doing some research about this topic. Here are some useful logs 😄 Note: I could't test all the osx machines (see: #3084, #3083 and #3034) Machine Logs test-nearform-macos10.15-x64-1Relevant logs (some clean up made)
test-nearform-macos11.0-arm64-1Relevant logs (some clean up made)
test-orka-macos10.14-x64-1Relevant logs (some clean up made)
test-orka-macos10.14-x64-2Relevant logs (some clean up made)
test-orka-macos10.14-x64-3Relevant logs (some clean up made)
test-orka-macos11-x64-1Relevant logs (some clean up made)
test-orka-macos11-x64-2Relevant logs (some clean up made)
rebooting Experiment Logs By just rebooting a machine the cache folder does not purge itself properlyRelevant logs (some clean up made)
Personal conclusions Just some examples:
Additionally We can also retrieve some marginal gigabytes from macos updates, downloads, etc... I see several options to overcome this challenge:
Potential roadmap
|
This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made. |
See https://ci.nodejs.org/job/node-test-commit-osx/44401/nodes=osx1015/#showFailuresLink
The text was updated successfully, but these errors were encountered: