No space left on macOS host?

[targos]

See https://ci.nodejs.org/job/node-test-commit-osx/44401/nodes=osx1015/#showFailuresLink

[richardlau]

🤷
There have been three node-test-commit-osx builds on the same machine, https://ci.nodejs.org/computer/test-nearform-macos10.15-x64-3/builds, since then which all passed:

And

test-nearform-macos10:~ iojs$ df -h /Users/iojs/build/workspace/
Filesystem     Size   Used  Avail Capacity iused     ifree %iused  Mounted on
/dev/disk1s1   51Gi   25Gi   15Gi    63%  309782 536799578    0%   /System/Volumes/Data
test-nearform-macos10:~ iojs$

[richardlau]

hmm. https://ci.nodejs.org/job/node-test-commit-osx/44429/nodes=osx1015/console also ran out of space on the same machine, test-nearform-macos10.15-x64-3.

13:54:44 fatal error: /Library/Developer/CommandLineTools/usr/bin/libtool: can't write to output file (No space left on device)
13:55:20 make[2]: *** [/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1015/out/Release/libv8_initializers.a] Error 1

Again we have around 13G of free space.

test-nearform-macos10:~ iojs$ df -h
Filesystem      Size   Used  Avail Capacity iused     ifree %iused  Mounted on
/dev/disk1s5    51Gi   10Gi   13Gi    44%  487478 536621882    0%   /
devfs          186Ki  186Ki    0Bi   100%     642         0  100%   /dev
/dev/disk1s1    51Gi   26Gi   13Gi    67%  357543 536751817    0%   /System/Volumes/Data
/dev/disk1s4    51Gi  1.0Gi   13Gi     8%       2 537109358    0%   /private/var/vm
map auto_home    0Bi    0Bi    0Bi   100%       0         0  100%   /System/Volumes/Data/home
test-nearform-macos10:~ iojs$ df -h /Users/iojs/
Filesystem     Size   Used  Avail Capacity iused     ifree %iused  Mounted on
/dev/disk1s1   51Gi   26Gi   13Gi    67%  357543 536751817    0%   /System/Volumes/Data
test-nearform-macos10:~ iojs$ df -h /Users/iojs/build/workspace/
Filesystem     Size   Used  Avail Capacity iused     ifree %iused  Mounted on
/dev/disk1s1   51Gi   26Gi   13Gi    67%  357543 536751817    0%   /System/Volumes/Data
test-nearform-macos10:~ iojs$ df -h /tmp
Filesystem     Size   Used  Avail Capacity iused     ifree %iused  Mounted on
/dev/disk1s1   51Gi   26Gi   13Gi    67%  357543 536751817    0%   /System/Volumes/Data
test-nearform-macos10:~ iojs$

The test-commit job does a git clean at the end. So somehow the job is chewing through 13G while running? (!)

[richardlau]

@AshCripps do you know if it's normal for /dev and /System/Volumes/Data/home to be 100% used on macs?

[AshCripps]

im going to say no, /System/Volumes/Data is the map of the user directory IIRC so its likely is a full disk. Catalina changed this reporting into a whole weird mess.

[tniessen]

So far, all three CI runs on nodejs/node#42628 failed with ENOSPC on osx.

[richardlau]

I'm just more and more confused:

[sxa]

FYI the builds really are using about 14GB while in flight:
During build (while running tests):

/dev/disk1s1    53710936 39596968   1474908    97%  284892 536824468    0%   /System/Volumes/Data

After build (Workspace cleared up):

Filesystem    1024-blocks     Used Available Capacity iused     ifree %iused  Mounted on
/dev/disk1s5     53710936 10873180  14855648    43%  487478 536621882    0%   /

Some of the files in out/Release are pretty big - these two account for 5GB alone;

3004068	libv8_base_without_compiler.a
2001752	libv8_initializers.a

and (probably obviously from the above) obj.target is quite large at over 6Gb

So we need to either get these expanded or find something else to delete. The disks are showing as around 51Gb at the moment. /var/db is quite large at about 8.7Gb but I don't know whether anything can be done about that. ccache for the iojs user is using 4.5Gb

[sxa]

Note - the -1 machine has a slightly larger disk than -2 and -3:

[sxa@sainz .ssh]$ for A in 1 2 3; do ssh test-nearform-macos10.15-x64-$A df -k .; done
Filesystem   1024-blocks     Used Available Capacity iused     ifree %iused  Mounted on
/dev/disk1s1    68359372 26111720  29608124    47%  293179 683300541    0%   /System/Volumes/Data
Filesystem   1024-blocks     Used Available Capacity iused     ifree %iused  Mounted on
/dev/disk1s1    53710936 24257204  16802816    60%  296611 536812749    0%   /System/Volumes/Data
Filesystem   1024-blocks     Used Available Capacity iused     ifree %iused  Mounted on
/dev/disk1s1    53710936 39309828   1762048    96%  284926 536824434    0%   /System/Volumes/Data
[sxa@sainz .ssh]$ 

[AshCripps]

Note these machines cant really be any bigger as the host they are on started hitting issues when the VMs were a bigger size. (This was the roughly the largest size that was stable)

[sxa]

issues when the VMs were a bigger size.

Is that disk size specifically? Any idea what the issues were? I could understand if RAM/CPU allocation was overdone it would lead to issues but disk seems a little odd.

[AshCripps]

I think it was due to apples autoupdate feature meaning the host needed enough headroom other it crashes - im sure there was an issue on this in the build repo.

[richardlau]

Jenkins has automatically taken test-nearform-macos10.15-x64-3 offline due to low disk space.

We're now also seeing disk space issues on test-orka-macos11-x64-2,
e.g. https://ci.nodejs.org/job/node-test-commit-osx/44542/nodes=osx11-x64/console

10:55:10 not ok 3236 pummel/test-fs-readfile-tostring-fail
10:55:10   ---
10:55:10   duration_ms: 0.203
10:55:10   severity: fail
10:55:10   exitcode: 7
10:55:10   stack: |-
10:55:10     /Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64/test/pummel/test-fs-readfile-tostring-fail.js:69
10:55:10       throw err;
10:55:10       ^
10:55:10     
10:55:10     [Error: ENOSPC: no space left on device, write] {
10:55:10       errno: -28,
10:55:10       code: 'ENOSPC',
10:55:10       syscall: 'write'
10:55:10     }
10:55:10     
10:55:10     Node.js v18.0.0-pre
10:55:10   ...

[richardlau]

For test-orka-macos11-x64-2 I've removed /Users/iojs/build/workspace/node-test-commit-osx-as-shared-mdawson which was 1.5GB.

[richardlau]

test-orka-macos11-x64-2:

administrator@test-orka-macos11-x64-2 ~ % df -h
Filesystem       Size   Used  Avail Capacity iused     ifree %iused  Mounted on
/dev/disk2s5s1   90Gi   14Gi  1.8Gi    89%  553788 941116212    0%   /
devfs           189Ki  189Ki    0Bi   100%     653         0  100%   /dev
/dev/disk2s4     90Gi  1.0Gi  1.8Gi    36%       2 941669998    0%   /System/Volumes/VM
/dev/disk2s2     90Gi  305Mi  1.8Gi    15%    1038 941668962    0%   /System/Volumes/Preboot
/dev/disk2s6     90Gi  440Ki  1.8Gi     1%      18 941669982    0%   /System/Volumes/Update
/dev/disk2s1     90Gi   72Gi  1.8Gi    98%  455861 941214139    0%   /System/Volumes/Data
map auto_home     0Bi    0Bi    0Bi   100%       0         0  100%   /System/Volumes/Data/home
administrator@test-orka-macos11-x64-2 ~ %

administrator@test-orka-macos11-x64-2 ~ % sudo du -hs /System/Volumes/Data/*
 18M    /System/Volumes/Data/Applications
du: /System/Volumes/Data/Library/Caches/com.apple.aned: Operation not permitted
2.5G    /System/Volumes/Data/Library
271M    /System/Volumes/Data/Previous Content
3.2G    /System/Volumes/Data/System
 21G    /System/Volumes/Data/Users
  0B    /System/Volumes/Data/Volumes
266M    /System/Volumes/Data/boot
  0B    /System/Volumes/Data/cores
1.0K    /System/Volumes/Data/home
  0B    /System/Volumes/Data/mnt
  0B    /System/Volumes/Data/opt
du: /System/Volumes/Data/private/var/networkd/db: Operation not permitted
du: /System/Volumes/Data/private/var/db/appinstalld: Operation not permitted
du: /System/Volumes/Data/private/var/db/fpsd/dvp: Operation not permitted
du: /System/Volumes/Data/private/var/db/installcoordinationd: Operation not permitted
du: /System/Volumes/Data/private/var/db/oah: Operation not permitted
du: /System/Volumes/Data/private/var/db/lockdown: Operation not permitted
du: /System/Volumes/Data/private/var/db/KernelExtensionManagement/Staging: Operation not permitted
du: /System/Volumes/Data/private/var/db/DifferentialPrivacy: Operation not permitted
du: /System/Volumes/Data/private/var/db/MobileIdentityService: Operation not permitted
du: /System/Volumes/Data/private/var/db/searchparty: Operation not permitted
du: /System/Volumes/Data/private/var/db/CoreDuet/Knowledge: Operation not permitted
du: /System/Volumes/Data/private/var/db/ConfigurationProfiles/Store: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.ScreenTimeAgent/Store: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.lockoutagent: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.progressd/ClassKit: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.LaunchServices.dv: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/dmd: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.nsurlsessiond: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.SharedWebCredentials: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.routined/dv: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.ScreenTimeAgent/Store: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.lockoutagent: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.progressd/ClassKit: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.exchangesync: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.LaunchServices.dv: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/dmd: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.nsurlsessiond: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.SharedWebCredentials: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.routined/dv: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/C/com.apple.quicklook.ThumbnailsAgent/com.apple.QuickLook.thumbnailcache: Operation not permitted
du: /System/Volumes/Data/private/var/folders/7l/fb92_ds12k19tryhky7zbfnw0000gp/0/com.apple.LaunchServices.dv: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000sm00006d/0: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000sm00006d/C: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000y800007k/0/com.apple.nsurlsessiond: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.ScreenTimeAgent/Store: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.lockoutagent: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/dmd: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.nsurlsessiond: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.routined/dv: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/C/com.apple.quicklook.ThumbnailsAgent/com.apple.QuickLook.thumbnailcache: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/0/com.apple.LaunchServices.dv: Operation not permitted
du: /System/Volumes/Data/private/var/protected/trustd/private: Operation not permitted
3.8G    /System/Volumes/Data/private
  0B    /System/Volumes/Data/sw
943M    /System/Volumes/Data/usr
administrator@test-orka-macos11-x64-2 ~ %

21G in /System/Volumes/Data/Users is mostly the builds (workspace and ccache):

administrator@test-orka-macos11-x64-2 ~ % sudo du -hs /Users/iojs/build/workspace/node-test-commit-osx
 16G    /Users/iojs/build/workspace/node-test-commit-osx
administrator@test-orka-macos11-x64-2 ~ % sudo du -hs /Users/iojs/Library/Caches/ccache
4.1G    /Users/iojs/Library/Caches/ccache

but that leaves ~50G unaccounted for 😞.

[AshCripps]

you are probably best off not looking in /System/Volumes/Data as it should be just a mirror that you cant access of filesystem, just regular checks under / and /Users should help to find the data. Besides that I wonder if its started downloading an os update as they can be fairly hefty.

[richardlau]

administrator@test-orka-macos11-x64-2 ~ % sudo du -hs /*
 18M    /Applications
du: /Library/Caches/com.apple.aned: Operation not permitted
2.5G    /Library
du: /System/Library/Templates/Data/private/var/db/oah: Operation not permitted
du: /System/Volumes/Data/Library/Caches/com.apple.aned: Operation not permitted
du: /System/Volumes/Data/private/var/networkd/db: Operation not permitted
du: /System/Volumes/Data/private/var/db/appinstalld: Operation not permitted
du: /System/Volumes/Data/private/var/db/fpsd/dvp: Operation not permitted
du: /System/Volumes/Data/private/var/db/installcoordinationd: Operation not permitted
du: /System/Volumes/Data/private/var/db/oah: Operation not permitted
du: /System/Volumes/Data/private/var/db/lockdown: Operation not permitted
du: /System/Volumes/Data/private/var/db/KernelExtensionManagement/Staging: Operation not permitted
du: /System/Volumes/Data/private/var/db/DifferentialPrivacy: Operation not permitted
du: /System/Volumes/Data/private/var/db/MobileIdentityService: Operation not permitted
du: /System/Volumes/Data/private/var/db/searchparty: Operation not permitted
du: /System/Volumes/Data/private/var/db/CoreDuet/Knowledge: Operation not permitted
du: /System/Volumes/Data/private/var/db/ConfigurationProfiles/Store: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.ScreenTimeAgent/Store: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.lockoutagent: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.progressd/ClassKit: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.LaunchServices.dv: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/dmd: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.nsurlsessiond: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.SharedWebCredentials: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.routined/dv: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.ScreenTimeAgent/Store: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.lockoutagent: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.progressd/ClassKit: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.exchangesync: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.LaunchServices.dv: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/dmd: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.nsurlsessiond: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.SharedWebCredentials: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.routined/dv: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/C/com.apple.quicklook.ThumbnailsAgent/com.apple.QuickLook.thumbnailcache: Operation not permitted
du: /System/Volumes/Data/private/var/folders/7l/fb92_ds12k19tryhky7zbfnw0000gp/0/com.apple.LaunchServices.dv: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000sm00006d/0: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000sm00006d/C: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000y800007k/0/com.apple.nsurlsessiond: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.ScreenTimeAgent/Store: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.lockoutagent: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/dmd: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.nsurlsessiond: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.routined/dv: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/C/com.apple.quicklook.ThumbnailsAgent/com.apple.QuickLook.thumbnailcache: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/0/com.apple.LaunchServices.dv: Operation not permitted
du: /System/Volumes/Data/private/var/protected/trustd/private: Operation not permitted
 38G    /System
7.0G    /Users
  0B    /Volumes
4.6M    /bin
  0B    /cores
4.5K    /dev
  0B    /etc
  0B    /home
  0B    /opt
du: /private/var/networkd/db: Operation not permitted
du: /private/var/db/appinstalld: Operation not permitted
du: /private/var/db/fpsd/dvp: Operation not permitted
du: /private/var/db/installcoordinationd: Operation not permitted
du: /private/var/db/oah: Operation not permitted
du: /private/var/db/lockdown: Operation not permitted
du: /private/var/db/KernelExtensionManagement/Staging: Operation not permitted
du: /private/var/db/DifferentialPrivacy: Operation not permitted
du: /private/var/db/MobileIdentityService: Operation not permitted
du: /private/var/db/searchparty: Operation not permitted
du: /private/var/db/CoreDuet/Knowledge: Operation not permitted
du: /private/var/db/ConfigurationProfiles/Store: Operation not permitted
du: /private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.ScreenTimeAgent/Store: Operation not permitted
du: /private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.lockoutagent: Operation not permitted
du: /private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.progressd/ClassKit: Operation not permitted
du: /private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.LaunchServices.dv: Operation not permitted
du: /private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/dmd: Operation not permitted
du: /private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.nsurlsessiond: Operation not permitted
du: /private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.SharedWebCredentials: Operation not permitted
du: /private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.routined/dv: Operation not permitted
du: /private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.ScreenTimeAgent/Store: Operation not permitted
du: /private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.lockoutagent: Operation not permitted
du: /private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.progressd/ClassKit: Operation not permitted
du: /private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.exchangesync: Operation not permitted
du: /private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.LaunchServices.dv: Operation not permitted
du: /private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/dmd: Operation not permitted
du: /private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.nsurlsessiond: Operation not permitted
du: /private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.SharedWebCredentials: Operation not permitted
du: /private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.routined/dv: Operation not permitted
du: /private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/C/com.apple.quicklook.ThumbnailsAgent/com.apple.QuickLook.thumbnailcache: Operation not permitted
du: /private/var/folders/7l/fb92_ds12k19tryhky7zbfnw0000gp/0/com.apple.LaunchServices.dv: Operation not permitted
du: /private/var/folders/zz/zyxvpxvq6csfxvn_n00000sm00006d/0: Operation not permitted
du: /private/var/folders/zz/zyxvpxvq6csfxvn_n00000sm00006d/C: Operation not permitted
du: /private/var/folders/zz/zyxvpxvq6csfxvn_n00000y800007k/0/com.apple.nsurlsessiond: Operation not permitted
du: /private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.ScreenTimeAgent/Store: Operation not permitted
du: /private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.lockoutagent: Operation not permitted
du: /private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/dmd: Operation not permitted
du: /private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.nsurlsessiond: Operation not permitted
du: /private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.routined/dv: Operation not permitted
du: /private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/C/com.apple.quicklook.ThumbnailsAgent/com.apple.QuickLook.thumbnailcache: Operation not permitted
du: /private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/0/com.apple.LaunchServices.dv: Operation not permitted
du: /private/var/protected/trustd/private: Operation not permitted
3.8G    /private
2.0M    /sbin
  0B    /tmp
1.5G    /usr
  0B    /var
administrator@test-orka-macos11-x64-2 ~ %

administrator@test-orka-macos11-x64-2 ~ % sudo du -hs /Users/*
1.5M    /Users/Shared
568M    /Users/administrator
6.4G    /Users/iojs
administrator@test-orka-macos11-x64-2 ~ %

🤷

[targos]

Can you use the GUI somehow? Maybe macOS tools (for example the "Storage" tab, in "About This Mac") will be able to find what's taking the remaining space.

[richardlau]

I have no idea. The internal Build docs say the Orka machines have VNC disabled but then give a username/password for screen sharing. I do not have a mac device and only have access to Windows and Linux locally.

[joyeecheung]

If we are unable to find what's occupying the disk space, is it possible to just erase the disk and set up the machines from scratch again?

[sxa]

Yeah the mac VNC client is a bit differnet from the others. I've been able to use realvnc to connect to a mac from another platform but in general it's a bit easier from a mac client :-)

Having said that I cannot establish a connection to the VNC server on those machines even when on the private network.

[sxa]

By the time I logged into to have a look there seemed to be 16Gb of space free so it's probably ok now again.

[richardlau]

By the time I logged into to have a look there seemed to be 16Gb of space free so it's probably ok now again.

I think that's because #2917 (comment) freed up enough space for node-test-commit-osx to run successfully (seems to need around the 16G mark to run). node-test-commit-osx runs make distclean at the end to reclaim disk space used for the compiled artifacts.

[tniessen]

osx1015 is failing tests with ENOSPC again: https://ci.nodejs.org/job/node-test-commit-osx/44792/nodes=osx1015/

[tniessen]

Again: https://ci.nodejs.org/job/node-test-commit-osx/45454/nodes=osx11-x64/

[targos]

Again: https://ci.nodejs.org/job/node-test-commit-osx/47635/nodes=osx1015/

[AshCripps]

So ive had a little poke around on nearform-2 (the problem machine targos linked above) and have managed to free up some space with a simple reboot. I believe the root of our issue is the bloat of /System/Volumes/Data/private which is a temporary cache of sorts mac uses (hence why a reboot free a GB or so) but its still at 6GB. This needs some investigation into why its getting so bloated - it seems you can just blow away the folder but that runs the risk of bricking the machine depending on whats using it so would need to be done when there is time to rebuild it with no issue.

The big problem with the nearform x64 machines is they are 2 vms hosted on a physical mac mini which means we need space for 3 os's all our tools and builds on only 200GB of disk space - there is around 40GB of head room on the host itself so we can give some leeway but if /private is just going to keep bloating thats not much help.

I wil try to find a solution for this file bloat in the meantime but if this keeps happening I will increase the size of the VMs as a stop gap.

[targos]

Again:

• https://ci.nodejs.org/job/node-test-commit-osx/47902/nodes=osx11-x64/testReport/junit/(root)/test/pummel_test_fs_readfile_tostring_fail/
• https://ci.nodejs.org/job/node-test-commit-osx/47899/nodes=osx11-x64/testReport/junit/(root)/test/pummel_test_fs_readfile_tostring_fail/

[targos]

https://ci.nodejs.org/job/node-test-commit-osx/47935/nodes=osx11-x64/testReport/junit/(root)/test/pummel_test_fs_readfile_tostring_fail/

[aduh95]

https://ci.nodejs.org/job/node-test-commit-osx/48075/nodes=osx11-x64/testReport/junit/(root)/test/pummel_test_fs_readfile_tostring_fail/

[targos]

https://ci.nodejs.org/job/node-test-commit-osx/48110/nodes=osx1015/testReport/junit/(root)/test/pummel_test_fs_largefile/

[targos]

I don't know if it's only because of this issue, but at the moment, macOS CI is almost permanently red.

[AshCripps]

Ive just logged into orka-macos11-x64-2 which is the latest failure and there is 15gb of free space available - surely thats enough to build node is it not? or has the binary really ballooned this past year?

[richardlau]

Ive just logged into orka-macos11-x64-2 which is the latest failure and there is 15gb of free space available - surely thats enough to build node is it not? or has the binary really ballooned this past year?

Maybe someone with a mac can verify?

The test CI runs make clean at the end of the job so it's hard to catch peak disk usage, but the release jobs do not and the workspaces for the release machines are in that ballpark, with the arm64 almost double presumably because it builds both x64 and arm64:

$ ssh release-orka-macos10.15-x64-1 du -hs /Users/iojs/build/ws/
 17G    /Users/iojs/build/ws/
$ ssh release-nearform-macos10.15-x64-1 du -hs /Users/iojs/build/ws/
 17G    /Users/iojs/build/ws/
$ ssh release-nearform-macos11.0-arm64-1 du -hs /Users/iojs/build/ws/
 31G    /Users/iojs/build/ws/

and the release job doesn't run tests either so that doesn't account for extra space required by the tests such as test-fs-readfile-tostring-fail.

[targos]

My node folder takes 16G (arm64 mac)

[AshCripps]

ah ok well that is a problem then. Also once again the private folder on the machine I rebooted a couple weeks back is over 7.6GB again so this needs some consideration. It might be worth putting in a weekly reboot into these machines (for reference my mac private folder is ~60% of that size and its turned off every day) but thats also only a stop gap.

I think we need to have a think about what to do with these machines and what solutions we can put in place.

[AshCripps]

test-orka-macos11-x64-2 had a file /System/Volumes/Data/System/Library/Caches/com.apple.coresymbolicationd/data which was 39GB!! have removed, rebooted and brought back on line.

[anonrig]

I got the same error on today's build: #3071

[UlisesGascon]

I was doing some research about this topic. Here are some useful logs 😄

Note: I could't test all the osx machines (see: #3084, #3083 and #3034)

Machine Logs

test-nearform-macos10.15-x64-1

Relevant logs (some clean up made)

  administrator@test-nearform-macos10 ~ % uptime
  8:50  up 52 days,  5:48, 1 user, load averages: 1,51 1,44 1,50
  administrator@test-nearform-macos10 ~ % df -h
  Filesystem      Size   Used  Avail Capacity iused     ifree %iused  Mounted on
  /dev/disk1s5    65Gi   10Gi   21Gi    33%  487478 683106242    0%   /
  devfs          186Ki  186Ki    0Bi   100%     642         0  100%   /dev
  /dev/disk1s1    65Gi   32Gi   21Gi    61%  399854 683193866    0%   /System/Volumes/Data
  /dev/disk1s4    65Gi  1.0Gi   21Gi     5%       2 683593718    0%   /private/var/vm
  map auto_home    0Bi    0Bi    0Bi   100%       0         0  100%   /System/Volumes/Data/home
  administrator@test-nearform-macos10 ~ % sudo find / -size +200M -exec du -hs {} \;
  288M	/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/.git/objects/pack/pack-6769fbbe19b6f8e0b4f6d56b36f4cf84862d1e09.pack
  344M	/Library/Updates/001-89745/CLTools_Executables.pkg
  482M	/Library/Updates/001-51037/macOSUpdCombo10.15.7Auto.RecoveryHDUpdate.pkg
  3,7G	/Library/Updates/001-51037/macOSUpdCombo10.15.7Auto.pkg
  282M	/System/Library/Desktop Pictures/Mojave Night.heic
  282M	/System/Library/Desktop Pictures/Mojave Day.heic
  282M	/System/Library/Desktop Pictures/Mojave.heic
  306M	/System/Library/Speech/Voices/AlexCompact.SpeechVoice/Contents/Resources/PCMWave
  320M	/System/Library/Templates/Data/System/Library/Speech/Voices/AlexCompact.SpeechVoice/Contents/Resources/PCMWave
  128M	/System/Library/Caches/com.apple.coresymbolicationd/data
  288M	/System/Volumes/Data/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/.git/objects/pack/pack-6769fbbe19b6f8e0b4f6d56b36f4cf84862d1e09.pack
  344M	/System/Volumes/Data/Library/Updates/001-89745/CLTools_Executables.pkg
  482M	/System/Volumes/Data/Library/Updates/001-51037/macOSUpdCombo10.15.7Auto.RecoveryHDUpdate.pkg
  3,7G	/System/Volumes/Data/Library/Updates/001-51037/macOSUpdCombo10.15.7Auto.pkg
  306M	/System/Volumes/Data/System/Library/Speech/Voices/AlexCompact.SpeechVoice/Contents/Resources/PCMWave
  128M	/System/Volumes/Data/System/Library/Caches/com.apple.coresymbolicationd/data
  1,0G	/System/Volumes/Data/private/var/vm/swapfile0
  2,0G	/System/Volumes/Data/private/var/db/dyld/dyld_shared_cache_x86_64h
  945M	/System/Volumes/Data/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1015/.git/objects/pack/pack-8aa2f69375349f716c18139246a3042878ca2e8b.pack
  782M	/System/Volumes/Data/Users/iojs/build/workspace/node-test-commit-osx-as-shared-mdawson/nodes/osx1015/.git/objects/pack/pack-2570bfa359596f92a681c6a7e7e76d00141d925d.pack
  1,0G	/private/var/vm/swapfile0
  2,0G	/private/var/db/dyld/dyld_shared_cache_x86_64h
  945M	/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1015/.git/objects/pack/pack-8aa2f69375349f716c18139246a3042878ca2e8b.pack
  782M	/Users/iojs/build/workspace/node-test-commit-osx-as-shared-mdawson/nodes/osx1015/.git/objects/pack/pack-2570bfa359596f92a681c6a7e7e76d00141d925d.pack

test-nearform-macos11.0-arm64-1

Relevant logs (some clean up made)

  admin@test-nearform-macos11 ~ % uptime
  16:55  up 15 days,  7:31, 3 users, load averages: 1,12 1,08 1,16
  admin@test-nearform-macos11 ~ % df -h
  Filesystem       Size   Used  Avail Capacity iused      ifree %iused  Mounted on
  /dev/disk3s1s1  926Gi   21Gi  649Gi     4%  553788 9712948012    0%   /
  devfs           202Ki  202Ki    0Bi   100%     700          0  100%   /dev
  /dev/disk3s6    926Gi  1.0Gi  649Gi     1%       1 9713501799    0%   /System/Volumes/VM
  /dev/disk3s2    926Gi  361Mi  649Gi     1%    1128 9713500672    0%   /System/Volumes/Preboot
  /dev/disk3s4    926Gi  1.6Gi  649Gi     1%     298 9713501502    0%   /System/Volumes/Update
  /dev/disk1s2    500Mi  6.0Mi  482Mi     2%       3    5119997    0%   /System/Volumes/xarts
  /dev/disk1s1    500Mi  7.3Mi  482Mi     2%      39    5119961    0%   /System/Volumes/iSCPreboot
  /dev/disk1s3    500Mi  228Ki  482Mi     1%      29    5119971    0%   /System/Volumes/Hardware
  /dev/disk3s5    926Gi  252Gi  649Gi    28% 1531395 9711970405    0%   /System/Volumes/Data
  /dev/disk3s1    926Gi   21Gi  649Gi     4%  553681 9712948119    0%   /System/Volumes/Update/mnt1
  map auto_home     0Bi    0Bi    0Bi   100%       0          0  100%   /System/Volumes/Data/home
  admin@test-nearform-macos11 ~ % sudo find / -size +200M -exec du -hs {} \;
  84M	/Library/Developer/CommandLineTools/usr/bin/swift-frontend
  85M	/Library/Developer/CommandLineTools/Library/PrivateFrameworks/LLDB.framework/Versions/A/LLDB
  464M	/Library/Updates/002-41708/CLTools_Executables.pkg
  306M	/System/Library/SpeechBase/Voices/AlexCompact.SpeechVoice/Contents/Resources/PCMWave
  897M	/System/Library/dyld/dyld_shared_cache_arm64e
  1,0G	/System/Library/dyld/aot_shared_cache
  959M	/System/Library/dyld/dyld_shared_cache_x86_64h
  960M	/System/Library/dyld/dyld_shared_cache_x86_64
  167M	/System/Library/KernelCollections/SystemKernelExtensions.kc
  480M	/System/Library/AssetsV2/com_apple_MobileAsset_SFRSoftwareUpdate/c8b50620ac9b503c17c9a122f33bff80fe4c30c9.asset/AssetData/payloadv2/basesystem_patches/arm64eBaseSystem.dmg
  593M	/System/Library/AssetsV2/com_apple_MobileAsset_MacSoftwareUpdate/a292a0d3e2365aafb4f60ef55dc9eb6c2474f984.asset/AssetData/Restore/BaseSystem.dmg
  941M	/System/Library/AssetsV2/com_apple_MobileAsset_MacSoftwareUpdate/a292a0d3e2365aafb4f60ef55dc9eb6c2474f984.asset/AssetData/Restore/022-13976-024.dmg
  186M	/System/Library/Fonts/Apple Color Emoji.ttc
  182G	/System/Library/Caches/com.apple.coresymbolicationd/data
  1,1G	/System/Volumes/Update/restore.log
  306M	/System/Volumes/Update/mnt1/System/Library/SpeechBase/Voices/AlexCompact.SpeechVoice/Contents/Resources/PCMWave
  896M	/System/Volumes/Update/mnt1/System/Library/dyld/dyld_shared_cache_arm64e
  1,0G	/System/Volumes/Update/mnt1/System/Library/dyld/aot_shared_cache
  958M	/System/Volumes/Update/mnt1/System/Library/dyld/dyld_shared_cache_x86_64h
  960M	/System/Volumes/Update/mnt1/System/Library/dyld/dyld_shared_cache_x86_64
  167M	/System/Volumes/Update/mnt1/System/Library/KernelCollections/SystemKernelExtensions.kc
  186M	/System/Volumes/Update/mnt1/System/Library/Fonts/Apple Color Emoji.ttc
  1,0G	/System/Volumes/VM/swapfile0
  84M	/System/Volumes/Data/Library/Developer/CommandLineTools/usr/bin/swift-frontend
  85M	/System/Volumes/Data/Library/Developer/CommandLineTools/Library/PrivateFrameworks/LLDB.framework/Versions/A/LLDB
  464M	/System/Volumes/Data/Library/Updates/002-41708/CLTools_Executables.pkg
  474M	/System/Volumes/Data/System/Library/AssetsV2/com_apple_MobileAsset_SFRSoftwareUpdate/c8b50620ac9b503c17c9a122f33bff80fe4c30c9.asset/AssetData/payloadv2/basesystem_patches/arm64eBaseSystem.dmg
  591M	/System/Volumes/Data/System/Library/AssetsV2/com_apple_MobileAsset_MacSoftwareUpdate/a292a0d3e2365aafb4f60ef55dc9eb6c2474f984.asset/AssetData/Restore/BaseSystem.dmg
  941M	/System/Volumes/Data/System/Library/AssetsV2/com_apple_MobileAsset_MacSoftwareUpdate/a292a0d3e2365aafb4f60ef55dc9eb6c2474f984.asset/AssetData/Restore/022-13976-024.dmg
  182G	/System/Volumes/Data/System/Library/Caches/com.apple.coresymbolicationd/data
  11G	/System/Volumes/Data/Users/admin/Downloads/Xcode_12.4.xip
  945M	/System/Volumes/Data/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/.git/objects/pack/pack-80dc9e8413257c5e1d54466e102b820796389382.pack
  12G	/System/Volumes/Data/Applications/Install macOS Monterey.app/Contents/SharedSupport/SharedSupport.dmg
  376M	/System/Volumes/Data/opt/homebrew/Library/Taps/homebrew/homebrew-core/.git/objects/pack/pack-17e32ab6c7738e60d567f53596d7ad6a8ff08dd9.pack
  11G	/Users/admin/Downloads/Xcode_12.4.xip
  945M	/Users/iojs/build/workspace/node-test-commit-osx-arm/nodes/osx11/.git/objects/pack/pack-80dc9e8413257c5e1d54466e102b820796389382.pack
  12G	/Applications/Install macOS Monterey.app/Contents/SharedSupport/SharedSupport.dmg
  376M	/opt/homebrew/Library/Taps/homebrew/homebrew-core/.git/objects/pack/pack-17e32ab6c7738e60d567f53596d7ad6a8ff08dd9.pack

test-orka-macos10.14-x64-1

Relevant logs (some clean up made)

  test-orka-macos10:~ administrator$ uptime
  9:09  up 159 days,  3:20, 1 user, load averages: 2,30 2,59 2,40
  test-orka-macos10:~ administrator$ df -h
  Filesystem      Size   Used  Avail Capacity iused               ifree %iused  Mounted on
  /dev/disk2s1    90Gi   24Gi   64Gi    28%  664125 9223372036854111682    0%   /
  devfs          185Ki  185Ki    0Bi   100%     641                   0  100%   /dev
  /dev/disk2s4    90Gi  1.0Gi   64Gi     2%       1 9223372036854775806    0%   /private/var/vm
  map -hosts       0Bi    0Bi    0Bi   100%       0                   0  100%   /net
  map auto_home    0Bi    0Bi    0Bi   100%       0                   0  100%   /home
  test-orka-macos10:~ administrator$ sudo find / -size +200M -exec du -hs {} \;
  241M	/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/.git/objects/pack/pack-112706dcd8ed84be706b0ba26f0b556e3caf7bcc.pack
  289M	/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/.git/objects/pack/pack-fb2958b11abd3b35cd17a5709f42a6c55bd55ed9.pack
  464M	/Library/Updates/061-41419/macOSUpdCombo10.14.6Auto.RecoveryHDUpdate.pkg
  1,7G	/Library/Updates/061-41419/macOSUpdCombo10.14.6Auto.pkg
  320M	/System/Library/Speech/Voices/AlexCompact.SpeechVoice/Contents/Resources/PCMWave
  1,0G	/private/var/vm/swapfile0
  510M	/private/var/db/dyld/dyld_shared_cache_i386
  1,3G	/private/var/db/dyld/dyld_shared_cache_x86_64
  560M	/Users/iojs/build/workspace/node-test-commit-osx-AshCripps/nodes/test-orka-macos10.14-x64-1/.git/objects/pack/pack-3b65c08ccc95f45a89e97f46c16fa606ae9fa98a.pack
  896M	/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1014/.git/objects/pack/pack-a53c6f2e355156c51b56609d50d66817dd880d94.pack

test-orka-macos10.14-x64-2

Relevant logs (some clean up made)

  test-orka-macos10:~ administrator$ uptime
  9:13  up 159 days,  3:24, 1 user, load averages: 1,58 1,88 1,88
  test-orka-macos10:~ administrator$ df -h
  Filesystem      Size   Used  Avail Capacity iused               ifree %iused  Mounted on
  /dev/disk2s1    90Gi   25Gi   63Gi    29%  689330 9223372036854086477    0%   /
  devfs          185Ki  185Ki    0Bi   100%     641                   0  100%   /dev
  /dev/disk2s4    90Gi  1.0Gi   63Gi     2%       1 9223372036854775806    0%   /private/var/vm
  map -hosts       0Bi    0Bi    0Bi   100%       0                   0  100%   /net
  map auto_home    0Bi    0Bi    0Bi   100%       0                   0  100%   /home
  test-orka-macos10:~ administrator$ sudo find / -size +200M -exec du -hs {} \;
  240M	/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/.git/objects/pack/pack-112706dcd8ed84be706b0ba26f0b556e3caf7bcc.pack
  289M	/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/.git/objects/pack/pack-fb2958b11abd3b35cd17a5709f42a6c55bd55ed9.pack
  464M	/Library/Updates/061-41419/macOSUpdCombo10.14.6Auto.RecoveryHDUpdate.pkg
  1,7G	/Library/Updates/061-41419/macOSUpdCombo10.14.6Auto.pkg
  320M	/System/Library/Speech/Voices/AlexCompact.SpeechVoice/Contents/Resources/PCMWave
  1,0G	/private/var/vm/swapfile0
  510M	/private/var/db/dyld/dyld_shared_cache_i386
  1,3G	/private/var/db/dyld/dyld_shared_cache_x86_64
  560M	/Users/iojs/build/workspace/node-test-commit-osx-AshCripps/nodes/test-orka-macos10.14-x64-1/.git/objects/pack/pack-3b65c08ccc95f45a89e97f46c16fa606ae9fa98a.pack
  881M	/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1014/.git/objects/pack/pack-b8dfa42f6fa17066c126ad052b67cbc5c1cc2778.pack

test-orka-macos10.14-x64-3

Relevant logs (some clean up made)

  test-orka-macos10:~ administrator$ uptime
  9:17  up 159 days,  1:47, 1 user, load averages: 1,56 2,03 2,06
  test-orka-macos10:~ administrator$ df -h
  Filesystem      Size   Used  Avail Capacity iused               ifree %iused  Mounted on
  /dev/disk2s1    90Gi   25Gi   63Gi    29%  715424 9223372036854060383    0%   /
  devfs          184Ki  184Ki    0Bi   100%     638                   0  100%   /dev
  /dev/disk2s4    90Gi  1.0Gi   63Gi     2%       1 9223372036854775806    0%   /private/var/vm
  map -hosts       0Bi    0Bi    0Bi   100%       0                   0  100%   /net
  map auto_home    0Bi    0Bi    0Bi   100%       0                   0  100%   /home
  test-orka-macos10:~ administrator$ sudo find / -size +200M -exec du -hs {} \;
  241M	/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/.git/objects/pack/pack-2e4125cdea08492bec9a12d31e0e44cf14ea61e1.pack
  289M	/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/.git/objects/pack/pack-fb2958b11abd3b35cd17a5709f42a6c55bd55ed9.pack
  464M	/Library/Updates/061-41419/macOSUpdCombo10.14.6Auto.RecoveryHDUpdate.pkg
  1,7G	/Library/Updates/061-41419/macOSUpdCombo10.14.6Auto.pkg
  320M	/System/Library/Speech/Voices/AlexCompact.SpeechVoice/Contents/Resources/PCMWave
  1,0G	/private/var/vm/swapfile0
  510M	/private/var/db/dyld/dyld_shared_cache_i386
  1,3G	/private/var/db/dyld/dyld_shared_cache_x86_64
  560M	/Users/iojs/build/workspace/node-test-commit-osx-AshCripps/nodes/test-orka-macos10.14-x64-1/.git/objects/pack/pack-3b65c08ccc95f45a89e97f46c16fa606ae9fa98a.pack
  907M	/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1014/.git/objects/pack/pack-83edf1b65d83a18497e8b2d9aa7aa10d061b938b.pack

test-orka-macos11-x64-1

Relevant logs (some clean up made)

  administrator@test-orka-macos11-x64-1 ~ % uptime
  9:10  up 18:43, 1 user, load averages: 2,57 2,34 2,07
  administrator@test-orka-macos11-x64-1 ~ % df -h

  Filesystem       Size   Used  Avail Capacity iused     ifree %iused  Mounted on
  /dev/disk2s5s1   90Gi   14Gi   19Gi    44%  553788 941116212    0%   /
  devfs           189Ki  189Ki    0Bi   100%     654         0  100%   /dev
  /dev/disk2s4     90Gi  515Mi   19Gi     3%       2 941669998    0%   /System/Volumes/VM
  /dev/disk2s2     90Gi  305Mi   19Gi     2%    1038 941668962    0%   /System/Volumes/Preboot
  /dev/disk2s6     90Gi  504Ki   19Gi     1%      17 941669983    0%   /System/Volumes/Update
  /dev/disk2s1     90Gi   55Gi   19Gi    75%  381076 941288924    0%   /System/Volumes/Data
  map auto_home     0Bi    0Bi    0Bi   100%       0         0  100%   /System/Volumes/Data/home
  administrator@test-orka-macos11-x64-1 ~ % sudo find / -size +200M -exec du -hs {} \;
  395M	/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/.git/objects/pack/pack-3a0743b6a6abca2de79dc7381cc965175cab5bb2.pack
  86M	/Library/Developer/CommandLineTools/usr/bin/swift-frontend
  82M	/Library/Developer/CommandLineTools/usr/lib/swift/macosx/lib_InternalSwiftScan.dylib
  88M	/Library/Developer/CommandLineTools/Library/PrivateFrameworks/LLDB.framework/Versions/A/LLDB
  306M	/System/Library/SpeechBase/Voices/AlexCompact.SpeechVoice/Contents/Resources/PCMWave
  897M	/System/Library/dyld/dyld_shared_cache_arm64e
  1,0G	/System/Library/dyld/aot_shared_cache
  959M	/System/Library/dyld/dyld_shared_cache_x86_64h
  960M	/System/Library/dyld/dyld_shared_cache_x86_64
  167M	/System/Library/KernelCollections/SystemKernelExtensions.kc
  608M	/System/Library/AssetsV2/com_apple_MobileAsset_MacSoftwareUpdate/c7a6cff95465966476cbebf21c6580581582eabc.asset/AssetData/Restore/BaseSystem.dmg
  912M	/System/Library/AssetsV2/com_apple_MobileAsset_MacSoftwareUpdate/c7a6cff95465966476cbebf21c6580581582eabc.asset/AssetData/Restore/022-12376-025.dmg
  186M	/System/Library/Fonts/Apple Color Emoji.ttc
  36G	/System/Library/Caches/com.apple.coresymbolicationd/data
  512M	/System/Volumes/VM/swapfile0
  395M	/System/Volumes/Data/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/.git/objects/pack/pack-3a0743b6a6abca2de79dc7381cc965175cab5bb2.pack
  86M	/System/Volumes/Data/Library/Developer/CommandLineTools/usr/bin/swift-frontend
  82M	/System/Volumes/Data/Library/Developer/CommandLineTools/usr/lib/swift/macosx/lib_InternalSwiftScan.dylib
  88M	/System/Volumes/Data/Library/Developer/CommandLineTools/Library/PrivateFrameworks/LLDB.framework/Versions/A/LLDB
  594M	/System/Volumes/Data/System/Library/AssetsV2/com_apple_MobileAsset_MacSoftwareUpdate/c7a6cff95465966476cbebf21c6580581582eabc.asset/AssetData/Restore/BaseSystem.dmg
  911M	/System/Volumes/Data/System/Library/AssetsV2/com_apple_MobileAsset_MacSoftwareUpdate/c7a6cff95465966476cbebf21c6580581582eabc.asset/AssetData/Restore/022-12376-025.dmg
  36G	/System/Volumes/Data/System/Library/Caches/com.apple.coresymbolicationd/data
  911M	/System/Volumes/Data/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64/.git/objects/pack/pack-131ed30e160c47cd7a9e02d1fb24e38c3d071ee4.pack
  911M	/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64/.git/objects/pack/pack-131ed30e160c47cd7a9e02d1fb24e38c3d071ee4.pack

test-orka-macos11-x64-2

Relevant logs (some clean up made)

  administrator@test-orka-macos11-x64-2 ~ % uptime
  9:17  up 25 days,  4:13, 1 user, load averages: 1,80 1,87 1,91
  administrator@test-orka-macos11-x64-2 ~ % df -h
  Filesystem       Size   Used  Avail Capacity iused     ifree %iused  Mounted on
  /dev/disk2s5s1   90Gi   14Gi   29Gi    34%  553788 941116212    0%   /
  devfs           189Ki  189Ki    0Bi   100%     654         0  100%   /dev
  /dev/disk2s4     90Gi  1.0Gi   29Gi     4%       2 941669998    0%   /System/Volumes/VM
  /dev/disk2s2     90Gi  305Mi   29Gi     2%    1038 941668962    0%   /System/Volumes/Preboot
  /dev/disk2s6     90Gi  516Ki   29Gi     1%      17 941669983    0%   /System/Volumes/Update
  /dev/disk2s1     90Gi   45Gi   29Gi    61%  407971 941262029    0%   /System/Volumes/Data
  map auto_home     0Bi    0Bi    0Bi   100%       0         0  100%   /System/Volumes/Data/home
  administrator@test-orka-macos11-x64-2 ~ % sudo find / -size +200M -exec du -hs {} \;
  395M	/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/.git/objects/pack/pack-3a0743b6a6abca2de79dc7381cc965175cab5bb2.pack
  86M	/Library/Developer/CommandLineTools/usr/bin/swift-frontend
  82M	/Library/Developer/CommandLineTools/usr/lib/swift/macosx/lib_InternalSwiftScan.dylib
  88M	/Library/Developer/CommandLineTools/Library/PrivateFrameworks/LLDB.framework/Versions/A/LLDB
  464M	/Library/Updates/002-41708/CLTools_Executables.pkg
  306M	/System/Library/SpeechBase/Voices/AlexCompact.SpeechVoice/Contents/Resources/PCMWave
  897M	/System/Library/dyld/dyld_shared_cache_arm64e
  1,0G	/System/Library/dyld/aot_shared_cache
  959M	/System/Library/dyld/dyld_shared_cache_x86_64h
  960M	/System/Library/dyld/dyld_shared_cache_x86_64
  167M	/System/Library/KernelCollections/SystemKernelExtensions.kc
  186M	/System/Library/Fonts/Apple Color Emoji.ttc
  24G	/System/Library/Caches/com.apple.coresymbolicationd/data
  1,0G	/System/Volumes/VM/swapfile0
  395M	/System/Volumes/Data/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/.git/objects/pack/pack-3a0743b6a6abca2de79dc7381cc965175cab5bb2.pack
  86M	/System/Volumes/Data/Library/Developer/CommandLineTools/usr/bin/swift-frontend
  82M	/System/Volumes/Data/Library/Developer/CommandLineTools/usr/lib/swift/macosx/lib_InternalSwiftScan.dylib
  88M	/System/Volumes/Data/Library/Developer/CommandLineTools/Library/PrivateFrameworks/LLDB.framework/Versions/A/LLDB
  464M	/System/Volumes/Data/Library/Updates/002-41708/CLTools_Executables.pkg
  24G	/System/Volumes/Data/System/Library/Caches/com.apple.coresymbolicationd/data
  908M	/System/Volumes/Data/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64/.git/objects/pack/pack-bca316b5e68c8ca94543de28f4bfeae397d92458.pack
  908M	/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64/.git/objects/pack/pack-bca316b5e68c8ca94543de28f4bfeae397d92458.pack

rebooting Experiment Logs

By just rebooting a machine the cache folder does not purge itself properly

Relevant logs (some clean up made)

  ➜  ~ ssh administrator@test-orka-macos11-x64-1
  administrator@test-orka-macos11-x64-1 ~ % uptime
  14:33  up 158 days,  7:04, 1 user, load averages: 1,89 2,03 1,93
  administrator@test-orka-macos11-x64-1 ~ % sudo du -sh /System/Library/Caches/* | sort -h
  0B	/System/Library/Caches/OnDemandResources
  0B	/System/Library/Caches/com.apple.CVMS
  0B	/System/Library/Caches/com.apple.factorydata
  0B	/System/Library/Caches/com.apple.kernelcaches
  0B	/System/Library/Caches/com.apple.kext.caches
  4,0K	/System/Library/Caches/MobileActivation
  4,0K	/System/Library/Caches/com.apple.SystemMigrationUserSizeCache
  4,0K	/System/Library/Caches/com.apple.apfsd.stats.plist
  8,0K	/System/Library/Caches/com.apple.Components2.SystemCache.Components
  12K	/System/Library/Caches/com.apple.IntlDataCache.le
  124K	/System/Library/Caches/com.apple.IntlDataCache.le.kbdx
  9,6M	/System/Library/Caches/com.apple.preferencepanes.systemcache
  36G	/System/Library/Caches/com.apple.coresymbolicationd
  administrator@test-orka-macos11-x64-1 ~ % df -h
  Filesystem       Size   Used  Avail Capacity iused     ifree %iused  Mounted on
  /dev/disk2s5s1   90Gi   14Gi   18Gi    45%  553788 941116212    0%   /
  devfs           189Ki  189Ki    0Bi   100%     653         0  100%   /dev
  /dev/disk2s4     90Gi  1.0Gi   18Gi     6%       2 941669998    0%   /System/Volumes/VM
  /dev/disk2s2     90Gi  305Mi   18Gi     2%    1038 941668962    0%   /System/Volumes/Preboot
  /dev/disk2s6     90Gi  496Ki   18Gi     1%      18 941669982    0%   /System/Volumes/Update
  /dev/disk2s1     90Gi   56Gi   18Gi    76%  380860 941289140    0%   /System/Volumes/Data
  map auto_home     0Bi    0Bi    0Bi   100%       0         0  100%   /System/Volumes/Data/home
  administrator@test-orka-macos11-x64-1 ~ % sudo reboot
  administrator@test-orka-macos11-x64-1 ~ % Connection to 199.7.167.102 closed by remote host.
  ➜  ~ ssh administrator@test-orka-macos11-x64-1
  administrator@test-orka-macos11-x64-1 ~ % uptime
  14:28  up 46 secs, 1 user, load averages: 3,48 1,03 0,38
  administrator@test-orka-macos11-x64-1 ~ % sudo du -sh /System/Library/Caches/* | sort -h
  0B	/System/Library/Caches/OnDemandResources
  0B	/System/Library/Caches/com.apple.CVMS
  0B	/System/Library/Caches/com.apple.factorydata
  0B	/System/Library/Caches/com.apple.kernelcaches
  0B	/System/Library/Caches/com.apple.kext.caches
  4,0K	/System/Library/Caches/MobileActivation
  4,0K	/System/Library/Caches/com.apple.SystemMigrationUserSizeCache
  4,0K	/System/Library/Caches/com.apple.apfsd.stats.plist
  8,0K	/System/Library/Caches/com.apple.Components2.SystemCache.Components
  12K	/System/Library/Caches/com.apple.IntlDataCache.le
  124K	/System/Library/Caches/com.apple.IntlDataCache.le.kbdx
  9,6M	/System/Library/Caches/com.apple.preferencepanes.systemcache
  36G	/System/Library/Caches/com.apple.coresymbolicationd
  administrator@test-orka-macos11-x64-1 ~ % df -h
  Filesystem       Size   Used  Avail Capacity iused     ifree %iused  Mounted on
  /dev/disk2s5s1   90Gi   14Gi   19Gi    43%  553788 941116212    0%   /
  devfs           188Ki  188Ki    0Bi   100%     650         0  100%   /dev
  /dev/disk2s4     90Gi  1.0Mi   19Gi     1%       1 941669999    0%   /System/Volumes/VM
  /dev/disk2s2     90Gi  305Mi   19Gi     2%    1038 941668962    0%   /System/Volumes/Preboot
  /dev/disk2s6     90Gi  504Ki   19Gi     1%      17 941669983    0%   /System/Volumes/Update
  /dev/disk2s1     90Gi   55Gi   19Gi    75%  379982 941290018    0%   /System/Volumes/Data
  map auto_home     0Bi    0Bi    0Bi   100%       0         0  100%   /System/Volumes/Data/home
  administrator@test-orka-macos11-x64-1 ~ %

Personal conclusions
I can confirm the hypothesis made by @AshCripps, seems a solid solution in long term, we can restore a lot of space if we delete the cache files on the machine and reboot.

Just some examples:

• test-nearform-macos11.0-arm64-1 has 182G /System/Volumes/Data/System/Library/Caches/com.apple.coresymbolicationd/data.
• test-orka-macos11-x64-1 has 36G /System/Library/Caches/com.apple.coresymbolicationd/data
• test-orka-macos11-x64-2 has 24G /System/Library/Caches/com.apple.coresymbolicationd/data

Additionally We can also retrieve some marginal gigabytes from macos updates, downloads, etc...

I see several options to overcome this challenge:

1. We might want to setup some monitorization/metrics that we can track in Grafana in order to we aware of machines with disk space problems (as we do already for infra)
2. We need some kind of automation to delete cache files and reboot the machines once in a while, but keeping in mind that M1 Nearform machines can be a problem (TBC @mhdawson )

Potential roadmap
I have an old iMac machine that I can repurpose for local testing the following ideas:

• Define and test a simple script to clean cache and reboot, before we develop a proper Jenkins job ( Clean up macOS cache and more #3089)
• Define a clear mesure on how much space do we actually need to run the build as suggested by @richardlau
• Define a PoC to send machines metrics to Grafana using Node exporter and prometheus or similar

[github-actions]

This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.