New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"No space left on device" after a bunch of builds #35

Closed
apjanke opened this Issue Jun 25, 2018 · 5 comments

Comments

2 participants
@apjanke
Contributor

apjanke commented Jun 25, 2018

So, apparently, on a macOS High Sierra 10.13.5 system with an APFS drive, doing a bunch of these builds, possibly in combination with using trash, can fill up your drive and leave you with a system that is COMPLETELY FUXXORED.

I tried to create a new VM in vmWare Fusion, and it gave me a "low disk space" error. At that point, my system started misbehaving.

I tried to empty the Trash by right-clicking on the Trash icon in the dock, and it brought up an "Emptying the Trash..." dialog that hung. I just get a spinning beach ball when I move my cursor over it.

Finder is hung; spinning beach ball there.

df shows my drives as full, and reports a nonsensical number for ifree for my APFS volumes. Smells like an overflow error.

$ df                                                                                                                                                  ✘ 1 master
Filesystem                                         Size   Used  Avail Capacity   iused               ifree %iused  Mounted on
/dev/disk1s1                                      743Gi  723Gi  204Mi   100%   6750637 9223372036848025170    0%   /
devfs                                             204Ki  204Ki    0Bi   100%       706                   0  100%   /dev
/dev/disk1s4                                      743Gi   19Gi  204Mi    99%        20 9223372036854775787    0%   /private/var/vm
/dev/disk0s3                                      188Gi  179Gi  8.5Gi    96%   1353850             9039086   13%   /Volumes/BOOTCAMP
map -hosts                                          0Bi    0Bi    0Bi   100%         0                   0  100%   /net
map auto_home                                       0Bi    0Bi    0Bi   100%         0                   0  100%   /home
//janke@DiskStation._afpovertcp._tcp.local/share  2.7Ti  1.8Ti  853Gi    69% 496162589           223651949   69%   /Volumes/share

rm and unlink do not work.

[~/Downloads]
$ rm -f Octave_44_Yosemite.dmg                                                                                                                       ✘ 64 master
rm: Octave_44_Yosemite.dmg: No space left on device
[~/Downloads]
[~/Downloads]
$ unlink Octave_44_Yosemite.dmg                                                                                                                       ✘ 1 master
unlink: Octave_44_Yosemite.dmg: No space left on device
[~/Downloads]
[~/Downloads]
$

I deleted all my saved Time Machine local snapshots, and that didn't help anything.

$ tmutil deletelocalsnapshots 2018-06-15-150317                                                                                                           master
Deleted local snapshot '2018-06-15-150317'
$ tmutil deletelocalsnapshots 2018-06-24-150317                                                                                                           master
Deleted local snapshot '2018-06-24-150317'
$ tmutil deletelocalsnapshots 2018-06-25-020452                                                                                                           master
Deleted local snapshot '2018-06-25-020452'
$ tmutil listlocalsnapshots /
$

sudo does not work.

$ sudo mount -t hdfs /dev/disk2s1 /Volumes/EXTERNAL_DRIVE master
sudo: unable to write to /var/db/sudo/ts/janke: No space left on device

I ran Disk Utility by doing a cd /Applications/Utilities/Disk Utility.app/Contents; "./MacOS/Disk Utility", clicked "First Aid", and clicked "Run", and then "Continue". It said:

First Aid found corruption that needs to be repaired. To repair the startup volume, run First Aid from Recovery. Click Done to continue.

Under the Show Details, it said:

Verifying file system.
Volume could not be unmounted.
Using live mode.
Performing fsck_apfs -n -l -x /dev/rdisk1s1
error: mkdir: /private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/fsck_apfs.30326.1: No space left on device
error: nowhere to mount crypto dev
File system check exit code is 73.
Restoring the original state found as mounted.
Operation successful.

"No space left on device". I'm seeing a recurring theme here.

diskutil apfs list hangs after producing this output:

$ diskutil apfs list
APFS Container (1 found)
|
+-- Container disk1 D956ED19-DCA9-490B-9B8B-35550CF581A9
    ====================================================
    APFS Container Reference:     disk1
    Size (Capacity Ceiling):      798011404288 B (798.0 GB)

I'm out of ideas, except to reboot and hope APFS's equivalent of fsck fixes things.

To try to prevent this in the future, remove our use of trash and replace it with plain rm -rf.

@apjanke apjanke added the bug label Jun 25, 2018

@apjanke apjanke added this to Needs triage in Octave.app via automation Jun 25, 2018

@apjanke apjanke moved this from Needs triage to Low priority in Octave.app Jun 25, 2018

@apjanke apjanke moved this from Low priority to High priority in Octave.app Jun 25, 2018

@schoeps

This comment has been minimized.

Show comment
Hide comment
@schoeps

schoeps Jun 25, 2018

Contributor

We could just rm old builds instead of putting them into the trash? I wanted to be polite but for a bot this may not matter :)

"$trash" "$dmg_dir/$dmg_name"

Or is this error from the DMG? Then we may increase the size

--disk-image-size 2000 \

Contributor

schoeps commented Jun 25, 2018

We could just rm old builds instead of putting them into the trash? I wanted to be polite but for a bot this may not matter :)

"$trash" "$dmg_dir/$dmg_name"

Or is this error from the DMG? Then we may increase the size

--disk-image-size 2000 \

@apjanke

This comment has been minimized.

Show comment
Hide comment
@apjanke

apjanke Jun 25, 2018

Contributor

Yeah: I'm just guessing that trash is involved here, but yeah, I would like to switch it to do rm -rf instead, to be conservative.

I've also added a recommendation about excluding the /Applications and build directories from Time Machine backups, since I suspect that's part of the root issue here: octave-app/octave-app.github.io@23f7272

It's not the DMG that's running out of room, it's my actual system hard drive. :(

Contributor

apjanke commented Jun 25, 2018

Yeah: I'm just guessing that trash is involved here, but yeah, I would like to switch it to do rm -rf instead, to be conservative.

I've also added a recommendation about excluding the /Applications and build directories from Time Machine backups, since I suspect that's part of the root issue here: octave-app/octave-app.github.io@23f7272

It's not the DMG that's running out of room, it's my actual system hard drive. :(

@apjanke

This comment has been minimized.

Show comment
Hide comment
@apjanke

apjanke Jun 25, 2018

Contributor

Here's the change for switching to rm: 72f868d

Contributor

apjanke commented Jun 25, 2018

Here's the change for switching to rm: 72f868d

@apjanke

This comment has been minimized.

Show comment
Hide comment
@apjanke

apjanke Jun 25, 2018

Contributor

(BTW, I suspect this whole issue is really an APFS + Time Machine bug. This behavior should Not Happen on a well-behaved filesystem, regardless of how much file churn there has been.)

Contributor

apjanke commented Jun 25, 2018

(BTW, I suspect this whole issue is really an APFS + Time Machine bug. This behavior should Not Happen on a well-behaved filesystem, regardless of how much file churn there has been.)

@apjanke

This comment has been minimized.

Show comment
Hide comment
@apjanke

apjanke Jun 27, 2018

Contributor

Calling this closed since I think we have a workaround for devs to use, and it doesn't affect users.

Contributor

apjanke commented Jun 27, 2018

Calling this closed since I think we have a workaround for devs to use, and it doesn't affect users.

@apjanke apjanke closed this Jun 27, 2018

Octave.app automation moved this from High priority to Closed Jun 27, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment