No space left on device error (even though there's plenty)

[realyze]

First, thanks for mup, it's an awesome tool!

I'm deploying on EC2 and I've been getting "no disk space" errors recently. But when I SSH into the box and run df -h, I see:

Filesystem      Size  Used Avail Use% Mounted on
udev            484M   12K  484M   1% /dev
tmpfs           100M  572K   99M   1% /run
/dev/xvda1       24G   13G  9.7G  57% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none            497M  476K  496M   1% /run/shm
none            100M     0  100M   0% /run/user

So there should be enough space (I'd think 9.7 GB is plenty).

This is the error I'm getting:

Error processing tar file(exit status 1): open /programs/server/npm/node_modules/react-router/node_modules/create-react-class/node_modules/fbjs/lib/someSet.js.flow: no space left on device
	-----------------------------------STDOUT-----------------------------------
	base: Pulling from abernix/meteord
	Digest: sha256:680d55194f4cce47dac3373ad54d7c3ca53e28e7c8f40d29ef43f82fa460da4b
	Status: Image is up to date for abernix/meteord:base
	sharewaste
	Sending build context to Docker daemon  256.6MB


	Step 1/15 : FROM abernix/meteord:base
	# Executing 1 build trigger...
	Step 1/1 : ENV NODE_VERSION ${NODE_VERSION:-4.8.3}
	 ---> Using cache
	 ---> 95e2bed2dd09
	Step 2/15 : RUN mkdir /built_app
	 ---> Using cache
	 ---> 15751d26d2dd
	Step 3/15 : COPY ./ /built_app
	sharewaste

Any ideas please?

[nzomedia]

If there's enough space on disk you could check to see if there's still free inodes with df -ih
source: https://unix.stackexchange.com/questions/40218/tar-no-space-left-on-device

[elie222]

I get the same error. On MUP 1.3.3. As for inodes I get:

Filesystem     Inodes IUsed IFree IUse% Mounted on
devtmpfs         469K   444  469K    1% /dev
tmpfs            471K     1  471K    1% /dev/shm
/dev/xvda1       512K  439K   74K   86% /

[elie222]

@realyze any luck with this so far?

[edemaine]

FWIW, I recently ran out of inodes on my server, so I appreciate @nzomedia's response!

[derouck]

I encountered the same problem. It turned out there were plenty of diff dirs under /var/lib/docker/aufs/diff

It was a problem in docker itself that the docker prune command did not clean enough. After updating to docker 17.10.0-ce the problem was resolved by doing the prune command.

docker system prune -af

[realyze]

Sorry guys, I lost track of this. My inode count was pretty low so I guess that was it. I have updated docker to 17.11 and ran docker system prune -a and that helped massively.