New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inode count incorrectly at 100% when not presented from host #317

Open
funkypenguin opened this Issue Jul 27, 2017 · 3 comments

Comments

Projects
None yet
3 participants
@funkypenguin
Copy link

funkypenguin commented Jul 27, 2017

I'm using a cephfs-mounted filesystem for persistent container storage, and have just encountered an error on the minio/minio container, which seems to be rooted in the alpine image.

Background

CephFS mount point does not show any inode count, when running 'df -i' on the docker host. This is (I believe) because inodes are irrelevant under Ceph:

[root@ds3 data]# df -i
Filesystem                  Inodes IUsed    IFree IUse% Mounted on
/dev/mapper/atomicos-root 12021760 36991 11984769    1% /
devtmpfs                    480059   424   479635    1% /dev
tmpfs                       485235     1   485234    1% /dev/shm
tmpfs                       485235   855   484380    1% /run
tmpfs                       485235    16   485219    1% /sys/fs/cgroup
/dev/vda1                   153600   328   153272    1% /boot
tmpfs                       485235     1   485234    1% /run/user/1000
ds3:6789:/                   28552     -        -     - /var/data
[root@ds3 data]#

Fault under Alpine

When I bind-mount a volume in an Alpine container under /var/data (above), the "ash" shell within the container reports 100% inode usage (which breaks minio):

[root@ds3 data]# docker run -d --name alpinetest -v `pwd`/test/ubuntu:/export alpine sleep 10m
Unable to find image 'alpine:latest' locally
Trying to pull repository docker.io/library/alpine ...
latest: Pulling from docker.io/library/alpine
88286f41530e: Already exists
Digest: sha256:1072e499f3f655a032e88542330cf75b02e7bdf673278f701d7ba61629ee3ebe
Status: Downloaded newer image for docker.io/alpine:latest
6574342761d3724bdf0272862e0f5846b342cc588077fa0b97f3aac3e3781b6d
[root@ds3 data]#
[root@ds3 data]# docker exec -it alpinetest ash
/ # df -i
Filesystem              Inodes      Used Available Use% Mounted on
/dev/mapper/docker-253:0-5101458-749ad06efdcce72d2483a872b1a63fabb9950f3517f8566fada2eedf6615a106
                       5238784       481   5238303   0% /
tmpfs                   485235        16    485219   0% /dev
tmpfs                   485235        16    485219   0% /sys/fs/cgroup
ds3:6789:/               28556     28557 18446744073709551615 100% /export
/dev/mapper/atomicos-root
                      12021760     36991  11984769   0% /etc/resolv.conf
/dev/mapper/atomicos-root
                      12021760     36991  11984769   0% /etc/hostname
/dev/mapper/atomicos-root
                      12021760     36991  11984769   0% /etc/hosts
shm                     485235         1    485234   0% /dev/shm
/dev/mapper/atomicos-root
                      12021760     36991  11984769   0% /run/secrets
tmpfs                   485235        16    485219   0% /proc/kcore
tmpfs                   485235        16    485219   0% /proc/timer_list
tmpfs                   485235        16    485219   0% /proc/timer_stats
tmpfs                   485235        16    485219   0% /proc/sched_debug
tmpfs                   485235         1    485234   0% /sys/firmware
/ #

Fault not reproducible under Ubuntu

I first assumed this to be a docker error, but testing under Ubuntu does not reproduce the issue:

[root@ds3 data]# docker run -d --name ubuntutest -v `pwd`/test/ubuntu:/export ubuntu sleep 10m
0bbbbdbff943067fb20cea0f3db23b486dbd5c41b53dc20cc982254ce7cf7b10
[root@ds3 data]# docker exec -it ubuntutest bash
root@0bbbbdbff943:/# df -i
Filesystem                                                                                          Inodes IUsed    IFree IUse% Mounted on
/dev/mapper/docker-253:0-5101458-ab27088cd7b9e4cbc413f07f149b9abdbae6c6f45e087369bdee4d82af754b7a  5238784  5511  5233273    1% /
tmpfs                                                                                               485235    16   485219    1% /dev
tmpfs                                                                                               485235    16   485219    1% /sys/fs/cgroup
ds3:6789:/                                                                                           28552     -        -     - /export
/dev/mapper/atomicos-root                                                                         12021760 36988 11984772    1% /etc/hosts
shm                                                                                                 485235     1   485234    1% /dev/shm
tmpfs                                                                                               485235     1   485234    1% /sys/firmware
root@0bbbbdbff943:/#
@tianon

This comment has been minimized.

Copy link
Collaborator

tianon commented Jul 28, 2017

@funkypenguin

This comment has been minimized.

Copy link

funkypenguin commented Jul 28, 2017

So, assuming that coreutils makes GNU's "df' the default, I now get the following (correct) result:

[root@ds3 ~]# docker exec -it alpinetest ash
/ # apk add coreutils --no-cache
fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/community/x86_64/APKINDEX.tar.gz
(1/3) Installing libattr (2.4.47-r6)
(2/3) Installing libacl (2.2.52-r3)
(3/3) Installing coreutils (8.27-r0)
Executing busybox-1.26.2-r5.trigger
OK: 9 MiB in 14 packages
/ # df -i
Filesystem                                                                                          Inodes IUsed    IFree IUse% Mounted on
/dev/mapper/docker-253:0-5101458-368c8374af46c372d8403b60df5a24ec47e6b2f80d3edea7a560654734898d3e  5238784   517  5238267    1% /
tmpfs                                                                                               485235    16   485219    1% /dev
tmpfs                                                                                               485235    16   485219    1% /sys/fs/cgroup
ds3:6789:/                                                                                           28603     -        -     - /export
/dev/mapper/atomicos-root                                                                         12021760 37217 11984543    1% /etc/hosts
shm                                                                                                 485235     1   485234    1% /dev/shm
tmpfs                                                                                               485235     1   485234    1% /sys/firmware
/ #

@andyshinn andyshinn added the question label Jul 28, 2017

@funkypenguin

This comment has been minimized.

Copy link

funkypenguin commented Jul 29, 2017

So, although the results in "df" are correct with the addition of coreutils, minio still refuses to accept new files, claiming "disk is full". The minio commit which introduced this behaviour is minio/minio@1ad5fb8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment