Consume container fails to start in Docker installation

[SummittDweller]

This looks like an absolutely terrific project! I've been trying to 'Dockerize' most of my home automation lately and this really fits the bill... but I can't seem to get it working. 8^(

I've spun up the Docker version on a CentOS 7 desktop and see no evidence that it is consuming anything. So, I added my favourite Docker companion, Portainer, to the mix to try and see what's up. Well, Portainer shows me that the 'consume' container is stopped. Checking its logs I find this...

Operations to perform:
  Apply all migrations: admin, auth, contenttypes, documents, reminders, sessions
Running migrations:
  No migrations to apply.
Starting document consumer at /consume with inotify
Traceback (most recent call last):
  File "/usr/src/paperless/src/manage.py", line 11, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python3.6/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python3.6/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/usr/src/paperless/src/documents/management/commands/document_consumer.py", line 98, in handle
    self.loop_inotify(mail_delta)
  File "/usr/src/paperless/src/documents/management/commands/document_consumer.py", line 128, in loop_inotify
    inotify.add_watch(directory, flags.CLOSE_WRITE | flags.MOVED_TO)
  File "/usr/lib/python3.6/site-packages/inotify_simple/inotify_simple.py", line 103, in add_watch
    return _libc_call(_libc.inotify_add_watch, self.fd, path, mask)
  File "/usr/lib/python3.6/site-packages/inotify_simple/inotify_simple.py", line 68, in _libc_call
    raise OSError(errno, os.strerror(errno))
OSError: [Errno 28] No space left on device

But I can't find any volumes on this device that are lacking space. In fact, the /home folder here, where nearly everything lives, has almost 1.5 TB of free space.

I've tried mapping my 'consume' folder from the project root and from my user's home directory with the same result in both cases. Currently 'consume' is wide open (chmod 777) and as shown here:

[mark@centos7 consume]$ pwd
/home/mark/consume
[mark@centos7 consume]$ ll
total 3896
-rw-r--r--. 1 mark mark   83059 Jun 24 13:15 Contact-_1.pdf
-rw-r--r--. 1 mark mark   20332 Jun 24 13:13 Contact-24hawki.pdf

And my 'docker-compose.yml' maps this as:

        volumes:
            - data:/usr/src/paperless/data
            - media:/usr/src/paperless/media
            # You have to adapt the local path you want the consumption
            # directory to mount to by modifying the part before the ':'.
            - /home/mark/consume:/consume

What am I missing here? Thanks in advance for any assistance. And may I suggest incorporating Portainer into your 'docker-compose.yml' config if you also find it useful.

[Strubbl]

• Have you enough inodes free on your home directory?
• Are you running any encryption on your consume directory?
• Is the path to the consume directory mounted read-write and not read-only?
• What type of filesystem is the consume directory?

I do not know portainer. What is this software?

Maybe this article is also relevant for your problem, because you are running out of inotify watchers: https://github.com/guard/listen/wiki/Increasing-the-amount-of-inotify-watchers

[SummittDweller]

Check it out... https://portainer.io/. I find it very useful.

I don't think it's an inode shortage...

[mark@centos7 paperless]$ df -i
Filesystem                  Inodes  IUsed      IFree IUse% Mounted on
/dev/mapper/centos-root   26214400 226228   25988172    1% /
devtmpfs                    984827    518     984309    1% /dev
tmpfs                       989164     36     989128    1% /dev/shm
tmpfs                       989164   1074     988090    1% /run
tmpfs                       989164     16     989148    1% /sys/fs/cgroup
/dev/sda2                   524288     46     524242    1% /boot
/dev/sda1                        0      0          0     - /boot/efi
/dev/mapper/centos-home  140332224 275869  140056355    1% /home
tmpfs                       989164     37     989127    1% /run/user/1000
/dev/sdf1               4294967295 488504 4294478791    1% /run/media/mark/Private

No encryption that I know of.
The path to the 'consume' directory is local, not a mount and it's wide open...777.
The filesystem looks like 'xfs'...

[mark@centos7 paperless]$ lsblk -f
NAME            FSTYPE      LABEL   UUID                                   MOUNTPOINT
sda                                                                        
├─sda1          vfat                F5A7-AF83                              /boot/efi
├─sda2          xfs                 159e4558-3221-42f2-8185-90389a5815c4   /boot
└─sda3          LVM2_member         VYFrEp-pc3I-dvey-Gvf6-Tph5-fSVl-dt16CP 
  ├─centos-root xfs                 054147dc-a676-4977-983e-b7918a1f2df1   /
  ├─centos-swap swap                f0b722e0-5977-47dc-875f-ab25c5aa6d1d   [SWAP]
  └─centos-home xfs                 6ca1d288-b6c2-44f9-8b0f-aaec7995e5cf   /home

[Strubbl]

Can you check the output of lsof | grep inotify | wc -l against the limit (/proc/sys/fs/inotify/max_user_watches)?

[SummittDweller]

910 vs. 8192, but that's on the 'host'. Do those values also apply to the container? I'm guessing they do not, but am not sure how to check that since the container starts and stops within seconds of its creation.

[SummittDweller]

I may try spinning this up 'bare metal' on the same CentOS 7 box just to see what happens.

Er, maybe not. Looking at the requirements and having second thoughts. I guess that's why I appreciate Docker so much. 8^)

[SummittDweller]

This thing rocks! So, I honestly didn't believe that making a config change to the host would solve this problem, but it did! The change I made was suggested for a similar error in a different Docker context...

cat /proc/sys/fs/inotify/max_user_watches    # default is 8192 
sudo sysctl fs.inotify.max_user_watches=1048576    # increase to 1048576

Now the consume container runs and it does indeed consume documents dropped in the target folder. Awesome.