df (and other commands) fail on /root/.cache/doc

[amurashkin17]

df fails on /root/.cache/doc. Various system automation scripts break down as the result (scripts relying on df returning code 0).

# df
df: /root/.cache/doc: Operation not permitted
Filesystem                      1K-blocks      Used Available Use% Mounted on
devtmpfs                         65781416         0  65781416   0% /dev
tmpfs                            65836036   1674056  64161980   3% /dev/shm
tmpfs                            65836036      3132  65832904   1% /run
tmpfs                            65836036         0  65836036   0% /sys/fs/cgroup

# echo -e "exit code $?"
exit code 1

Commands to pinpoint the issue

# ps -ef | grep fuse
root     1210882 1210873  0 Jul05 ?        00:00:00 fusermount -o rw,nosuid,nodev,fsname=portal,auto_unmount,subtype=portal -- /root/.cache/doc

# pstree -psula 1210882
systemd,1 --switched-root --system --deserialize 30
  └─xdg-document-po,1210873
      └─fusermount,1210882 -o rw,nosuid,nodev,fsname=portal,auto_unmount,subtype=portal -- /root/.cache/doc

# mount | grep cache/doc
portal on /root/.cache/doc type fuse.portal (rw,nosuid,nodev,relatime,user_id=0,group_id=0)

Software versions

• Fedora 32
• kernel-5.6.14-300.fc32.x86_64
• xdg-desktop-portal-1.7.2-1.fc32.x86_64
• fuse-2.9.9-9.fc32.x86_64

The same issue has been reported in Ubuntu 20.04 - df command throws error on /run/user/1000/doc ...

[matthiasclasen]

Don't log in as root, I'd say. If you are using the root account for a normal user session, you'll get typical session components running. I don't think that is surprising, or a bug.

[TomaszGasior]

@matthiasclasen This bug is not about running something as root, just example given in first post is misleading. This issue should be reopened or new created IMHO.

The problem occurs with /run/user/1000/doc path for example. To see the problem, please run some Flatpak app, Shortwave in my example, and then try to run df. You will get df: /run/user/1000/doc: Operation not permitted error and non-zero exit code.

[rocketraman]

Don't log in as root, I'd say. If you are using the root account for a normal user session, you'll get typical session components running. I don't think that is surprising, or a bug.

@matthiasclasen I just ran into this issue on my Fedora 32 box. I don't use root as a normal user session, so I have no idea why this exists.

[TomaszGasior]

@rocketraman This problem is not related to be using root account at all. That's just confusing example in original issue.

[rocketraman]

@TomaszGasior Thanks. I can't pretend to understand what is happening here, as I know very little about flatpak (and to my knowledge, have never used it) but some more information.

Like the OP, I get the error on /root/.cache/doc, not just /run/user/1000/doc. Looking at the pstree, I see a process with parent id 1 xdg-document-portal:

root     1836537       1  0 Sep29 ?        00:00:00 /usr/libexec/xdg-document-portal
root     1836549 1836537  0 Sep29 ?        00:00:00 fusermount -o rw,nosuid,nodev,fsname=portal,auto_unmount,subtype=portal -- /root/.cache/doc

Doing a systemctl status on that PID shows it is associated with my normal user session, not with any type of root session:

# systemctl status 1836537
● session-3.scope - Session 3 of user raman
     Loaded: loaded (/run/systemd/transient/session-3.scope; transient)
  Transient: yes
     Active: active (running) since Fri 2020-09-25 14:14:52 EDT; 1 weeks 4 days ago
      Tasks: 4031
     Memory: 33.9G
        CPU: 1w 2d 5h 34min 12.490s
     CGroup: /user.slice/user-1000.slice/session-3.scope
...

[rocketraman]

Given that information above that the xdg-document-portal process that is causing the issue belongs to a user session, not a root session, should this issue be re-opened @matthiasclasen ?

[mmstick]

Ubuntu 20.10 upgrades / installs are now seeing this with the flatpak package.

[ghost]

Ubuntu 20.10 upgrades / installs are now seeing this with the flatpak package.

Just a random user here, and I can confirm. I got this error when I tried running my script which uses df.

[rowanthorpe]

This seems to be low-level, to do with xdg-desktop-portal (ex xdg-document-portal), in particular "portal" mounts, and possibly to do with the display-manager running things as wrong user(?), or a zombie dbus-session that should never have been run - perhaps launched unnecessarily by a tty-login as root(?) - see below. I encountered this because I started receiving the following daily email from anacron on my Debian system:

/etc/cron.daily/checksecurity:
df: /root/.cache/doc: Operation not permitted

The "checksecurity" package (in particular the check-diskfree plugin) is configured by default to test for & alert if >70% usage on any mounted disks, as follows:

/etc/cron.daily/checksecurity =>
 => /usr/sbin/checksecurity
   -> /etc/checksecurity.conf (CHECK_DISKFREE="TRUE")
   -> /etc/checksecurity/check-diskfree.conf (CHECK_DISK_PERCENT="xx%"
 => /usr/share/checksecurity/check-diskfree [= df -klP | ... =]

When I run df -klPas root I get the same error-output for /root/.cache/doc as is triggered by the cron job so the issue is triggered on the df level, not on the checksecurity level. When I run it as logged in desktop user I don't strike this problem, so I assume flatpak users are striking the issue with non-root users too because it sets up something equivalently incorrectly to what root has on my system. Also, when I even run plain df (with no flags) as root I still see the error so it is unrelated to specific flags.

From what I briefly read it seems there are three problems (or at least two problems, and one non-obvious "non-problem"). The first is when df tries to read the portal userspace filesystem which is a pseudo-fs frontend to dbus (socket?) semantics, and df is prevented by the kernel from trying to read it the usual way (if correct that means one part of the fix should be somehow blocklisting the portal filesystem from typical parsing by tools like df without them throwing an error). The second part seems to be that xdg-desktop-portal, xdg-document-portal, etc are being launched as root user - either incorrectly, or perhaps deliberately (because it seems to be a "permissions firewall" to add finer-tuned ACLs when using the blunt-instrument perms of dbus anyway, so running it as the logged-in user would mean obfuscation rather than true enforcement). The third part is that - as can be seen in the fusermount line in the ps output below - the fuse.portal FS is mounted with auto_unmount option but is still sticking around more than a week after it could have ever been used, so is clearly not auto-unmounting.

In my output of mount I see the following:

portal on /root/.cache/doc type fuse.portal (rw,nosuid,nodev,relatime,user_id=0,group_id=0)

and output of ps axufww includes the following for root (all launched at the same time 10 days ago) but not for my actual desktop user. IIRC about ten days ago I did a general update of system packages using apt, so they may have got autolaunched during that. I think I restarted my user-desktop since then so that might be why only the root-user ones remain:

root      413664  0.0  0.0  13892   660 ?        S    Oct25   0:00  \_ dbus-launch --autolaunch=[XXX] --binary-syntax --close-stderr
root      413665  0.0  0.0   9848   428 ?        Ss   Oct25   0:00  \_ /usr/bin/dbus-daemon --syslog-only --fork --print-pid 5 --print-address 7 --session
root      413668  0.0  0.0 462620  1864 ?        Sl   Oct25   0:07  \_ /usr/libexec/xdg-desktop-portal
root      413673  0.0  0.0 454788  1116 ?        Sl   Oct25   0:00  \_ /usr/libexec/xdg-document-portal
root      413682  0.0  0.0   2324   484 ?        Ss   Oct25   0:00  |   \_ fusermount -o rw,nosuid,nodev,fsname=portal,auto_unmount,subtype=portal -- /root/.cache/doc
root      413677  0.0  0.0 232856   676 ?        Sl   Oct25   0:00  \_ /usr/libexec/xdg-permission-store
root      413688  0.0  0.0 495956  4020 ?        Sl   Oct25   0:25  \_ /usr/libexec/xdg-desktop-portal-gtk
root      413692  0.0  0.0 236996  2416 ?        Sl   Oct25   0:00  \_ /usr/libexec/gvfsd
root      413697  0.0  0.0 379904   820 ?        Sl   Oct25   0:00  \_ /usr/libexec/gvfsd-fuse /root/.cache/gvfs -f
root      413708  0.0  0.0 155992  1924 ?        Sl   Oct25   0:00  \_ /usr/libexec/dconf-service
root      413715  0.0  0.0 237300  2224 ?        Sl   Oct25   0:00  \_ /usr/bin/gnome-keyring-daemon --start --foreground --components=secrets

FYI: I am very careful to never run any graphical desktop environments or window-managers as root though, and only very rarely run a graphical app as root (mapping DESKTOP= to my user's desktop) when necessary for obscure things like packet-inspection when there isn't time to mess around with tweaking posix capabilities the "right way" (or if I need to compare/edit root-owned config-files with meld, etc). Am using lightdm DM with cinnamon DE at the moment. If you need other info let me know.

I guess other users on this thread might see similar outputs to the above for other users in addition to/instead of root(?).

I just found the following messages in a thread where Fedora users are in the same position as me:

• https://www.spinics.net/linux/fedora/fedora-users/msg500522.html
• https://www.spinics.net/linux/fedora/fedora-users/msg500523.html
• https://www.spinics.net/linux/fedora/fedora-users/msg500524.html

...and regarding the --user systemd services listed in that second email, I don't even have any of them on my system (now). I remember I pruned a lot of stuff during my big update, and there is a chance that something I pruned had previously caused flatpak to be autoinstalled, and it was consequently uninstalled by pruning that. I ran checkrestart to see what was still running as an orphan after removal/upgrade of packages, and sure enough found the following:

% sudo checkrestart
...
These processes (31) do not seem to have an associated init script to restart them:
...
xdg-desktop-portal:
	413668	/usr/libexec/xdg-desktop-portal
xdg-desktop-portal-gtk:
	413688	/usr/libexec/xdg-desktop-portal-gtk
...

So my personal solution is simple - to just kill those processes which are no longer even present or needed on my system. Note that I had to run sudo fusermount -u /root/.cache/doc rather than just killing the fusermount process, and that I had to kill the dbus-launch process with HUP signal (or maybe I was just too impatient to wait for TERM to reap it...). For others who actually need to keep these though, but to have them run correctly, my solution is obviously not theirs. Hopefully my diagnosis above helps point to their solution though. HTH.

[isi-floss]

This comment describes that the error for /run/user/1000/doc is expected behaviour if df is run as non-root:

What happens here is that df tries to use statfs call on all mountpoints but that's being denied in default config for fuse filesystems which is what /run/user/1000/doc is. There is no bug here, just things working as expected.

You may easily reproduce it with stat -f /run/user/1000/doc.

If you desperately want to get rid of Operation not permitted message then run df as root, i.e. sudo df however I don't see the point of this.

[ricardoamaro]

This was affecting my Fedora Server box, that I use as a server and not as a destop. Not sure why xdg-desktop-portal comes with it. Therefore, even monitoring of partitions was seeing this error.
I stopped the service:
systemctl --user stop xdg-document-portal.service and df -hT worked fine.
Particular case, but since I don't use flatpak here I will be removing it.
Hope that helps.

[matthiasclasen]

I would guess that the document portal fuse implementation doesn't implement some operation that df needs, or doesn't implement it in the right way. I think that is very common with fuse implementations...

[alexlarsson]

The document portal is a fuse mount, so it is denied access to by any uid other than the owning uid, including root. So, I expect the regular /run/user/1000/doc to be inaccessible by root. However, that is probably not an issue because df rarely runs in /run, being a tmpfs etc.

However, if XDG_RUNTIME_DIR is not set, then the default value is $XDG_CACHE_HOME (which defaults to ~/.cache/, so if that happens we'll get a mount inside the homedir which may be problematic for things liks disk free checks.

The main issue though is that people seem to be getting an xdg-document-portal running as root, which isn't right. The document portal is automatically started by dbus when something tries to talk to it. The primary user of this is flatpak run, but It doesn't seem like people have been calling this as root. Maybe something else started the document portal? For example the gnome control center? (For the permissions pages.) Still, why would that be running as root?

[alexlarsson]

Also, people seem to be getting full gnome dbus sessions, including daemons like gvfs, dconf and gnome-keyring, so this isn't really purely a portal issue.

[rocketraman]

Also, people seem to be getting full gnome dbus sessions, including daemons like gvfs, dconf and gnome-keyring, so this isn't really purely a portal issue.

I can confirm this is right:

# ps -ef | grep -E "xdg-document-portal|gnome-keyring|gvfs|dconf"
raman      44416       1  0 Nov12 ?        00:00:01 /usr/bin/gnome-keyring-daemon --daemonize --login
raman      45375    3205  0 Nov12 ?        00:00:00 /usr/libexec/gvfsd
raman      47373    3205  0 Nov12 ?        00:00:00 /usr/libexec/dconf-service
raman      48789    3205  0 Nov12 ?        00:00:01 /usr/libexec/xdg-document-portal
raman      49444    3205  0 Nov12 ?        00:00:00 /usr/bin/gnome-keyring-daemon --start --foreground --components=secrets
root      110432       1  0 Nov12 ?        00:00:00 /usr/libexec/xdg-document-portal
root      110495       1  0 Nov12 ?        00:00:00 /usr/libexec/dconf-service
root      110514       1  0 Nov12 ?        00:00:00 /usr/libexec/gvfsd
root      110616       1  0 Nov12 ?        00:00:00 /usr/bin/gnome-keyring-daemon --start --foreground --components=secrets
raman     404891    3205  0 Nov13 ?        00:00:59 /usr/libexec/gvfs-udisks2-volume-monitor
raman     404896   45375  0 Nov13 ?        00:00:00 /usr/libexec/gvfsd-trash --spawner :1.4 /org/gtk/gvfs/exec_spaw/0
raman     404915   45375  0 Nov13 ?        00:00:00 /usr/libexec/gvfsd-network --spawner :1.4 /org/gtk/gvfs/exec_spaw/1
raman     404948   45375  0 Nov13 ?        00:00:00 /usr/libexec/gvfsd-dnssd --spawner :1.4 /org/gtk/gvfs/exec_spaw/3
raman    1691960    3205  0 Nov15 ?        00:00:00 /usr/libexec/gvfsd-metadata
raman    2481129       1  0 Nov17 ?        00:00:00 /usr/libexec/gvfsd
root     3343059  104584  0 09:25 pts/0    00:00:00 grep --color=auto -E xdg-document-portal|gnome-keyring|gvfs|dconf

I note that all of these running as root were started on Nov12th.

systemctl status <pid> on any of the root PIDs shows these are all associated with my normal user's session scope.

[christophgysin]

Please reopen this issue.

[julian-klode]

Just to note, a huge number of other fuse file systems don't show:

I have mounted:

$ grep fuse /proc/mounts 
fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0
https://sd2dav.1und1.de/ /media/jak/smartdrive fuse rw,nosuid,nodev,noexec,relatime,user_id=1000,group_id=1000,allow_other,max_read=16384 0 0
gvfsd-fuse /run/user/1000/gvfs fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=1000 0 0
portal /run/user/1000/doc fuse.portal rw,nosuid,nodev,relatime,user_id=1000,group_id=1000 0 0
keybase-redirector /keybase fuse ro,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0
/dev/fuse /run/user/1000/keybase/kbfs fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=1000 0 0

but:

$ /usr/bin/df  -T | grep fuse
/usr/bin/df: /run/user/1000/doc: Operation not permitted
https://sd2dav.1und1.de/      fuse  1333333332 800000000 533333332  61% /media/jak/smartdrive
/dev/fuse                     fuse   262144000         1 262144000   1% /run/user/1000/keybase/kbfs

Looking at some of them, they do implement statfs, and return 0 for all things.

$ stat -f /keybase/
  File: "/keybase/"
    ID: 0        Namelen: 0       Type: fuseblk
Block size: 0          Fundamental block size: 0
Blocks: Total: 0          Free: 0          Available: 0
Inodes: Total: 0          Free: 0

$ stat -f /run/user/1000/gvfs
  File: "/run/user/1000/gvfs"
    ID: 0        Namelen: 1024    Type: fuseblk
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 0          Free: 0          Available: 0
Inodes: Total: 0          Free: 0

Heck stat -f actually works on /run/user/1000/docs too, but this thing is owned by a root process. Which makes sense, as fusermount is setuid root.

This is the only FUSE filesystem using fusermount. So maybe that's the issue?

[joeyh]

Issue remains (debian unstable). I removed your software as a consequence of thus bug.

[ckujau]

See the comments above: this had been reported and fixed via a gnulib commit (see the comments above), so once the fixed version of du trickles down to your distribution (here: Debian/unstable), this should no longer be an issue. If this remains an issue, upstream should be notified about this. Removing xdg-desktop-portal may not always be possible, but I'm glad it worked for you.

[rocketraman]

so once the fixed version of du trickles down to your distribution (here: Debian/unstable), this should no longer be an issue.

I can confirm this is no longer an issue on Fedora 35.

[julian-klode]

while the issue has been worked around in df, I disagree with the assessment that this has been fixed. statfs() on the doc mountpoint returns EPERM. Other file systems happily return 0ed struct statfs and thus just work.

[pakair]

This is still an issue on Linux Mint 20.3 ... from a fresh install on a VM https://imgur.com/a/KcjsIPc

[TomaszGasior]

Ubuntu-based LTS-es tend to offer outdated software. When reporting the issue to upstream developers, to reproduce the issue it's better to use distros like Fedora or at least non-LTS Ubuntu. On my up-to-date Fedora 35 I am not able to reproduce the issue with df but it still occurs with dfc for example.

[spacelama]

The correct place to fix a FUSE filesystem failing to implement a system call is that fuse filesystem, not every binary that tries to stat filesystems as part of their job.

Every other fuse filesystem has no problem with this. sshfs?

35853,14> df sshfs/weinberg
Filesystem     1K-blocks  Used Available Use% Mounted on
afuse                  0     0         0    - /home/spacelama/sshfs
35854,15> echo $?
0

But to somehow consider stat() of my own filesystem an error and return EPERM breaking every single piece of user space code out there, is... a bit poor:

35855,16> df
df: /run/user/738/doc: Operation not permitted
...
35856,17> echo $?
1

[ParthSindhu]

This is still an issue in elementary OS 6.1 Jólnir

Although im not sure why, This cause firefox to repeatedly crash and freeze the system on saving downloaded file. By default firefox tries to download all file to /run/user/1000/doc/365bcfd0 and does not remember other download locations.

The issue was fixed by running systemctl --user stop xdg-document-portal.service