Can't drop capabilities from bounding set on ARM

[mariospr]

As mentioned in #149 (comment), we recently upgraded bubblewrap to 1.7 on Endless and observed this error now whenever we try to run any flatpak on our ARM product (it's fine on amd64, fwiw):

$ flatpak run -v com.endlessm.encyclopedia.pt
XA: No installations directory in /etc/flatpak/installations.d. Skipping
XA: Allowing dri access
XA: Allowing x11 access
XA: Allowing pulseaudio access
XA: Running '/usr/bin/bwrap --args 19 /usr/lib/flatpak/flatpak-dbus-proxy --fd=16 unix:abstract=/tmp/dbus-hwXi4V2Q6k,guid=470e40e6fb763ef90b027f125894dd37 /run/user/1001/.dbus-proxy/session-bus-proxy-9284UY --filter --own=com.endlessm.encyclopedia.pt --own=com.endlessm.encyclopedia.pt.* --talk=org.freedesktop.portal.*'
XA: Running '/usr/bin/bwrap --args 18 /usr/lib/flatpak/flatpak-dbus-proxy --fd=16 unix:path=/var/run/dbus/system_bus_socket /run/user/1001/.dbus-proxy/system-bus-proxy-X384UY --filter --talk=com.endlessm.Metrics'
Dropping capability 0 from bounds: Operation not permitted

Reading through man prctl, it's suggested that this EPERM error would mean that we don't have the CAP_SETPCAP capability in the calling thread, so one question is why that cap is not there on our ARM kernel (a question I'm actively trying to answer now already, but need some feedback of our kernel team first).

I was wondering if it would make sense to call prctl(PR_CAPBSET_READ, CAP_SETPCAP) in drop_cap_bounding_set(), early returning if it returns zero, but then I've tried the following patch:

--- bubblewrap-0.1.7.orig/bubblewrap.c
+++ bubblewrap-0.1.7/bubblewrap.c
@@ -482,6 +482,12 @@ drop_cap_bounding_set (void)
 {
   unsigned long cap;

+  if (prctl (PR_CAPBSET_READ, CAP_SETPCAP, 0, 0, 0) != 1)
+    return;
+
   for (cap = 0; cap <= 63; cap++)
     {
       int res = prctl (PR_CAPBSET_DROP, cap, 0, 0, 0);

...but flatpak run keeps failing anyway with the very same Dropping capability 0 from bounds: Operation not permitted error, so I'm extra confused now.

Also, I installed libcap-ng-utils and setpcap seems to be available:

$ filecap -d | grep cap
setpcap
setfcap
cap_37

Again, this only fails on ARM, so I'm pasting here a link to our kernel configuration for that case, in case it helps understanding the issue: https://github.com/endlessm/linux-meson/blob/debian-master/debian/config/armhf/config.meson8b

Last, I've got a device with me where I can reliably reproduce the problem by doing flatpak run <appid> so I'd be more than happy to do any test that were needed.

[cgwalters]

What does capsh --print show before invoking bwrap?

[mariospr]

$ capsh --print
Current: =
Bounding set =cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend
Securebits: 00/0x0/1'b0
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
uid=1001(mario)
gid=1001(mario)
groups=4(adm),27(sudo),108(lpadmin),121(systemd-journal),1001(mario)

$ flatpak run org.maemo.Numptyphysics 
Dropping capability 0 from bounds: Operation not permitted

[mariospr]

Spawning capsh --print from the bubblewrap process (right from drop_cap_bounding_set()) returns a similar output, but with the original user and group ids thrown away:

Current: =
Bounding set =cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend
Securebits: 00/0x0/1'b0
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
uid=65534(nobody)
gid=65534(nogroup)
groups=65534(nogroup),65534(nogroup),65534(nogroup),65534(nogroup),65534(nogroup)

[mariospr]

Some more info: this is fairly easy to reproduce outside of flatpak:

mario@endless:~/bubblewrap$ bwrap --ro-bind / / --unshare-user ls /
Dropping capability 0 from bounds: Operation not permitted

While this is what I get if I revert commit b35f84a:

mario@endless:~/bubblewrap$ bwrap --ro-bind / / --unshare-user ls /
bin  boot  dev	etc  flatpak  home  lib  lost+found  mnt  opt  ostree  proc  root  run	sbin  srv  sys	sysroot  tmp  usr  var

Note the --unshare-user. It seems to work fine without that

[mariospr]

Some more information before leaving this debug session for the day:

Consistently with what I found that passing the --unshare-user parameter breaks things on ARM, I've just confirmed that the call to drop_cap_bounding_set() works fine on ARM when it's the one called from acquire_privs(), and that it fails when it's the one called from `switch_to_user_with_privs(), while it works fine in both cases for amd64.

Sleeping for a while right before dropping the caps in the second case, and then inspecting the processes this is what I got on ARM:

$ ps aux | grep bwrap
mario     4799  0.0  0.0   2152   496 pts/2    S    19:23   0:00 bwrap --ro-bind / / --unshare-user ls /
root      4800  0.0  0.0   2152   120 pts/2    S    19:23   0:00 bwrap --ro-bind / / --unshare-user ls /

$ grep ^Cap /proc/4799/status 
CapInh:	0000000000000000
CapPrm:	0000000000000000
CapEff:	0000000000000000
CapBnd:	0000000000000000

mario@endless:~/bubblewrap$ grep ^Cap /proc/4800/status 
CapInh:	0000000000000000
CapPrm:	0000001fffffffff
CapEff:	0000001fffffffff
CapBnd:	0000001fffffffff

On Intel 64bit I get similar results, but the mask it's different (difference is that it also includes cap_kill, according to capsh --decode) AND includes an additional new "Ambient capability", CapAmb:

$ ps aux | grep bwrap
mario     1402  0.0  0.0  13320  1528 pts/0    S    19:22   0:00 bwrap --ro-bind / / --unshare-user ls /
root      1410  0.0  0.0  13320   156 pts/0    S    19:22   0:00 bwrap --ro-bind / / --unshare-user ls /

$ grep ^Cap /proc/1402/status 
CapInh:	0000000000000000
CapPrm:	0000000000000000
CapEff:	0000000000000000
CapBnd:	0000000000000000
CapAmb:	0000000000000000

$ grep ^Cap /proc/1410/status 
CapInh:	0000000000000000
CapPrm:	0000003fffffffff
CapEff:	0000003fffffffff
CapBnd:	0000003fffffffff
CapAmb:	0000000000000000

I'm not entirely sure whether all this information will be useful to solve the problem I'm seeing right not, but I have to leave now and I hope this will ring any bell. Otherwise, I'll continue tomorrow.

Thanks in advance for any help!

[cgwalters]

I looked at this briefly and left confused as to how capabilities work in userns; I'd expected to see a full current set from capsh.

I don't have a theory right now for why this would work on one architecture/kernel config and not another. Is bwrap full suid in both cases?

[mariospr]

Yes, it's full suid in both cases. I've definitely checked that (and made sure that would be the case after building from sources + installing)

[cgwalters]

It's safe to revert b35f84a temporarily while we debug this.

In #172 I'm making it optional.

[mariospr]

@cgwalters Thanks for the feedback. I checked your PR at #172 and I'm not sure that would fix the problem I'm seeing as bubblewrap would still be dropping the caps in the case I'm seeing unless --keep-caps was passed, but then it would die because the bwrap binary is a suid.

About reverting b35f84a temporarily, the problem I've observed also only seems to happen when dropping caps from switch_to_user_with_privs(), so I think we could still keep the caps dropping bits from acquire_privs(), no need to revert everything. Just placed a PR for that here: #175

[mariospr]

We finally figured out what the problem was with our ARM kernel. We need to cherry pick https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/security/commoncap.c?id=160da84dbb39443fdade7151bc63a88f8e953077

More details: #175 (comment)