bubblewrap Sandboxed Tor Browser fails to start in Qubes Debian based AppVM - firefox: Can't mount proc on /newroot/proc

[adrelanos]

Qubes OS version (e.g., R3.2):

R3.2 with Qubes testing repository

Affected TemplateVMs (e.g., fedora-23, if applicable):

debian, whonix-ws

---

Expected behavior:

sandboxed-tor-browser working.

Actual behavior:

sandboxed-tor-browser fails to start.

2016/12/25 05:35:20 firefox: Can't mount proc on /newroot/proc
2016/12/25 05:35:20 firefox: : Operation not permitted
2016/12/25 05:35:20 fatal error in the user interface: waitid: no child processes

Steps to reproduce the behavior:

1. Boot a Debian jessie TemplateVM.

2. Enable jessie-backports.

sudo su -c "echo -e 'deb http://http.debian.net/debian jessie-backports main' > /etc/apt/sources.list.d/jessie-backports.list"

3. Consider using apt pinning.

4. Install dependencies from jessie-backports.

sudo apt-get -t jessie-backports install golang bubblewrap libseccomp2 libseccomp-dev

5. Shut down the TemplateVM and start a Debian based AppVM.

6. Download and unzip Sandboxed Tor Browser.

Download links can be found here (scroll down to Tor Browser Sandbox):
https://www.torproject.org/projects/torbrowser.html.en#downloads-alpha

For your convenience:

• wget https://www.torproject.org/dist/torbrowser/6.5a6/sandbox-0.0.2-linux64.zip
• wget https://www.torproject.org/dist/torbrowser/6.5a6/sandbox-0.0.2-linux64.zip.asc
• gpg --recv-keys "EF6E 286D DA85 EA2A 4BA7 DE68 4E2C 6E87 9329 8290"
• gpg --verify sandbox-0.0.2-linux64.zip.asc
• unzip sandbox-0.0.2-linux64.zip

6. start

./sandboxed-tor-browser

choose hardened Tor Browser.

General notes:

Sandboxed Tor Browser works for me with the same instructions inside a Non-Qubes Debian jessie VirtualBox VM.

Debug output:

Full output of ./sandboxed-tor-browser -debug can be found here:
https://trac.torproject.org/projects/tor/ticket/2107

---

Related issues:

• https://phabricator.whonix.org/T586
• https://forums.whonix.org/t/tor-browser-sandbox-linux-alpha-coming-soon
• https://trac.torproject.org/projects/tor/ticket/21077

[marmarek]

Do you have grsec kernel and/or apparmor enabled?

…

-- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing?

[marmarek]

Do you have grsec kernel and/or apparmor enabled?

…

-- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing?

[adrelanos]

Apparmor, yes.

[adrelanos]

Apparmor, yes.

[marmarek]

On Sun, Dec 25, 2016 at 01:31:31PM -0800, Patrick Schleizer wrote: Apparmor, yes.

Are you sure it isn't responsible for this denial? I have very little experience with apparmor...

…

-- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing?

[marmarek]

On Sun, Dec 25, 2016 at 01:31:31PM -0800, Patrick Schleizer wrote: Apparmor, yes.

Are you sure it isn't responsible for this denial? I have very little experience with apparmor...

…

-- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing?

[adrelanos]

Just now finished testing with apparmor removed kernel command line. (Verified using sudo aa-status.) Didn't fix this.

---

• grsecurity not installed.
• https://trac.torproject.org/projects/tor/wiki/doc/TorBrowser/Sandbox/Linux#Tickets
• Broken things that are unlikely to ever be supported: - "Using the "hardened" series with a grsec kernel (ASAN/PaX conflict, not limited to the sandbox). "
• Official 'sandboxed-tor-browser' binaries break with PaX MPROTECT. - https://trac.torproject.org/projects/tor/ticket/20976

[adrelanos]

Just now finished testing with apparmor removed kernel command line. (Verified using sudo aa-status.) Didn't fix this.

---

• grsecurity not installed.
• https://trac.torproject.org/projects/tor/wiki/doc/TorBrowser/Sandbox/Linux#Tickets
• Broken things that are unlikely to ever be supported: - "Using the "hardened" series with a grsec kernel (ASAN/PaX conflict, not limited to the sandbox). "
• Official 'sandboxed-tor-browser' binaries break with PaX MPROTECT. - https://trac.torproject.org/projects/tor/ticket/20976

[adrelanos]

Also tested this also in a Qubes Debian stretch VM (upgrades a jessie template to stretch). Didn't fix this issue.

Also thought perhaps the kernel provided by dom0 might be incompatible with bubblewrap. To rule that out I switched that Qubes Debian stretch VM to Debian stretch linux-image-amd64 using Qubes VM kernel / pvgrub. Didn't fix this issue either.

Also asked the bubblewrap maintainers for any insights:
projectatomic/bubblewrap#134

[adrelanos]

Also tested this also in a Qubes Debian stretch VM (upgrades a jessie template to stretch). Didn't fix this issue.

Also thought perhaps the kernel provided by dom0 might be incompatible with bubblewrap. To rule that out I switched that Qubes Debian stretch VM to Debian stretch linux-image-amd64 using Qubes VM kernel / pvgrub. Didn't fix this issue either.

Also asked the bubblewrap maintainers for any insights:
projectatomic/bubblewrap#134

[adrelanos]

Quote @marmarek projectatomic/bubblewrap#134 (comment):

Indeed after unmounting /proc/xen it does work. I wonder if anything still use /proc/xen in Qubes... AFAIR it's legacy location and the new one is /dev/xen. There were more problems with /proc/xen (where "normal files" behaves like character devices...). The fact that I could unmount it without killing anything suggests it isn't used anymore :)

That is great news!

/dev/xen is hardcoded a few times in multiple Qubes packages. Would solving this ticket now mean to change any mention of /dev/xen to /proc/xen? If so, should I create the pull requests to change this?

[adrelanos]

Quote @marmarek projectatomic/bubblewrap#134 (comment):

Indeed after unmounting /proc/xen it does work. I wonder if anything still use /proc/xen in Qubes... AFAIR it's legacy location and the new one is /dev/xen. There were more problems with /proc/xen (where "normal files" behaves like character devices...). The fact that I could unmount it without killing anything suggests it isn't used anymore :)

That is great news!

/dev/xen is hardcoded a few times in multiple Qubes packages. Would solving this ticket now mean to change any mention of /dev/xen to /proc/xen? If so, should I create the pull requests to change this?

[marmarek]

Would solving this ticket now mean to change any mention of /dev/xen to /proc/xen?

The other way around.
The great thing about /dev/xen is that it don't require to be specifically mounted (but still you require some modules being loaded) - so one potential race condition less.

[marmarek]

Would solving this ticket now mean to change any mention of /dev/xen to /proc/xen?

The other way around.
The great thing about /dev/xen is that it don't require to be specifically mounted (but still you require some modules being loaded) - so one potential race condition less.

[adrelanos]

Here is the first PR.

marmarek/qubes-builder-debian#27

Does it look alright? If so, I would be going forward with the other components.

[adrelanos]

Here is the first PR.

marmarek/qubes-builder-debian#27

Does it look alright? If so, I would be going forward with the other components.

[adrelanos]

Actually, not as simple as a search, replace and pull request action as I thought. ( marmarek/qubes-builder-debian#27 / marmarek/qubes-builder-debian#28 )

[adrelanos]

Actually, not as simple as a search, replace and pull request action as I thought. ( marmarek/qubes-builder-debian#27 / marmarek/qubes-builder-debian#28 )

[marmarek]

Just tested - mostly works. The only problem I've found is missing /proc/xen/capabilities, which don't have corresponding file in /dev/xen. It's used here for checking if running in dom0 or VM. And probably few more places for the same purpose.

I'm looking at /sys/hypervisor/properties/features, where 0x800 bit is missing in VM (compared to the same in dom0). In theory it's XENFEAT_dom0, but it's described as "operation as Dom0 is supported" - so, not necessary running as dom0 (at least this is my understanding). Looking at the code setting this flag, it's totally not obvious if this is the same.

[marmarek]

Just tested - mostly works. The only problem I've found is missing /proc/xen/capabilities, which don't have corresponding file in /dev/xen. It's used here for checking if running in dom0 or VM. And probably few more places for the same purpose.

I'm looking at /sys/hypervisor/properties/features, where 0x800 bit is missing in VM (compared to the same in dom0). In theory it's XENFEAT_dom0, but it's described as "operation as Dom0 is supported" - so, not necessary running as dom0 (at least this is my understanding). Looking at the code setting this flag, it's totally not obvious if this is the same.

[adrelanos]

Maybe some dom0 package should ship (or already has a suitable) status file that only exists in dom0? And then use that file to detect if running inside dom0 or VM?

[adrelanos]

Maybe some dom0 package should ship (or already has a suitable) status file that only exists in dom0? And then use that file to detect if running inside dom0 or VM?

[alex-mazzariol]

This issue affects a lot of flatpak-distributed software (i.e. MonoDevelop), not only sandboxed TOR browser. It seems that flatpak internally uses bubblewrap too.

[alex-mazzariol]

This issue affects a lot of flatpak-distributed software (i.e. MonoDevelop), not only sandboxed TOR browser. It seems that flatpak internally uses bubblewrap too.