Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VirtualBox: Virtual machines fail to start. #65564

Closed
ambrop72 opened this issue Jul 29, 2019 · 21 comments
Closed

VirtualBox: Virtual machines fail to start. #65564

ambrop72 opened this issue Jul 29, 2019 · 21 comments
Labels
0.kind: bug Something is broken 1.severity: blocker This is preventing another PR or issue from being completed
Milestone

Comments

@ambrop72
Copy link
Contributor

Describe the bug

On the latest nixos-unstable (b5f5c97), starting any virtual machine gives me error:

The virtual machine 'Fedora' has terminated unexpectedly during startup because of signal 6.
Result Code: | NS_ERROR_FAILURE (0x80004005)
Component: | MachineWrap
Interface: | IMachine {5047460a-265d-4538-b23e-ddba5fb84976}

To Reproduce
Steps to reproduce the behavior:

  1. Use VirtualBox on nixos-unstable
  2. Start a virtual machine

Screenshots
Screenshot_2019-07-29_20-11-34

Additional context
Seems to have been caused by updating.

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute: linuxPackages.virtualbox
# a list of nixos modules affected by the problem
module: virtualbox
@ambrop72 ambrop72 added the 0.kind: bug Something is broken label Jul 29, 2019
@oxalica
Copy link
Contributor

oxalica commented Jul 30, 2019

I've met the same issue with "signal 6". Rebooting does not help.

But I can run VM with --type=headless successfully. Seems there is something wrong with display.

@flokli
Copy link
Contributor

flokli commented Aug 10, 2019

I digged a bit.

Virtualbox seems to have problems when loading VBoxSharedCrOpenGL:

#0  0x00007fbbd957ba0f in crHashtableWalkUnlocked (hash=0x0, walkFunc=walkFunc@entry=0x7fbbd954c3a2 <renderspuVBoxCompositorClearAllCB>, dataPtr2=dataPtr2@entry=0x0) at /build/VirtualBox-6.0.8/src/VBox/GuestHost/OpenGL/util/hash.c:497
#1  0x00007fbbd954c526 in renderspuVBoxCompositorClearAll () at /build/VirtualBox-6.0.8/src/VBox/HostServices/SharedOpenGL/render/renderspu.c:1139
#2  0x00007fbbd954dd6b in renderspuCleanupBase (fDeleteTables=fDeleteTables@entry=true) at /build/VirtualBox-6.0.8/src/VBox/HostServices/SharedOpenGL/render/renderspu_init.c:491
#3  0x00007fbbd954dee3 in renderSPUCleanup () at /build/VirtualBox-6.0.8/src/VBox/HostServices/SharedOpenGL/render/renderspu_init.c:536
#4  0x00007fbbd967c4d8 in crSPUUnloadChain (headSPU=headSPU@entry=0x7fbbb4130180) at /build/VirtualBox-6.0.8/src/VBox/GuestHost/OpenGL/spu_loader/spuload.c:283
#5  0x00007fbbd967c744 in crSPULoad (child=child@entry=0x0, id=0, name=0x7fbbd968eb8f "render", dir=dir@entry=0x0, server=server@entry=0x7fbbd96cc680 <cr_server>) at /build/VirtualBox-6.0.8/src/VBox/GuestHost/OpenGL/spu_loader/spuload.c:166
#6  0x00007fbbd967c806 in crSPULoadChain (count=count@entry=1, ids=ids@entry=0x7fbbd975718c, names=names@entry=0x7fbbd9757180, dir=dir@entry=0x0, server=server@entry=0x7fbbd96cc680 <cr_server>) at /build/VirtualBox-6.0.8/src/VBox/GuestHost/OpenGL/spu_loader/spuload.c:201
#7  0x00007fbbd95e3fbf in crServerSetVBoxConfigurationHGCM () at /build/VirtualBox-6.0.8/src/VBox/HostServices/SharedOpenGL/crserverlib/server_config.c:327
#8  0x00007fbbd95cc904 in crVBoxServerInit () at /build/VirtualBox-6.0.8/src/VBox/HostServices/SharedOpenGL/crserverlib/server_main.c:582
#9  0x00007fbbd95bbb66 in VBoxHGCMSvcLoad (ptable=<optimized out>) at /build/VirtualBox-6.0.8/src/VBox/HostServices/SharedOpenGL/crserver/crservice.cpp:1562
#10 0x00007fbbdbe2ea7b in HGCMService::loadServiceDLL (this=this@entry=0x7fbbb80037f0) at /build/VirtualBox-6.0.8/src/VBox/Main/src-client/HGCM.cpp:348
#11 0x00007fbbdbe2ecb4 in hgcmServiceThread (pThread=0x7fbbb8003950, pvUser=0x7fbbb80037f0) at /build/VirtualBox-6.0.8/src/VBox/Main/src-client/HGCM.cpp:637
#12 0x00007fbbdbe2ceaf in hgcmWorkerThreadFunc (hThreadSelf=<optimized out>, pvUser=0x7fbbb8003950) at /build/VirtualBox-6.0.8/src/VBox/Main/src-client/HGCMThread.cpp:200
#13 0x00007fbbea3e0146 in rtThreadMain (pThread=pThread@entry=0x7fbbb8003b80, NativeThread=NativeThread@entry=140444783970048, pszThreadName=pszThreadName@entry=0x7fbbb8004460 "ShCrOpenGL") at /build/VirtualBox-6.0.8/src/VBox/Runtime/common/misc/thread.cpp:719
#14 0x00007fbbea48a619 in rtThreadNativeMain (pvArgs=0x7fbbb8003b80) at /build/VirtualBox-6.0.8/src/VBox/Runtime/r3/posix/thread-posix.cpp:327
#15 0x00007fbbea763ef7 in start_thread () from /nix/store/n2wvg362fsilqh3z3262argzxk681qpz-glibc-multi-2.27/lib/libpthread.so.0
#16 0x00007fbbea69422f in clone () from /nix/store/n2wvg362fsilqh3z3262argzxk681qpz-glibc-multi-2.27/lib/libc.so.6

I guessed it might be opengl not finding mesa or some x11 libraries, so I tried adding addOpenGLRunpath $out/libexec/virtualbox/VBoxSharedCrOpenGL.so as described in #60985 but seems this wasn't enough, and I'm not sure if that's the cause of the bug at all.

There's more libraries that seem to run dlopen, which might need patching too…

I'll try to bisect what broke it, maybe this will help tracking down what caused it.

@flokli
Copy link
Contributor

flokli commented Aug 12, 2019

I finished bisecting, and found the following commit causing it to break:

f79fd2e826dd95b3b64839d3e0bec8ae1dfab17e is the first bad commit
commit f79fd2e826dd95b3b64839d3e0bec8ae1dfab17e
Author: Thomas Tuegel <ttuegel@mailbox.org>
Date:   Fri Jul 5 10:41:41 2019 -0500

    wrapQtAppsHook: wrap Qt applications for runtime dependencies

:040000 040000 a23c095c35d4c774f7ee64d5262888ead078a784 097d2fc2a69cf74da570a0371fd2cfb647ed60fc M nixos
:040000 040000 dca1fe8cd6b2e856fe9ab6136f0814ecb2d78316 5e6859cc779955cf034a32979bca4f314e9b0970 M pkgs
bisect run success

[root@flokli:~/nixpkgs]# git bisect log
git bisect start
# bad: [984851a9bfa3a7b5dacb436d7686f2f09b5e2e85] Merge pull request #66509 from emilazy/add-git-revise
git bisect bad 984851a9bfa3a7b5dacb436d7686f2f09b5e2e85
# good: [72216e12de053dc488707fb070e56047b5c91020] Merge #62393: virtualboxHeadless: Fix build
git bisect good 72216e12de053dc488707fb070e56047b5c91020
# good: [d0453ef9ce9f14e6f54a1a19ef3101d64d2f4240] bcachefs: 2019-07-13 (#64732)
git bisect good d0453ef9ce9f14e6f54a1a19ef3101d64d2f4240
# bad: [0a1bf4734fb93ebc7d2240a9d98babdacfcf6100] libtensorflow: add binary build and add automatic generation
git bisect bad 0a1bf4734fb93ebc7d2240a9d98babdacfcf6100
# bad: [8e04c3cbf10a206ad6a3a01d70725a281924c1f3] lolcat: 99.9.69 -> 99.9.99
git bisect bad 8e04c3cbf10a206ad6a3a01d70725a281924c1f3
# bad: [d7cdd895fa65334dc063ae6cd2e01d48e962bc3b] rsyslog: 8.1905.0 -> 8.1907.0
git bisect bad d7cdd895fa65334dc063ae6cd2e01d48e962bc3b
# bad: [325d2e2352c621c227edcbf6217eb5935261cc0f] Merge pull request #64912 from r-ryantm/auto-update/python3.7-texttable
git bisect bad 325d2e2352c621c227edcbf6217eb5935261cc0f
# good: [713a45ecf79a4b4c632819f1c898d3e66c77bdd2] python37Packages.Wand: 0.5.4 -> 0.5.5 (#64914)
git bisect good 713a45ecf79a4b4c632819f1c898d3e66c77bdd2
# good: [066491c2e18d6277e0765a3647068c93650fcd19] Merge pull request #63999 from dingxiangfei2009/appamor-cross-compile
git bisect good 066491c2e18d6277e0765a3647068c93650fcd19
# bad: [5bf68e1354d2a6ad3663c829675ddf90b6996e24] Merge #64742: firefox 67 -> 68, and related updates
git bisect bad 5bf68e1354d2a6ad3663c829675ddf90b6996e24
# bad: [6049c39bdb3d8bfc01e2a86d2fc6ef6e46f9b0be] keybase: fix eval after merge mistake
git bisect bad 6049c39bdb3d8bfc01e2a86d2fc6ef6e46f9b0be
# bad: [a88e319591f072afd109239237e5f3927654dc74] python36: 3.6.8 -> 3.6.9
git bisect bad a88e319591f072afd109239237e5f3927654dc74
# bad: [3adc9d04870605ca20969a98fe820535fc7a88a5] doc/languages-frameworks/qt.xml: Update for wrapQtAppsHook
git bisect bad 3adc9d04870605ca20969a98fe820535fc7a88a5
# bad: [51d78034a1db78f8ce0ea3ba6fa20be590ed0ca2] wrapQtAppsHook: Remove ad hoc Qt wrappers
git bisect bad 51d78034a1db78f8ce0ea3ba6fa20be590ed0ca2
# bad: [f79fd2e826dd95b3b64839d3e0bec8ae1dfab17e] wrapQtAppsHook: wrap Qt applications for runtime dependencies
git bisect bad f79fd2e826dd95b3b64839d3e0bec8ae1dfab17e
# first bad commit: [f79fd2e826dd95b3b64839d3e0bec8ae1dfab17e] wrapQtAppsHook: wrap Qt applications for runtime dependencies

I made sure to only mark commits bad that were really failing the virtulabox test with the specific error message posted above by using the following script for git bisect run:

#!/usr/bin/env bash
nixBuild() {
  DRV=$(nix-instantiate "$@");
  nix-build $DRV --builders "" --no-out-link;
  S=$?;
  if [ $S -ne 0 ]; then
    # trigger a git bisect bad if the bad line was found
    nix log $DRV | grep -i "has terminated unexpectedly during startup" && exit 42;
    # else trigger a skip (some other random build failure)
    exit 125;
  fi;
  # build was successful
  exit 0
}
nixBuild nixos/tests/virtualbox.nix -A simple-cli

So it's indeed some problems in how qt applications are wrapped cc @ttuegel / #65399

@flokli
Copy link
Contributor

flokli commented Aug 12, 2019

Wrapping simply all qt binaries via wrapQtAppsHook won't work, as there's some suid wrappers being used.

@worldofpeace
Copy link
Contributor

worldofpeace commented Aug 13, 2019

@flokli I did 2bd649b recently.
Would that change the current state of things, or did you test with this commit?

@flokli
Copy link
Contributor

flokli commented Aug 13, 2019 via email

@samueldr
Copy link
Member

Had a look, see what the issue could be. I think the issue is that VirtualBoxVM (maybe among others) in libexec would need to be wrapped... Except that wrapping them breaks them.

Here's why I think it's related:

@santaslittlehelper ~ $ VirtualBoxVM --startvm "Tiny Core Linux"
Qt WARNING: Could not find the Qt platform plugin "xcb" in ""
Qt FATAL: This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Aborted

diff --git a/pkgs/applications/virtualization/virtualbox/default.nix b/pkgs/applications/virtualization/virtualbox/default.nix
index 1a6ba5ac527..836c5e5b65a 100644
--- a/pkgs/applications/virtualization/virtualbox/default.nix
+++ b/pkgs/applications/virtualization/virtualbox/default.nix
@@ -183,6 +183,7 @@ in stdenv.mkDerivation {

   preFixup = optionalString (!headless) ''
     wrapQtApp $out/bin/VirtualBox
+    wrapQtApp $out/libexec/virtualbox/VirtualBoxVM
   '';

   passthru = {

Once wrapped, the suid wrapper seems to not work as expected

@santaslittlehelper ~ $ VirtualBoxVM --startvm "Tiny Core Linux"
VirtualBoxVM: Error -10 in SUPR3HardenedMain!
VirtualBoxVM: Effective UID is not root (euid=1000 egid=100 uid=1000 gid=100)

VirtualBoxVM: Tip! It may help to reinstall VirtualBox.

And, just as a test, tried to run the wrapped binary through sudo:

@santaslittlehelper ~ 1 $ sudo /nix/store/84kwxvr5wv7ikfb53ywynwvl435jvic5-virtualbox-6.0.8/libexec/virtualbox/VirtualBoxVM
[sudo] password for samuel:
VirtualBoxVM: supR3HardenedVerifySameFile: "/nix/store/84kwxvr5wv7ikfb53ywynwvl435jvic5-virtualbox-6.0.8/libexec/virtualbox/.VirtualBoxVM-wrapped" isn't the same as "/nix/store/84kwxvr5wv7ikfb53ywynwvl435jvic5-virtualbox-6.0.8/libexec/virtualbox/VirtualBoxVM"

So, there's a bit more to it than simply wrapping it.

@ambrop72
Copy link
Contributor Author

ambrop72 commented Aug 14, 2019

Does simply not wrapping it make it work? If not, why did it work before?

@ambrop72
Copy link
Contributor Author

Anyway, I think that we need to make sure that VirtualBoxVM is not wrapped, and hack it however we need to make it run, e.g. by inserting calls to QCoreApplication::addLibraryPath to main for all plugins that are needed.

@worldofpeace
Copy link
Contributor

Last part where you ran it with sudo appears to trigger a check in a hardening function which obviously finds our shell wrapper suspicious.

@ambrop72 It does appear that in this case a patch may also be needed.

@samueldr
Copy link
Member

Does simply not wrapping it make it work? If not, why did it work before?

There recently were changes in how Qt applications are packaged. The change (or an equivalent change) is required to fix an issue with mixed Qt applications in a user's environment. The issue this fixes is way worse (and hard to debug) than the issues it causes in the end.

Why did it work before? It worked because Qt would discover libraries (e.g. .../lib/qt-.../plugins/platforms/libqxcb.so) by peeking at any paths in the PATH, which could in theory bring the wrong versions in, e.g. versions from another incompatible Qt. The fixes makes all checks more strict, which breaks what seemed to work flawlessly beforehand. Most packages don't end up sanitizing the environment for Qt at runtime, though it looks like this is what Qt does (and rightly so due to hardening).


Anyway, I think that we need to make sure that VirtualBoxVM is not wrapped, and hack it however we need to make it run, e.g. by inserting calls to QCoreApplication::addLibraryPath to main for all plugins that are needed.

It would work, though the patch would need to be dynamic, as the full library paths would need to be used.

Adding to Qt's library path is what #44047 was doing to solve the issue. (Though it was doing so at initialization-time, by looking into a "registry" file of paths to add for the derivation.)

@ambrop72
Copy link
Contributor Author

ambrop72 commented Aug 14, 2019

Testing this might be easier when setting virtualisation.virtualbox.host.enableHardening = false, (e.g. you can gdb it and it should respect QT_PLUGIN_PATH for testing which plugins are needed).

@flokli
Copy link
Contributor

flokli commented Aug 16, 2019

In case this helps somebody:

I added #66725. After using wrapQtAppsHook to wrap every executable (and only those, not the .so files present in $out/libexec), I tried to run the virtualbox tests again (with hardening disabled), see https://github.com/flokli/nixpkgs/commits/wrapqtappshook-virtualbox .

The tests seem to not crash immediately anymore, but the virtual machines still didn't boot up. I did not yet start a machine interactively to investigate.

@matthewbauer matthewbauer added this to the 19.09 milestone Aug 16, 2019
@matthewbauer matthewbauer added the 1.severity: blocker This is preventing another PR or issue from being completed label Aug 16, 2019
@jcumming
Copy link
Contributor

In case this helps somebody:

I added #66725. After using wrapQtAppsHook to wrap every executable (and only those, not the .so files present in $out/libexec), I tried to run the virtualbox tests again (with hardening disabled), see https://github.com/flokli/nixpkgs/commits/wrapqtappshook-virtualbox .

The tests seem to not crash immediately anymore, but the virtual machines still didn't boot up. I did not yet start a machine interactively to investigate.

This works for me. Thanks!

@flokli
Copy link
Contributor

flokli commented Aug 27, 2019

@jcumming can you elaborate?

You tried to manually run Virtualbox and start a VM through the GUI? Did you enable hardening?

@flokli
Copy link
Contributor

flokli commented Aug 31, 2019

ping @jcumming

@B4dM4n
Copy link
Contributor

B4dM4n commented Aug 31, 2019

I got the Virtualbox VM Gui running by wrapping the VirtualBoxVM binary and disabling hardening.

--- a/pkgs/applications/virtualization/virtualbox/default.nix
+++ b/pkgs/applications/virtualization/virtualbox/default.nix
@@ -183,6 +183,7 @@ in stdenv.mkDerivation {

   preFixup = optionalString (!headless) ''
     wrapQtApp $out/bin/VirtualBox
+    wrapQtApp $out/libexec/virtualbox/VirtualBoxVM
   '';

   passthru = {

Enabling hardening doesn't work because the wrapper doesn't carry over the correct permissions and probably changes the expected binary path:

$ /run/wrappers/bin/VirtualBoxVM --startvm 51253cab-5d4a-4911-9296-8781ad765ee2
VirtualBoxVM: Error -10 in SUPR3HardenedMain!
VirtualBoxVM: Effective UID is not root (euid=1000 egid=1000 uid=1000 gid=1000)

@flokli
Copy link
Contributor

flokli commented Sep 1, 2019

@B4dM4n Yeah, I also got the qt app to run. But were you able to actually start a VM, too?

@B4dM4n
Copy link
Contributor

B4dM4n commented Sep 1, 2019

Yes, without hardening the VM GUI starts fine.

I also noticed that wrapping VirtualBoxVM doesn't make a difference without hardening, because the VirtualBox binary seems to start the VM's without invoking VirtualBoxVM itself.

@ambrop72
Copy link
Contributor Author

ambrop72 commented Sep 2, 2019

I've opened pull request #67968 which fixes this.

@flokli flokli closed this as completed in b52dfd3 Sep 4, 2019
flokli added a commit that referenced this issue Sep 4, 2019
dtzWill pushed a commit to dtzWill/nixpkgs that referenced this issue Mar 14, 2020
Fixes NixOS#65564. When hardening is enabled, we cannot use a wrapper for
VirtualBoxVM, so patch the source code to set QT_PLUGIN_PATH as required.

(cherry picked from commit b52dfd3)
@nixos-discourse
Copy link

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/running-an-ova-in-virtualbox/11340/1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0.kind: bug Something is broken 1.severity: blocker This is preventing another PR or issue from being completed
Projects
None yet
Development

No branches or pull requests

9 participants