Problem starting an AppVM

[marmarek]

Reported by joanna on 1 Aug 2011 19:05 UTC

[~](joanna@dom0)$ qvm-start work-pub
--> Cleaning volatile image: /var/lib/qubes/appvms/work-pub/volatile.img...
--> Loading the VM (type = AppVM)...
libxl: error: libxl_create.c:317:libxl__domain_make domain creation fail
cannot make domain: -3
libxl: error: libxl.c:711:libxl_domain_destroy non-existant domain -1
ERROR: Failed to load VM config

The xl-work-pub.log says:

Waiting for domain work-pub (domid 10) to die [11924](pid)                                                                       
Domain 10 is dead                                                                                                               
Action for shutdown reason code 0 is destroy
Domain 10 needs to be cleaned up: destroying the domain
Done. Exiting now            

This repeated a few time when I tried to do qvm-start for this domain. Then I started some other AppVM, which started fine, and then I could start this work-pub domain no problem.

Migrated-From: https://wiki.qubes-os.org/ticket/318

[marmarek]

Comment by joanna on 2 Aug 2011 10:29 UTC
Another example of an error when starting an AppVM:

[~](joanna@dom0)$ qvm-start red
--> Cleaning volatile image: /var/lib/qubes/appvms/red/volatile.img...
--> Loading the VM (type = AppVM)...
xc: error: panic: xc_dom_boot.c:79: launch_vm: SETVCPUCONTEXT failed (rc=-1): Internal error
libxl: error: libxl_dom.c:212:libxl__build_pv xc_dom_boot_image failed: Invalid argument
cannot (re-)build domain: -3
ERROR: Failed to load VM config

Next time I tried it it started up just fine...

[marmarek]

Comment by joanna on 2 Aug 2011 10:29 UTC
Another example of an error when starting an AppVM:

[~](joanna@dom0)$ qvm-start red
--> Cleaning volatile image: /var/lib/qubes/appvms/red/volatile.img...
--> Loading the VM (type = AppVM)...
xc: error: panic: xc_dom_boot.c:79: launch_vm: SETVCPUCONTEXT failed (rc=-1): Internal error
libxl: error: libxl_dom.c:212:libxl__build_pv xc_dom_boot_image failed: Invalid argument
cannot (re-)build domain: -3
ERROR: Failed to load VM config

Next time I tried it it started up just fine...

[marmarek]

Comment by rafal on 2 Aug 2011 13:57 UTC

libxl: error: libxl_create.c:317:libxl__domain_make domain creation fail
libxl is broken and does not even report xc_domain_create error code (because it is negative). Do we want a patch to improve logging ?

xc: error: panic: xc_dom_boot.c:79: launch_vm: SETVCPUCONTEXT failed (rc=-1): Internal error
Now we have good logging - unfortunately rc==-1 means EPERM. How EPERM can be intermittent head ? Nothing in the xc_domctl code path looks like it.

[marmarek]

Comment by rafal on 2 Aug 2011 13:57 UTC

libxl: error: libxl_create.c:317:libxl__domain_make domain creation fail
libxl is broken and does not even report xc_domain_create error code (because it is negative). Do we want a patch to improve logging ?

xc: error: panic: xc_dom_boot.c:79: launch_vm: SETVCPUCONTEXT failed (rc=-1): Internal error
Now we have good logging - unfortunately rc==-1 means EPERM. How EPERM can be intermittent head ? Nothing in the xc_domctl code path looks like it.

[marmarek]

Comment by joanna on 15 Aug 2011 09:33 UTC
The above problem is repeatable. Here I tried to start the 'work' VM first, when failed tried to start work-pub, failed again, but finally was able to start both of them -- auto-magically the problem was solved...):

[~](joanna@dom0)$ qvm-start work
--> Cleaning volatile image: /var/lib/qubes/appvms/work/volatile.img...
--> Loading the VM (type = AppVM)...
xc: error: panic: xc_dom_boot.c:79: launch_vm: SETVCPUCONTEXT failed (rc=-1): Internal error
libxl: error: libxl_dom.c:212:libxl__build_pv xc_dom_boot_image failed: Invalid argument
cannot (re-)build domain: -3
ERROR: Failed to load VM config
[~](joanna@dom0)$ qvm-start work-pub
--> Cleaning volatile image: /var/lib/qubes/appvms/work-pub/volatile.img...
--> Loading the VM (type = AppVM)...
xc: error: panic: xc_dom_boot.c:79: launch_vm: SETVCPUCONTEXT failed (rc=-1): Internal error
libxl: error: libxl_dom.c:212:libxl__build_pv xc_dom_boot_image failed: Invalid argument
cannot (re-)build domain: -3
ERROR: Failed to load VM config
[~](joanna@dom0)$ qvm-start work
--> Cleaning volatile image: /var/lib/qubes/appvms/work/volatile.img...
--> Loading the VM (type = AppVM)...
xc: error: panic: xc_dom_boot.c:79: launch_vm: SETVCPUCONTEXT failed (rc=-1): Internal error
libxl: error: libxl_dom.c:212:libxl__build_pv xc_dom_boot_image failed: Invalid argument
cannot (re-)build domain: -3
ERROR: Failed to load VM config
[~](joanna@dom0)$ qvm-start work-pub
--> Cleaning volatile image: /var/lib/qubes/appvms/work-pub/volatile.img...
--> Loading the VM (type = AppVM)...
Daemon running with PID 4030
--> Setting Xen Store info for the VM...
--> Updating firewall rules...
--> Starting the VM...
--> Starting the qrexec daemon...
Waiting for VM's qrexec agent..........connected
--> Starting Qubes GUId...
Connecting to VM's GUI agent: .connected

[marmarek]

Comment by joanna on 15 Aug 2011 09:33 UTC
The above problem is repeatable. Here I tried to start the 'work' VM first, when failed tried to start work-pub, failed again, but finally was able to start both of them -- auto-magically the problem was solved...):

[~](joanna@dom0)$ qvm-start work
--> Cleaning volatile image: /var/lib/qubes/appvms/work/volatile.img...
--> Loading the VM (type = AppVM)...
xc: error: panic: xc_dom_boot.c:79: launch_vm: SETVCPUCONTEXT failed (rc=-1): Internal error
libxl: error: libxl_dom.c:212:libxl__build_pv xc_dom_boot_image failed: Invalid argument
cannot (re-)build domain: -3
ERROR: Failed to load VM config
[~](joanna@dom0)$ qvm-start work-pub
--> Cleaning volatile image: /var/lib/qubes/appvms/work-pub/volatile.img...
--> Loading the VM (type = AppVM)...
xc: error: panic: xc_dom_boot.c:79: launch_vm: SETVCPUCONTEXT failed (rc=-1): Internal error
libxl: error: libxl_dom.c:212:libxl__build_pv xc_dom_boot_image failed: Invalid argument
cannot (re-)build domain: -3
ERROR: Failed to load VM config
[~](joanna@dom0)$ qvm-start work
--> Cleaning volatile image: /var/lib/qubes/appvms/work/volatile.img...
--> Loading the VM (type = AppVM)...
xc: error: panic: xc_dom_boot.c:79: launch_vm: SETVCPUCONTEXT failed (rc=-1): Internal error
libxl: error: libxl_dom.c:212:libxl__build_pv xc_dom_boot_image failed: Invalid argument
cannot (re-)build domain: -3
ERROR: Failed to load VM config
[~](joanna@dom0)$ qvm-start work-pub
--> Cleaning volatile image: /var/lib/qubes/appvms/work-pub/volatile.img...
--> Loading the VM (type = AppVM)...
Daemon running with PID 4030
--> Setting Xen Store info for the VM...
--> Updating firewall rules...
--> Starting the VM...
--> Starting the qrexec daemon...
Waiting for VM's qrexec agent..........connected
--> Starting Qubes GUId...
Connecting to VM's GUI agent: .connected

[marmarek]

Comment by rafal on 16 Aug 2011 08:20 UTC
Just after the "SETVCPUCONTEXT failed", what is the amount of xen free memory (as reported by xl info)?

[marmarek]

Comment by rafal on 16 Aug 2011 08:20 UTC
Just after the "SETVCPUCONTEXT failed", what is the amount of xen free memory (as reported by xl info)?

[marmarek]

Comment by joanna on 3 Sep 2011 09:38 UTC

[~](joanna@dom0)$ qvm-start work
--> Cleaning volatile image: /var/lib/qubes/appvms/work/volatile.img...
--> Loading the VM (type = AppVM)...
xc: error: panic: xc_dom_boot.c:79: launch_vm: SETVCPUCONTEXT failed (rc=-1): Internal error
libxl: error: libxl_dom.c:212:libxl__build_pv xc_dom_boot_image failed: Invalid argument
cannot (re-)build domain: -3
ERROR: Failed to load VM config
[~](joanna@dom0)$ xen info
bash: xen: command not found
[~](joanna@dom0)$ xl info
host                   : dom0
release                : 2.6.38.3-7.xenlinux.qubes.x86_64
version                : #1 SMP Sun Jul 24 18:53:41 CEST 2011
machine                : x86_64
nr_cpus                : 4
nr_nodes               : 1
cores_per_socket       : 2
threads_per_core       : 2
cpu_mhz                : 2393
hw_caps                : bfebfbff:28100800:00000000:00003f40:0298e3ff:00000000:00000001:00000000
virt_caps              : hvm hvm_directio
total_memory           : 5941
free_memory            : 55
free_cpus              : 0
xen_major              : 4
xen_minor              : 1
xen_extra              : .0
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
xen_commandline        : /xen.gz
cc_compiler            : gcc version 4.4.4 20100503 (Red Hat 4.4.4-2) (GCC) 
cc_compile_by          : joanna
cc_compile_domain      : 
cc_compile_date        : Tue Aug  2 14:57:19 CEST 2011
xend_config_format     : 4
[~](joanna@dom0)$ free
             total       used       free     shared    buffers     cached
Mem:       3303368    3059740     243628          0      18408    2481080
-/+ buffers/cache:     560252    2743116
Swap:      3080188          0    3080188

[marmarek]

Comment by joanna on 3 Sep 2011 09:38 UTC

[~](joanna@dom0)$ qvm-start work
--> Cleaning volatile image: /var/lib/qubes/appvms/work/volatile.img...
--> Loading the VM (type = AppVM)...
xc: error: panic: xc_dom_boot.c:79: launch_vm: SETVCPUCONTEXT failed (rc=-1): Internal error
libxl: error: libxl_dom.c:212:libxl__build_pv xc_dom_boot_image failed: Invalid argument
cannot (re-)build domain: -3
ERROR: Failed to load VM config
[~](joanna@dom0)$ xen info
bash: xen: command not found
[~](joanna@dom0)$ xl info
host                   : dom0
release                : 2.6.38.3-7.xenlinux.qubes.x86_64
version                : #1 SMP Sun Jul 24 18:53:41 CEST 2011
machine                : x86_64
nr_cpus                : 4
nr_nodes               : 1
cores_per_socket       : 2
threads_per_core       : 2
cpu_mhz                : 2393
hw_caps                : bfebfbff:28100800:00000000:00003f40:0298e3ff:00000000:00000001:00000000
virt_caps              : hvm hvm_directio
total_memory           : 5941
free_memory            : 55
free_cpus              : 0
xen_major              : 4
xen_minor              : 1
xen_extra              : .0
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
xen_commandline        : /xen.gz
cc_compiler            : gcc version 4.4.4 20100503 (Red Hat 4.4.4-2) (GCC) 
cc_compile_by          : joanna
cc_compile_domain      : 
cc_compile_date        : Tue Aug  2 14:57:19 CEST 2011
xend_config_format     : 4
[~](joanna@dom0)$ free
             total       used       free     shared    buffers     cached
Mem:       3303368    3059740     243628          0      18408    2481080
-/+ buffers/cache:     560252    2743116
Swap:      3080188          0    3080188

[marmarek]

Comment by joanna on 9 Sep 2011 18:19 UTC
I haven't seen this for a while... Closing for now.

[marmarek]

Comment by joanna on 9 Sep 2011 18:19 UTC
I haven't seen this for a while... Closing for now.