Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segmentation fault when updating bhyve tables #371

Closed
goligo opened this issue Jan 9, 2019 · 7 comments
Assignees

Comments

@goligo
Copy link

@goligo goligo commented Jan 9, 2019

FreeBSD version ( uname -a ): 11.2
CBSD version ( cbsd version ): 12.0.3

When running initenv after upgrading using pkg, the following error occured:

  • Update bhyve tables: bhyve_dskcontroller for ubuntu
    Segmentation fault (core dumped)
  • Update bhyve tables: cpu_topology for ubuntu
    Segmentation fault (core dumped)

I was updating from a pretty old version (11.1.11) and had issues with bhyve in the past, so I guess the database structure was not what cbsd expected it to be. Would it be possible to add some sanity checks in the update, so it shows a proper error message (like "Update failure - table missing"), rather than a segmentation fault?

Btw - I could neither start the bhyve VM nor edit its configuration after this happened, I ended up fixing the local.sqlite file of the affected VM by manually adding the missing tables.

@olevole olevole self-assigned this Jan 9, 2019
@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jan 9, 2019

I will try to reproduce it. you did not try to run 'cbsd initenv' second time? probably not one migration. Thanks!

@goligo

This comment has been minimized.

Copy link
Author

@goligo goligo commented Jan 9, 2019

I did only run initenv once. Here is the full log with obfuscated jail names:

-------[CBSD v.12.0.3]-------
 This is install/upgrade scripts for CBSD.
 Don't forget to backup.
-----------------------------
Do you want prepare or upgrade hier environment for CBSD now?
[yes(1) or no(0)]
yes
>>> Installing or upgrading
[Stage 0: upgrading and migration data from 11.1.11 to 12.0.3]
  * Xen tables re-created
  * Update bhyvedsk tables: controller_id for ubuntu
  * expose tables: remove UNIQUE pin for: jail1
  * create backup of original base to: /tmp/expose.jail1.sqlite.24424
  * expose tables: remove UNIQUE pin for: jail2
  * create backup of original base to: /tmp/expose.jail2.sqlite.24424
  * expose tables: remove UNIQUE pin for: jail3
  * create backup of original base to: /tmp/expose.jail3.sqlite.24424
  * expose tables: remove UNIQUE pin for: jail4
  * create backup of original base to: /tmp/expose.jail4.sqlite.24424
  * expose tables: remove UNIQUE pin for: jail5
  * create backup of original base to: /tmp/expose.jail5.sqlite.24424
  * expose tables: remove UNIQUE pin for: jail6
  * create backup of original base to: /tmp/expose.jail6.sqlite.24424
  * Update local tables: statsd_jail_enable
  * Update local tables: statsd_bhyve_enable
  * Update local tables: statsd_hoster_enable
  * Update bhyve tables: bhyve_dskcontroller for ubuntu
Segmentation fault (core dumped)
  * Update settings tables: vm_cpu_topology for ubuntu
  * Update settings tables: cd_boot_firmware for ubuntu
  * Update jails tables: add allow_read_msgbuf
  * Update settings tables: debug_engine for ubuntu
  * Update jails tables: add allow_raw_sockets
! Special warning ! CBSD 11.2.1 introduces new jail settings: allow_raw_sockets, which is set to 0 by default (disabled).
! Special warning ! Old jails still use the setting allow_raw_sockets=1 (on).
! Special warning ! Please invert allow_raw_settings options in jconstruct-tui/jconfig or
! Special warning !  use /usr/jails/etc/jail-freebsd-default.conf to overwrite global params to restore old behavior.
  * Update bhyve settings tables for ubuntu
   * moving:  vm_cpus vm_ram vm_os_type vm_boot virtio_type vm_hostbridge vm_os_profile vm_iso_path vm_console vm_efi vm_vnc_port protected hidden bhyve_generate_acpi bhyve_wire_memory bhyve_rts_keeps_utc bhyve_force_msi_irq bhyve_x2apic_mode bhyve_mptable_gen bhyve_ignore_msr_acc cd_vnc_wait bhyve_vnc_resolution bhyve_vnc_tcp_bind ip4_addr vnc_password bhyve_vnc_vgaconf
  * Update settings tables: media_auto_eject for ubuntu
  * Update jails tables: add allow_fusefs
  * Update jails tables: add allow_vmm
  * Update settings tables: xhci for ubuntu
  * Update bhyvedsk tables: crypto_key for ubuntu
  * Update bhyve tables: cpu_topology for ubuntu
Segmentation fault (core dumped)
[Stage 1: account & dir hier]
 * Check hier and permission...
./lib missing (created)
 * write directory id: jaildatadir
 * write directory id: jailsysdir
 * write directory id: jailrcconfdir
 * write directory id: dbdir
[Stage 2: build tools]
[Stage 3: local settings]
[Stage 4: update default skel resolv.conf]
[Stage 5: refreshing inventory]
[Stage 6: authentication keys]
[Stage 7: modules]
[Stage 9: cleanup]
 * Remove obsolete files...
	Removing obsolete file: /usr/jails/etc/defaults/vm-linux-ubuntuserver-x86-17.04.conf
	Removing obsolete file: /usr/jails/etc/defaults/vm-linux-ArchLinux-x86-2017.conf
	Removing obsolete file: /usr/jails/etc/defaults/vm-linux-Kali-2017-amd64.conf
	Removing obsolete file: /usr/jails/etc/defaults/vm-linux-Tails-3.0.1.conf
	Removing obsolete file: /usr/jails/etc/defaults/vm-linux-CentOS-7.3-x86_64.conf
	Removing obsolete file: /usr/jails/etc/defaults/vm-dflybsd-x86-4.conf
	Removing obsolete file: /usr/jails/etc/defaults/vm-dflybsd-x86-5.conf
	Removing obsolete file: /usr/jails/etc/defaults/vm-linux-TinyCore-x86-8.conf
	Removing obsolete file: /usr/jails/tools/logger
	Removing obsolete file: /usr/jails/etc/defaults/vm-linux-CentOS-7.4-x86_64.conf
	Removing obsolete file: /usr/jails/etc/defaults/vm-freebsd-TrueOS-x64-latest-unstable.conf
	Removing obsolete file: /usr/jails/etc/defaults/vm-freebsd-TrueOS-x64-latest-stable.conf
	Removing obsolete file: /usr/jails/etc/defaults/vm-freebsd-BSDRP-x64-1.80.conf
	Removing obsolete file: /usr/jails/etc/defaults/vm-linux-kubuntu-x86-16.conf
	Removing obsolete file: /usr/jails/etc/defaults/vm-linux-ubuntuserver-x86-16.04.conf
	Removing obsolete file: /usr/jails/etc/defaults/vm-other-Haiku-R1Alpha-4.1.conf
	Removing obsolete file: /usr/jails/etc/defaults/vm-freebsd-FreeBSD-x64-12.0-LATEST.conf
	Removing obsolete file: /usr/jails/etc/defaults/vm-freebsd-GhostBSD-x64-11.1.conf
	Removing obsolete file: /usr/jails/rc.d/cbsd-statsd
	Removing obsolete file: /usr/jails/tools/racct-statsd
[Stage X: upgrading]
  * Insert default topology into vm_cpu_topology table
>>> Done

I remember I already had some errors with the same VM, when upgrading to 11.1.11 - but the VM continued to work as expected, so I did not follow up on it. So it may be that the database was even older than 11.1.11, when trying to run the update scripts.

@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jan 9, 2019

11.1.11 was released 2017-12-11, so there were many migrations. Yes, we need improve the compatibility mechanism for long-term upgrade. I will take a timeout to reproduce and study the problem.

So, what is your current state? You fix schemas by hand, 'cbsd initenv' work without error now and old vm workable, new vm created without any issue?

@goligo

This comment has been minimized.

Copy link
Author

@goligo goligo commented Jan 9, 2019

Didn't try to run initenv again. Old VM working again, new VM created without issue.

One interesting change: Network interface name changed from enp0s4 to enp0s5 inside the VM, so I had to adapt /etc/network/interfaces to get networking going again.

@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jan 9, 2019

Didn't try to run initenv again. Old VM working again, new VM created without issue.

One interesting change: Network interface name changed from enp0s4 to enp0s5 inside the VM, so I had to adapt /etc/network/interfaces to get networking going again.

Yeah, interface naming in linux is now sensitive to bus id. Most likely, this means that another device has been added (for example NVME or xhci and CBSD init pci ID for new devices before the NIC).

At a minimum, this requires the addition of management functionality for BUS ID ( at the moment, CBSD initializes all devices, successively increasing the bus ID for each device). As the maximum, we have to save/store BUS id for each devices when it was initialized for the first time.

PS: I don't like the way Linux now calls network interfaces ;))

@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Feb 28, 2019

upd:

One interesting change: Network interface name changed from enp0s4 to enp0s5 inside the VM, so I had to adapt /etc/network/interfaces to get networking going again.

Now CBSD store initial device settings: 19f73d9

Starting with CBSD 12.0.6 adding/removing devices will not affect the rest devices ( PCI bus,pcislot,function now persistent (via SQLite3 database) for already initialized devices)

olevole added a commit that referenced this issue Jun 21, 2019
it prevents cyclical loop/segfault ( + protection when sub-included file not exist ). Fixes for Issue #371
Reported by: goligo@
@olevole

This comment has been minimized.

Copy link
Collaborator

@olevole olevole commented Jun 21, 2019

This should now be fixed. In addition in recent CBSD, there are no more duplicate files in the /usr/local/cbsd/ and $workdir. This could be another reason for incorrect work (different schema versions in $workdir/share/.schema and /usr/local/cbsd/share/.schema.

Please reopen issue if problem is not fixed with CBSD 12.0.12-RELEASE (next release).
Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.