Kernel panic during boot from live media on MacBook when SD slot is empty

[GlenHenshaw]

Describe the bug
I have an iso on a USB thumb drive and an using a 2017 Intel i5 MacBook Air. The system kernel panics about six seconds into the boot process.

To Reproduce
Use etcher to burn latest hello system iso file to thumb drive. Boot from thumb drive. Press the "s" key when the screen turns grey. The system kernel panics and then reboots.

Expected behavior
The system should boot into the hello system GUI.

Screenshots

Version (please complete the following information):

• Live ISO version hello-0.8.1_0H324-FreeBSD-13.1-amd64.iso

Computer (please complete the following information):

• Device: 2017 Intel i5-based MacBook Air, 8 GB RAM

Additional context
Add any other context about the problem here.

[probonopd]

Thanks @GlenHenshaw for letting us know. Unfortunately I have never seen this before and can't deduce from looking at it what might be going wrong here. Can you boot e.g., GhostBSD on that system?

[probonopd]

Also, please try a different USB stick to rule out a hardware defect or incompatiblity.

[GlenHenshaw]

GhostBSD boots fine on this system from the same USB stick. I've tried two sticks from different vendors, both exhibit the same behavior. Also, my server machine, based on an AsRock motherboard, boot into helloSystem using the same media.

Just to be complete, I also tried burning the iso two different ways, one using the command line 'dd' command and one using the balenaEtcher macOS app. Same behavior both ways.

So this really does look like some incompatibility between the iso and the MacBook.

[u20230812]

I'm seeing the same problem with both the latest release and the development version. I tried this months ago and even tried one release back and noticed the same issue. Same MacBook runs Linux just fine.

Mine is an A1502 MacBook Pro "Core i5" 2.4 13" Late 2013 2.4 GHz Core i5 (I5-4258U).

@GlenHenshaw how did you get such a clear screen grab? It goes by so fast I tried several different cameras up to 120 fps and still go a blurry frame.

Is there an easy way to enable the UART output and capture the output with a FTDI cable instead of the graphics display?

So this really does look like some incompatibility between the iso and the MacBook.

crash

macbook pro model

processor

[probonopd]

@u20230812 can you run GhostBSD on that machine?

We need to find out what is crashing and why it is crashing on helloSystem but not on GhostBSD.

[u20230812]

@u20230812 can you run GhostBSD on that machine?

Yes, I just downloaded and checked the latest and multiuser boots right up. I tried a number of ISO with Ventoy and the Ventoy spash screen and ISO selector worked, however, trying to boot any ISO would result in a white block cursor in the upper top left part of the screen and just hang indefinitely. I had to dd the GhostBSD ISO directly to the USB drive and it booted no issues.

e399a0c193d538f09cabbdf3df397317 GhostBSD-23.06.01.iso

Let me know if there are any logs, or debug builds I can try to help isolate the issue. If it would hang instead of rebooting and we had access to the serial console that would make things a lot easier to debug and capture.

[probonopd]

Can you boot into single user mode (press s as soon as the screen becomes gray)?

In verbose (v) and single (s) mode the system should wait 30 seconds before rebooting. If you can get into single user mode, you can set a different time like so:

sysctl kern.panic_reboot_wait_time=300
exit

• Make the system stop auto-rebooting for 10 seconds when a core crashes at boot time. #461

[probonopd]

access to the serial console

For the early boot messages? I'm not sure whether this works over serial-over-USB. According to https://forums.freebsd.org/threads/using-usb-serial-adapter-for-console-output.69759/ it looks like it might be possible.

By pressing Backspace early in the boot process you get access to the FreeBSD bootloader where you should be able to enter bootloader commands.

[probonopd]

Also, have you tried the older (0.7.x, 0.6.x,...) helloSystem ISOs?

[u20230812]

@probonopd here are the results

• hello-0.7.0_0G160-FreeBSD-13.0-amd64.iso does a crash reboot the same as 0.8 and experimental
• hello-0.6.0_0F54-FreeBSD-12.2-amd64.iso boots fine
• I was not able to get single user to stop it just displayed 'single user' then kept booting and ended up crash rebooting in the same manner
• I can look in to the UART TTL solution if needed
• FreeBSD 13.2 installs fine on this computer up and running KDE
• What are the next best steps to get to the bottom of this issue that we can help with?

[probonopd]

Thanks for testing. The results are interesting. We need to find out what causes the crash, but I don't know where to start yet. I don't know what is crashing there. A kernel module? Which one?

[probonopd]

@GlenHenshaw @u20230812 please test with and without a SD card in the slot. Possibly this issue is caused by the SD card slot being empty:

• A1278 crash and reboot when SD slot is empty #335

[u20230812]

@probonopd the empty SD slot was exactly the problem. With a card installed there and still booting the ISO from a separate USB key it came right up and did not panic/crash/reboot.

I spent a few days building the helloSystem ISO trying to get the serial console working with a couple of USB TTL cables which never ended up working for whatever reason.

Here's some feedback on what I found to work while building the ISO. Maybe there are newer instructions, or maybe my method was incorrect, or maybe the docs could use an update...

1. Extremely easy to figure out how to get all this working based on docs available and scripts provided
2. The FreeBSD VM qcow2 filesystem is too small
3. Building the Live ISO
   https://hellosystem.github.io/docs/developer/building.html
   sha is not a package on FreeBSD 13.2
   pkg install -y pkg git-lite zsync wget bash zip devel/py-xdg librsvg2 ca_root_nss sudo git zenity
4. You need to edit loader.conf before building the ISO
   vi + overlays/boot/boot/loader.conf
   uftdi_load="YES"
   boot_multicons="YES"
   boot_serial="YES"
   comconsole_speed="115200"
   console="comconsole,efi"
5. zenity package not found doing the build needs to be part of Building the Live ISO
   Installing furybsd-wifi-tool-0...
   pkg: Missing dependency 'zenity'
   see pkg install on item 12.2-RC3 #3
   5a) edit overlays/uzip/furybsd-wifi-tool/manifest and remove x11/zenity and put zenity since x11/zenity is not a package on 13.2
   5b) edit overlays/uzip/furybsd-xorg-tool/manifest and remove x11/zenity and put zenity since x11/zenity is not a package on 13.2
   5c) edit settings/packages.hello adding x11/zenity
6. Comment out xcb-util-cursor in settings/packages.hello
7. export DISPLAY=:0
   7a) sudo su
   7b) sh
8. export QT_QPA_PLATFORM=offscreen
9. QT_QPA_PLATFORM=offscreen ./build.sh hello
10. Turns out modules get removed from the builder script and don't work
    vi ./settings/rc.conf.common
    kld_list="fusefs cuse ig4 iicbus iichid utouch asmc if_urndis if_cdce if_ipheth usb ucom uftdi"

[probonopd]

Hi @u20230812, thanks for your valuable feedback.

For the serial console, 4. seems to be key. It might not be necessary to build special ISOs, as you can enter the FreeBSD bootloader and can type in the lines there. I will see if I can integrate this into the ISOs in an easier way.

Regarding 5., we don't use furybsd-wifi-tool anymore in helloSystem and we shouldn't need zenity anymore. Were you building another target than hello (./build.sh hello)?

But most importantly: Did you get it to work in the end?

Step 10. should not be necessary; if you are already loading uftdi at boot time (using the bootloader) then you don't need to load it again the later boot process using kld_list.

What are you trying to achieve with steps 7.-9.?

[probonopd]

Now, the question is: Can we fix booting with an empty SD slot using some USB Mass Storage Quirks?

[probonopd]

So far I coud not get serial console over USB to work; probably we need someone with more knowledge about the FreeBSD bootloader and its lua scripting capabilities.

ISO/overlays/boot/boot/lua/local.lua

Line 82 in ae0880e

-- Verbose boot with serial console over USB

I have opened its own ticket for that topic:

• Serial console over USB to see boot messages #515

[u20230812]

Now, the question is: Can we fix booting with an empty SD slot using some USB Mass Storage Quirks?

helloSystem test results on MacBook Pro with empty SD slot

1. Stock ISO hello-0.8.1_0H324-FreeBSD-13.1-amd64.iso hw.usb.quirk did not work still boots then crashes when the SD slot is empty
2. Custom ISO hello-0.8.2_git-FreeBSD-13.2-amd64.iso hw.usb.quirk did not work still boots then crashes when the SD slot is empty even with usb added to kld_list
   vi ./settings/rc.conf.common
   kld_list="fusefs cuse ig4 iicbus iichid utouch asmc if_urndis if_cdce if_ipheth usb ucom uftdi"
3. Custom ISO hello-0.8.2_git-FreeBSD-13.2-amd64.iso hw.usb.quirk did not work still boots then crashes when the SD slot is empty even with usb_quirk_load added to loader.conf
   vi + overlays/boot/boot/loader.conf
   usb_quirk_load="YES"

https://man.freebsd.org/cgi/man.cgi?query=usb_quirk&sektion=4&manpath=FreeBSD+13.2-RELEASE

[probonopd]

Can you reproduce the crashes with stock FreeBSD images and/or GhostBSD images?

[u20230812]

Can you reproduce the crashes with stock FreeBSD images and/or GhostBSD images?

I need to amend my previous test results. It seems I blindly trusted the vendor/product IDs instead of checking them against mine before testing. All testing was done with a USB key ISO with the SD slot empty.

lsusb -s 2:3

Bus 002 Device 003: ID 05ac:8406 Apple, Inc. Internal Memory Card Reader

Setting all these allows the stock ISO hello-0.8.1_0H324-FreeBSD-13.1-amd64.iso to boot

hw.usb.quirk.0="0x05ac 0x8406 0 0xffff UQ_MSC_NO_INQUIRY"
hw.usb.quirk.1="0x05ac 0x8406 0 0xffff UQ_MSC_NO_RS_CLEAR_UA"
hw.usb.quirk.2="0x05ac 0x8406 0 0xffff UQ_MSC_NO_TEST_UNIT_READY"
hw.usb.quirk.3="0x05ac 0x8406 0 0xffff UQ_MSC_NO_START_STOP"
hw.usb.quirk.4="0x05ac 0x8406 0 0xffff UQ_MSC_NO_INQUIRY_EVPD"

UQ_MSC_NO_INQUIRY and UQ_MSC_NO_RS_CLEAR_UA alone still ended in a panic crash. I don't have a way to connect a bus analyzer to the slot so it would be trial and error to figure out what minimal combination of the quirks above are necessary.

Setting only this one allows the stock ISO hello-0.8.1_0H324-FreeBSD-13.1-amd64.iso to boot
hw.usb.quirk.0="0x05ac 0x8406 0 0xffff UQ_MSC_IGNORE"

UQ_MSC_IGNORE will obviously ignore the whole device whereas the others just tell the driver to stop issuing discovery commands at boot for that specific device.

[probonopd]

Following 07e39aa by @u20230812, please retest on hello-0.9.0_0I39-FreeBSD-13.2-amd64.iso and newer.

[GlenHenshaw]

hello-0.9.0_0139-FreeBSD-13.2-amd64.iso boots on my machine!

It won't recognize the Wi-Fi device, but that's a different problem.

[probonopd]

Added b9a2d0d.

Please retest in build 0I41 or later.

If it is still crashing, please let me know

• The exact name of the computer model
• The exact name of the card reader model
• The card reader USB vendor ID
• The card reader USB device ID

Thanks!

[GlenHenshaw]

build 0I41 also boots on my machine