Image file will not boot

[chuckb]

Describe the bug
When I build an image file with https://github.com/waldheinz/fat32-lib (version 0.6.5) using the SuperFloppyFormatter and ImageBuilder classes, and deploy to an SD using dd, my Pi will not boot. No lights, no nothing.

To reproduce
boot.img file located:
https://drive.google.com/drive/folders/1bOQQDtUStYeXudJpcb74RGdh9JHHchZ1?usp=sharing

Copy image to an SD card.
sudo dd if=boot.img of=/dev/disk2 bs=512

NOTE: Replace /dev/disk2 with location of your SD card

Mount card on a host. Add uart_2ndstage=1 to the config.txt file.

Connect a serial cable to GPIO14 and 15. Start a serial terminal. Set baud rate to 115200. Power up PiZero. Nothing is printed to the terminal. No booting is occurring.

To repro behavior in additional context below, the individual image files are also located in the provided google folder.

Expected behaviour
I can deploy my image using dd on to an SD card and the Pi will boot.

Actual behaviour
Pi does not boot as expected.

System
PiZeroW
No OS...bare metal.
Firmware version: Tried both 1.20190925 and master.
Kernel = Serial bootloader located: https://github.com/chuckb/raspbootin

Additional context
Setting the uart_2ndstage=1 produces no output in terminal. When I mount the card on my Mac, I see bootcode.bin, start.elf, config.txt, kernel.img, and fixup.dat files as expected. Further, if I md5 the files, they match my source files.

If I then copy all my source files on top of SD card image deployed files on my Mac, then the Pi will boot as expected (I am using a custom serial bootloader kernel...not Linux).

If I copy just bootcode.bin (on top of a freshly deployed image), I get

Raspberry Pi Bootcode
Read File: config.txt, 84

and 4 short green LED flashes (start.elf not found according to docs...but nevertheless appears to be there according to Mac Finder).

If I then copy start.elf, then the Pi will boot correctly. Interestingly the log notes that it can load the kernel.img (which I did not copy over).

Raspberry Pi Bootcode
Read File: config.txt, 84
Read File: start.elf, 2877988 (bytes)
MESS:00:00:01.138486:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:01.142659:0: brfs: File read: 84 bytes
MESS:00:00:01.157027:0: HDMI:EDID error reading EDID block 0 attempt 0
MESS:00:00:01.163095:0: HDMI:EDID error reading EDID block 0 attempt 1
MESS:00:00:01.169339:0: HDMI:EDID error reading EDID block 0 attempt 2
MESS:00:00:01.175589:0: HDMI:EDID error reading EDID block 0 attempt 3
MESS:00:00:01.181840:0: HDMI:EDID error reading EDID block 0 attempt 4
MESS:00:00:01.188089:0: HDMI:EDID error reading EDID block 0 attempt 5
MESS:00:00:01.194339:0: HDMI:EDID error reading EDID block 0 attempt 6
MESS:00:00:01.200589:0: HDMI:EDID error reading EDID block 0 attempt 7
MESS:00:00:01.206839:0: HDMI:EDID error reading EDID block 0 attempt 8
MESS:00:00:01.213089:0: HDMI:EDID error reading EDID block 0 attempt 9
MESS:00:00:01.219103:0: HDMI:EDID giving up on reading EDID block 0
MESS:00:00:01.224392:0: HDMI:EDID error reading EDID block 0 attempt 0
MESS:00:00:01.231579:0: HDMI:EDID error reading EDID block 0 attempt 1
MESS:00:00:01.237830:0: HDMI:EDID error reading EDID block 0 attempt 2
MESS:00:00:01.244079:0: HDMI:EDID error reading EDID block 0 attempt 3
MESS:00:00:01.250329:0: HDMI:EDID error reading EDID block 0 attempt 4
MESS:00:00:01.256579:0: HDMI:EDID error reading EDID block 0 attempt 5
MESS:00:00:01.262830:0: HDMI:EDID error reading EDID block 0 attempt 6
MESS:00:00:01.269079:0: HDMI:EDID error reading EDID block 0 attempt 7
MESS:00:00:01.275329:0: HDMI:EDID error reading EDID block 0 attempt 8
MESS:00:00:01.281580:0: HDMI:EDID error reading EDID block 0 attempt 9
MESS:00:00:01.287593:0: HDMI:EDID giving up on reading EDID block 0
MESS:00:00:01.305479:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:01.309658:0: gpioman: gpioman_get_pin_num: pin LEDS_PWR_OK not define
d
MESS:00:00:01.498925:0: gpioman: gpioman_get_pin_num: pin LEDS_PWR_OK not define
d
MESS:00:00:01.504739:0: *** Restart logging
MESS:00:00:01.508624:0: brfs: File read: 84 bytes
MESS:00:00:01.513449:0: hdmi: HDMI:EDID error reading EDID block 0 attempt 0
MESS:00:00:01.521062:0: hdmi: HDMI:EDID error reading EDID block 0 attempt 1
MESS:00:00:01.527827:0: hdmi: HDMI:EDID error reading EDID block 0 attempt 2
MESS:00:00:01.534598:0: hdmi: HDMI:EDID error reading EDID block 0 attempt 3
MESS:00:00:01.541369:0: hdmi: HDMI:EDID error reading EDID block 0 attempt 4
MESS:00:00:01.548139:0: hdmi: HDMI:EDID error reading EDID block 0 attempt 5
MESS:00:00:01.554911:0: hdmi: HDMI:EDID error reading EDID block 0 attempt 6
MESS:00:00:01.561681:0: hdmi: HDMI:EDID error reading EDID block 0 attempt 7
MESS:00:00:01.568452:0: hdmi: HDMI:EDID error reading EDID block 0 attempt 8
MESS:00:00:01.575223:0: hdmi: HDMI:EDID error reading EDID block 0 attempt 9
MESS:00:00:01.581758:0: hdmi: HDMI:EDID giving up on reading EDID block 0
MESS:00:00:01.587576:0: hdmi: HDMI:EDID error reading EDID block 0 attempt 0
MESS:00:00:01.595275:0: hdmi: HDMI:EDID error reading EDID block 0 attempt 1
MESS:00:00:01.602046:0: hdmi: HDMI:EDID error reading EDID block 0 attempt 2
MESS:00:00:01.608818:0: hdmi: HDMI:EDID error reading EDID block 0 attempt 3
MESS:00:00:01.615587:0: hdmi: HDMI:EDID error reading EDID block 0 attempt 4
MESS:00:00:01.622358:0: hdmi: HDMI:EDID error reading EDID block 0 attempt 5
MESS:00:00:01.629129:0: hdmi: HDMI:EDID error reading EDID block 0 attempt 6
MESS:00:00:01.635900:0: hdmi: HDMI:EDID error reading EDID block 0 attempt 7
MESS:00:00:01.642671:0: hdmi: HDMI:EDID error reading EDID block 0 attempt 8
MESS:00:00:01.649442:0: hdmi: HDMI:EDID error reading EDID block 0 attempt 9
MESS:00:00:01.655976:0: hdmi: HDMI:EDID giving up on reading EDID block 0
MESS:00:00:01.661510:0: hdmi: HDMI:hdmi_get_state is deprecated, use hdmi_get_di
splay_state instead
MESS:00:00:01.670256:0: hdmi: HDMI:hdmi_get_state is deprecated, use hdmi_get_di
splay_state instead
MESS:00:00:01.680135:0: Failed to open command line file 'cmdline.txt'
MESS:00:00:01.690071:0: Failed to load Device Tree file 'bcm2708-rpi-0-w.dtb'
MESS:00:00:01.698115:0: brfs: File read: /mfs/sd/kernel.img
MESS:00:00:01.701990:0: Loading 'kernel.img' to 0x2000000 size 0x518
MESS:00:00:01.708099:0: gpioman: gpioman_get_pin_num: pin EMMC_ENABLE not define
d
MESS:00:00:01.716179:0: uart: Set PL011 baud rate to 103448.300000 Hz
MESS:00:00:01.722989:0: uart: Baud rate change done...
MESS:00:00:01.726418:0: uart: Baud rate change done...
MESS:00:00:01.731934:0: gpioman: gpioman_get_pin_num: pin SDCARD_CONTROL_POWER|
Raspbootin V2
#############
Built for: Raspberry Pi
#############

It appears to me that FAT file handling is different between loading bootcode.bin, within bootcode.bin, and then works further differently within start.elf (given that it can find kernel.img without an over copy).

I suspect that my FAT library and the FAT handling contained within the firmware have some compatibility issues. But my Mac has no problem mounting, reading/finding the files after image is deployed to the card with dd. I can also run a file system check after deploying the image and it reports no errors.

I am willing to troubleshoot the FAT library, but it would be great if I can get some more diagnostics from the firmware as to why it is having trouble reading the files. Any suggestions?

[chuckb]

I figured out the root cause. The Java FAT library was writing out long file name (LFN) directory entries even if files names were 8.3 (in this case, because the names were lower case). Once I changed the file names to upper case and rebuilt and deployed the image, then the Pi will boot.

Interestingly, I also confirmed the behavior of overwriting the file from the Mac, that even if the file is in lower case on the Mac file system, the Mac FAT driver does not write an LFN entry and further converts the file name to uppercase within the directory record (while setting bits 3 & 4 on the directory 0x0C byte). Flags were introduced in Windows NT and certain Windows XP versions that deal with lowercase handling of 8.3 file names without having to write a LFN, which appears to be what the Mac driver is doing.

Before I close this issue, can a RPi team member confirm whether first and second stage boot loaders deal with FAT volumes containing LFNs? I'd like to know because then I will look into how the Java lib builds them, because obviously something is not quite right.

[chuckb]

I forked and fixed the Java library https://github.com/chuckb/fat32-lib to write lower case short file names using the user attributes in the 0x0c offset of the fat directory entry. Pi bootloaders can now find files in volumes created by this library.

[timg236]

To confirm your comments both the ROM and bootcode.bin / SPI EEPROM all require short filenames.
https://www.raspberrypi.org/forums/viewtopic.php?t=20588#p198998

Start.elf supports long filenames for device-tree blobs and maybe the Pi4 EEPROM will in future. However, since the chip ROM requires short filenames for recovery.bin I think you should assume that the boot partition on Raspberry Pi must always contain the short filenames.

[chuckb]

The nuance and confusion stems from the FAT/VFAT specification, which has much "history". Mixed case file names, regardless of length, require long file names. In the early days of the FAT specification, 8.3 filenames did not support lower case, and any file name created would be stored with an upper case name within the directory structure. Then came along VFAT, which supported mixed case, UNICODE, long file names (LFN). Then came along a modification to VFAT supporting lowercase 8.3 basename and extension without requiring use of the LFN specification. So it seems like the 1st and 2nd stage bootloaders support a type of hybrid VFAT, because if I create lowercase 8.3 names with the 0x0c flags set, the bootloaders can find the files.