NVME (0x6) removal from BOOT_ORDER not honored

[AlphaDE]

Describe the bug

I added a Pineberry Pi BM1L HatDrive! Bottom NVME adapter with a 512GB NVME to my Raspberry Pi 5. After setting up, I changed the boot-order with sudo rpi-eeprom-config --edit to BOOT_ORDER=0xf416. Later due to an error in fstab, I got stuck in emergency mode and tried to recover. Despite removing "0x6" from the BOOT_ORDER, the Pi 5 bootet from the NVME as long as it was connected making it impossible to boot from sdcard, mount the NVME partition and fix the issue. Also changing the order (61) which should boot from sdcard first does boot from NVME despite a sdcard is inserted.
When 0x6 is removed from BOOT_ORDER or moved back, Pi 5 should not try to boot from NVME anymore or honor boot order. BOOT_ORDER=0xf461 boots from NVME despite connected sdcard.

An article on the German heise.de website mentions the issue, too: "Nachdem wir das Booten von der NVMe-SSD aktiviert hatten, konnten wir allerdings nicht mehr zur SD-Karte wechseln, selbst wenn wir die SSD komplett aus der Boot-Reihenfolge entfernen." After we activated booting from NVMe-SSD, we could not change back to the SD-card even when we removed the NVMe-SSD completely from the boot-order.

https://www.golem.de/news/nvme-adapter-hatdrive-im-test-das-enttaeuschende-ssd-power-up-fuer-den-raspberry-pi-5-2312-180512-2.html

Steps to reproduce the behaviour

Add NVME hat with properly imaged and bootable NVME
Set BOOT_ORDER to boot from NVME
Revert the setting
Pi 5 still boots from NVME as long connected

Device (s)

Raspberry Pi 5

Bootloader configuration.

[all]
BOOT_UART=1
POWER_OFF_ON_HALT=0
BOOT_ORDER=0xf416
PCIE_PROBE=1

System

Raspberry Pi reference 2023-12-05
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 70cd6f2a1e34d07f5cba7047aea5b92457372e05, stage4

2024/01/22 14:44:36
version e891ded664f30eae0d95f44db449a2f70dd66c29 (release)
timestamp 1705934676
update-time 1706198011
capabilities 0x0000007f

2024/01/22 14:44:36
Copyright (c) 2012 Broadcom
version e891ded6 (release) (embedded)

Linux raspberrypi5-alfter 6.1.0-rpi7-rpi-v8 #1 SMP PREEMPT Debian 1:6.1.63-1+rpt1 (2023-11-24) aarch64 GNU/Linux

Bootloader logs

No response

USB boot

No response

NVMe boot

/dev/nvme0n1 /dev/ng0n1 73CPH60RQMZK KBG50ZNS512G KIOXIA 1 512,11 GB / 512,11 GB 512 B + 0 B ANGA0102

NVME Identify Controller:
vid : 0x1e0f
ssvid : 0x1e0f
sn : 73CPH60RQMZK
mn : KBG50ZNS512G KIOXIA
fr : ANGA0102
rab : 2
ieee : 8ce38e
cmic : 0
[3:3] : 0 ANA not supported
[2:2] : 0 PCI
[1:1] : 0 Single Controller
[0:0] : 0 Single Port

mdts : 9
cntlid : 0
ver : 0x10400
rtd3r : 0x186a0
rtd3e : 0x7a120
oaes : 0x200
[31:31] : 0 Discovery Log Change Notice Not Supported
[27:27] : 0 Zone Descriptor Changed Notices Not Supported
[15:15] : 0 Normal NSS Shutdown Event Not Supported
[14:14] : 0 Endurance Group Event Aggregate Log Page Change Notice Not Supported
[13:13] : 0 LBA Status Information Notices Not Supported
[12:12] : 0 Predictable Latency Event Aggregate Log Change Notices Not Supported
[11:11] : 0 Asymmetric Namespace Access Change Notices Not Supported
[9:9] : 0x1 Firmware Activation Notices Supported
[8:8] : 0 Namespace Attribute Changed Event Not Supported

ctratt : 0x82
[19:19] : 0 Flexible Data Placement Not Supported
[15:15] : 0 Extended LBA Formats Not Supported
[14:14] : 0 Delete NVM Set Not Supported
[13:13] : 0 Delete Endurance Group Not Supported
[12:12] : 0 Variable Capacity Management Not Supported
[11:11] : 0 Fixed Capacity Management Not Supported
[10:10] : 0 Multi Domain Subsystem Not Supported
[9:9] : 0 UUID List Not Supported
[8:8] : 0 SQ Associations Not Supported
[7:7] : 0x1 Namespace Granularity Supported
[6:6] : 0 Traffic Based Keep Alive Not Supported
[5:5] : 0 Predictable Latency Mode Not Supported
[4:4] : 0 Endurance Groups Not Supported
[3:3] : 0 Read Recovery Levels Not Supported
[2:2] : 0 NVM Sets Not Supported
[1:1] : 0x1 Non-Operational Power State Permissive Supported
[0:0] : 0 128-bit Host Identifier Not Supported

rrls : 0
cntrltype : 1
[7:2] : 0 Reserved
[1:0] : 0x1 I/O Controller
fguid : 00000000-0000-0000-0000-000000000000
crdt1 : 0
crdt2 : 0
crdt3 : 0
nvmsr : 1
[1:1] : 0 NVM subsystem Not part of an Enclosure
[0:0] : 0x1 NVM subsystem part of an Storage Device

vwci : 0
[7:7] : 0 VPD Write Cycles Remaining field is Not valid.
[6:0] : 0 VPD Write Cycles Remaining

mec : 1
[1:1] : 0 NVM subsystem Not contains a Management Endpoint on a PCIe port
[0:0] : 0x1 NVM subsystem contains a Management Endpoint on an SMBus/I2C port

oacs : 0x1f
[10:10] : 0 Lockdown Command and Feature Not Supported
[9:9] : 0 Get LBA Status Capability Not Supported
[8:8] : 0 Doorbell Buffer Config Not Supported
[7:7] : 0 Virtualization Management Not Supported
[6:6] : 0 NVMe-MI Send and Receive Not Supported
[5:5] : 0 Directives Not Supported
[4:4] : 0x1 Device Self-test Supported
[3:3] : 0x1 NS Management and Attachment Supported
[2:2] : 0x1 FW Commit and Download Supported
[1:1] : 0x1 Format NVM Supported
[0:0] : 0x1 Security Send and Receive Supported

acl : 3
aerl : 7
frmw : 0x14
[5:5] : 0 Multiple FW or Boot Update Detection Not Supported
[4:4] : 0x1 Firmware Activate Without Reset Supported
[3:1] : 0x2 Number of Firmware Slots
[0:0] : 0 Firmware Slot 1 Read/Write

lpa : 0xe
[6:6] : 0 Telemetry Log Data Area 4 Not Supported
[5:5] : 0 LID 0x0, Scope of each command in LID 0x5, 0x12, 0x13 Not Supported
[4:4] : 0 Persistent Event log Not Supported
[3:3] : 0x1 Telemetry host/controller initiated log page Supported
[2:2] : 0x1 Extended data for Get Log Page Supported
[1:1] : 0x1 Command Effects Log Page Supported
[0:0] : 0 SMART/Health Log Page per NS Not Supported

elpe : 255
[7:0] : 255 (0's based) Error Log Page Entries (ELPE)

npss : 4
[7:0] : 4 (0's based) Number of Power States Support (NPSS)

avscc : 0
[0:0] : 0 Admin Vendor Specific Commands uses Vendor Specific Format

apsta : 0x1
[0:0] : 0x1 Autonomous Power State Transitions Supported

wctemp : 351
[15:0] : 78°C (351 Kelvin) Warning Composite Temperature Threshold (WCTEMP)

cctemp : 355
[15:0] : 82°C (355 Kelvin) Critical Composite Temperature Threshold (CCTEMP)

mtfa : 50
hmpre : 16384
hmmin : 3584
tnvmcap : 512.110.190.592
[127:0] : 512.110.190.592
Total NVM Capacity (TNVMCAP)

unvmcap : 0
[127:0] : 0
Unallocated NVM Capacity (UNVMCAP)

rpmbs : 0
[31:24]: 0 Access Size
[23:16]: 0 Total Size
[5:3] : 0 Authentication Method
[2:0] : 0 Number of RPMB Units

edstt : 13
dsto : 1
fwug : 1
kas : 0
hctma : 0x1
[0:0] : 0x1 Host Controlled Thermal Management Supported

mntmt : 273
[15:0] : 0°C (273 Kelvin) Minimum Thermal Management Temperature (MNTMT)

mxtmt : 356
[15:0] : 83°C (356 Kelvin) Maximum Thermal Management Temperature (MXTMT)

sanicap : 0xa0000002
[31:30] : 0x2 Media is additionally modified after sanitize operation completes successfully
[29:29] : 0x1 No-Deallocate After Sanitize bit in Sanitize command Not Supported
[2:2] : 0 Overwrite Sanitize Operation Not Supported
[1:1] : 0x1 Block Erase Sanitize Operation Supported
[0:0] : 0 Crypto Erase Sanitize Operation Not Supported

hmminds : 1
hmmaxd : 1024
nsetidmax : 0
endgidmax : 0
anatt : 0
anacap : 0
[7:7] : 0 Non-zero group ID Not Supported
[6:6] : 0 Group ID does change
[4:4] : 0 ANA Change state Not Supported
[3:3] : 0 ANA Persistent Loss state Not Supported
[2:2] : 0 ANA Inaccessible state Not Supported
[1:1] : 0 ANA Non-optimized state Not Supported
[0:0] : 0 ANA Optimized state Not Supported

anagrpmax : 0
nanagrpid : 0
pels : 0
domainid : 0
megcap : 0
sqes : 0x66
[7:4] : 0x6 Max SQ Entry Size (64)
[3:0] : 0x6 Min SQ Entry Size (64)

cqes : 0x44
[7:4] : 0x4 Max CQ Entry Size (16)
[3:0] : 0x4 Min CQ Entry Size (16)

maxcmd : 0
nn : 1
oncs : 0xdf
[8:8] : 0 Copy Not Supported
[7:7] : 0x1 Verify Supported
[6:6] : 0x1 Timestamp Supported
[5:5] : 0 Reservations Not Supported
[4:4] : 0x1 Save and Select Supported
[3:3] : 0x1 Write Zeroes Supported
[2:2] : 0x1 Data Set Management Supported
[1:1] : 0x1 Write Uncorrectable Supported
[0:0] : 0x1 Compare Supported

fuses : 0x1
[0:0] : 0x1 Fused Compare and Write Supported

fna : 0
[3:3] : 0 Format NVM Broadcast NSID (FFFFFFFFh) Supported
[2:2] : 0 Crypto Erase Not Supported as part of Secure Erase
[1:1] : 0 Crypto Erase Applies to Single Namespace(s)
[0:0] : 0 Format Applies to Single Namespace(s)

vwc : 0x7
[2:1] : 0x3 The Flush command supports NSID set to FFFFFFFFh
[0:0] : 0x1 Volatile Write Cache Present

awun : 31
awupf : 0
icsvscc : 0
[0:0] : 0 NVM Vendor Specific Commands uses Vendor Specific Format

nwpc : 0
[2:2] : 0 Permanent Write Protect Not Supported
[1:1] : 0 Write Protect Until Power Supply Not Supported
[0:0] : 0 No Write Protect and Write Protect Namespace Not Supported

acwu : 31
ocfs : 0
[1:1] : 0 Controller Copy Format 1h Not Supported
[0:0] : 0 Controller Copy Format 0h Not Supported

sgls : 0
[15:8] : 0 SGL Descriptor Threshold
[1:0] : 0 Scatter-Gather Lists Not Supported

mnan : 0
maxdna : 0
maxcna : 0
subnqn : nqn.2019-10.com.kioxia:KBG50ZNS512G KIOXIA:73CPH60RQMZK
ioccsz : 0
iorcsz : 0
icdoff : 0
fcatt : 0
[0:0] : 0 Dynamic Controller Model

msdbd : 0
ofcs : 0
[0:0] : 0 Disconnect command Not Supported

ps 0 : mp:4.10W operational enlat:1 exlat:1 rrt:0 rrl:0
rwt:0 rwl:0 idle_power:- active_power:-
active_power_workload:-
ps 1 : mp:2.40W operational enlat:1 exlat:1 rrt:1 rrl:1
rwt:1 rwl:1 idle_power:- active_power:-
active_power_workload:-
ps 2 : mp:1.90W operational enlat:1 exlat:1 rrt:2 rrl:2
rwt:2 rwl:2 idle_power:- active_power:-
active_power_workload:-
ps 3 : mp:0.0500W non-operational enlat:800 exlat:1000 rrt:3 rrl:3
rwt:3 rwl:3 idle_power:- active_power:-
active_power_workload:-
ps 4 : mp:0.0030W non-operational enlat:3000 exlat:32000 rrt:4 rrl:4
rwt:4 rwl:4 idle_power:- active_power:-
active_power_workload:-

NVME Identify Namespace 1:
nsze : 0x3b9e12b0
ncap : 0x3b9e12b0
nuse : 0x3b9e12b0
nsfeat : 0x2
[4:4] : 0 NPWG, NPWA, NPDG, NPDA, and NOWS are Not Supported
[3:3] : 0 NGUID and EUI64 fields if non-zero, Reused
[2:2] : 0 Deallocated or Unwritten Logical Block error Not Supported
[1:1] : 0x1 Namespace uses NAWUN, NAWUPF, and NACWU
[0:0] : 0 Thin Provisioning Not Supported

nlbaf : 1
flbas : 0
[6:5] : 0 Most significant 2 bits of Current LBA Format Selected
[4:4] : 0 Metadata Transferred in Separate Contiguous Buffer
[3:0] : 0 Least significant 4 bits of Current LBA Format Selected

mc : 0
[1:1] : 0 Metadata Pointer Not Supported
[0:0] : 0 Metadata as Part of Extended Data LBA Not Supported

dpc : 0
[4:4] : 0 Protection Information Transferred as Last 8 Bytes of Metadata Not Supported
[3:3] : 0 Protection Information Transferred as First 8 Bytes of Metadata Not Supported
[2:2] : 0 Protection Information Type 3 Not Supported
[1:1] : 0 Protection Information Type 2 Not Supported
[0:0] : 0 Protection Information Type 1 Not Supported

dps : 0
[3:3] : 0 Protection Information is Transferred as Last 8 Bytes of Metadata
[2:0] : 0 Protection Information Disabled

nmic : 0
[0:0] : 0 Namespace Multipath Not Capable

rescap : 0
[7:7] : 0 Ignore Existing Key - Used as defined in revision 1.2.1 or earlier
[6:6] : 0 Exclusive Access - All Registrants Not Supported
[5:5] : 0 Write Exclusive - All Registrants Not Supported
[4:4] : 0 Exclusive Access - Registrants Only Not Supported
[3:3] : 0 Write Exclusive - Registrants Only Not Supported
[2:2] : 0 Exclusive Access Not Supported
[1:1] : 0 Write Exclusive Not Supported
[0:0] : 0 Persist Through Power Loss Not Supported

fpi : 0
[7:7] : 0 Format Progress Indicator Not Supported

dlfeat : 1
[4:4] : 0 Guard Field of Deallocated Logical Blocks is set to 0xFFFF
[3:3] : 0 Deallocate Bit in the Write Zeroes Command is Not Supported
[2:0] : 0x1 Bytes Read From a Deallocated Logical Block and its Metadata are 0x00

nawun : 255
nawupf : 0
nacwu : 255
nabsn : 255
nabo : 0
nabspf : 0
noiob : 0
nvmcap : 512.110.190.592
mssrl : 0
mcl : 0
msrc : 0
nulbaf : 0
anagrpid: 0
nsattr : 0
nvmsetid: 0
endgid : 0
nguid : 00000000000000008ce38e04044f8bf1
eui64 : 8ce38e04044f8bf1
LBA Format 0 : Metadata Size: 0 bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good (in use)
LBA Format 1 : Metadata Size: 0 bytes - Data Size: 4096 bytes - Relative Performance: 0x1 Better

19:41 25.01.2024

Network (TFTP boot)

No response

[peterharperuk]

Works for me. I suspect it's something to do with PCIE_PROBE=1. Are the partition name different on your ssd and sd cards?

[AlphaDE]

The instructions I found refer to use this and docs say to use it, if the hat is not HAT+. I found no reference whether the Pineberry BM1L is Hat+ or not.

[timg236]

Please can you attach boot loader logs showing the behaviour and verify that you don’t have the same partition uuid on both the sd and the sd card. Otherwise, it’s not possible to predict which partition the kernel will mount.

[AlphaDE]

@timg236 The partition IDs might be the reason. The howtos for setting up an NVMe recommend to setup the system on sdcard and then use SD Card Copier to transfer the data to the NVMe. But that leads obviously to identical partition UUIDs on both media and then the "broken" NVME took precende. I'll see how I can modify the partition UUIDs on the sdcard.

[timg236]

That would be explain it.
I normally use rpi imager on pc with usb nvme adapter or boot from a one off rpi is sd boot and run the imager on rpi os to flash the local nvme.

[AlphaDE]

Digging deeper I found that partition 1 of the sdcard was mount to /boot/firmware and partition 2 of the NVMe as /, I changed the sdcard uuid with fdisk which should have fixed that. The NVMe How-To need to be reworked to avoid that confusion.

Should be fixed.