'isp-extract: Found 5/6 calibration files' during system update

[thasti]

While running pacman -Syu just a minute ago, I got the following message with a call to report this bug. Just hours before, I upgraded macOS to Sonoma. asahi-diagnose attached here: asahi-diagnose-20231008-134956.txt

Upgrading vendor firmware package
Traceback (most recent call last):
  File "/usr/bin/asahi-fwextract", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/lib/python3.11/site-packages/asahi_firmware/update.py", line 52, in main
    update_firmware(args.source, args.dest)
  File "/usr/lib/python3.11/site-packages/asahi_firmware/update.py", line 31, in update_firmware
    col = ISPFWCollection(str(tmpdir))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/asahi_firmware/isp.py", line 82, in __init__
    self.load(source_path)
  File "/usr/lib/python3.11/site-packages/asahi_firmware/isp.py", line 128, in load
    for fwf in self.extract_isp(data):
  File "/usr/lib/python3.11/site-packages/asahi_firmware/isp.py", line 109, in extract_isp
    raise ValueError(f"isp-extract: Found {found}/{ISP_SETFILE_COUNT} calibration files. Please report this bug.")
ValueError: isp-extract: Found 5/6 calibration files. Please report this bug.
error: command failed to execute correctly

[marcan]

Please update again, it should now be a warning instead of an error.

After the update, could you please reboot and paste the output of ls -al /lib/firmware/vendor/apple?

[jannau]

added files in 12.4 appleh13camerad compared to 12.3:

/usr/local/share/firmware/isp/1822_02XX.dat
/usr/local/share/firmware/isp/1922_02XX.dat

added files in 13.5 appleh13camerad compared to 12.4:

/usr/local/share/firmware/isp/2123_01XX.dat
/usr/local/share/firmware/isp/2922_02XX.dat
/usr/local/share/firmware/isp/3723_01XX.dat
/usr/local/share/firmware/isp/7523_01XX.dat
/usr/local/share/firmware/isp/8723_01XX.dat
/usr/local/share/firmware/isp/9723_01XX.dat

[marcan]

OK, so the extra ones in 12.4 are the "second versions" of IMX248 and IMX558 we don't even use (maybe we should look into that logic), and 13.5 added the "second version" of IMX364. That means 12.3 has all the firmwares we actually use right now.

We currently use the first version for everything. Worth noting that 12.4 was released with IMX248 and IMX558 M2 machines. Maybe the "second versions" have to do with the ISP version somehow? Any idea how we might determine which specific variant to use and whether it matters, @eiln?

[thasti]

I confirm, I only get a warning now:

WARNING:asahi_firmware.isp:isp-extract: Found 5/6 calibration files.
Firmware upgraded

Running ls -al /lib/firmware/vendor/apple after a reboot gives:

[thasti@slab ~]$ ls -al /lib/firmware/vendor/apple
total 496
drwxr-xr-x 2 root root   220  8 oct.  20:05 .
drwxr-xr-x 5 root root   120  8 oct.  20:05 ..
-rw-r--r-- 1 root root 66080  8 oct.  20:05 dfrmtfw-j293.bin
-rw-r--r-- 1 root root 66080  8 oct.  20:05 dfrmtfw-j493.bin
-rw-r--r-- 1 root root 17472  8 oct.  20:05 isp_1820_01XX.dat
-rw-r--r-- 1 root root 17472  8 oct.  20:05 isp_1822_02XX.dat
-rw-r--r-- 1 root root 44352  8 oct.  20:05 isp_1921_01XX.dat
-rw-r--r-- 1 root root 44352  8 oct.  20:05 isp_1922_02XX.dat
-rw-r--r-- 1 root root 14016  8 oct.  20:05 isp_8720_01XX.dat
-rw-r--r-- 1 root root 79110  8 oct.  20:05 tpmtfw-j413.bin
-rw-r--r-- 1 root root 78510  8 oct.  20:05 tpmtfw-j493.bin

[eiln]

@marcan The only way to find out is through a tracer hook that fetches the first four bytes of the macOS-loaded setfile (big endian magic number). macOS runs about 6 firmware GET commands to determine this version. Diffing the v1/v2 files does show something changed in the latter half, but at least on my machine (J293AP) I can't tell the difference between v1/v2. Is there a difference on newer machines? Is CH_INFO output the same across?