Encrypted bcachefs on a rootfs kernel unable to mount, with passphrase

[dblaber]

I am going to try to provide as much details as I can, however I am new to debugging kernel modules, and filesystems, please let me know if any additional information is needing, willing to debug if necessary. This does seem easy to reproduce though . My goal is to mount an encrypted bcachefs rootfs volume at boot.
Env:
-arch rolling
-util-linux 2.39.3
-kernel 6.7.x and 6.8.0
-config in mkinitcpio.conf for bcachefs in MODULES, HOOKS and BINARIES

How to reproduce:

1. Created a new filesystem (originally with bcachefs-tools last 1.3.x, updated to latest git version 1.6.4), --compression=zstd --acl --discard --metadata_replicas=2 --encryption, enter passphrase
2. Configured mkinitcpio.conf per above
3. Attempt to boot, asks for passphrase, I put in passphrase, enter it in, I see
   "fsck binary is version 1.6: btree_subvolume_children but filesystem is 1.3: rebalance_work and kernel is 1.4 member_seq, using kernel fsck"
   "bcachefs (someuid string): error requesting encryption key: ENOKEY"

Now if I use stable version of bcache-tools in arch (vs one in git, without the hooks, I see this when it tries to mount:
"mounting version 1.3: rebalance_work opts=ro,metadata_replicas=2,compression=zstd,degraded,fsck,fix_errors=yes,read_only
recovering from clean shutdown, journal seq 4302"
"Doing compatible version upgrade from 1.3: rebalance_work to 1.6: btree_subvolume_children"
running recovery passes: check_subvols,check_dirents"
... Along with a series of checks that say done"
and then finally see "Fatal error: No such device or address (os error 6)"
Failed to mount PARTLABEL=arch on real root

In emergency shell, If I try to mount manually with stable 1.6.4(mount /dev/sda4 /root) I get same error with os error 6, I get same error if using bcachefs mount. If I do bcachefs unlock /dev/sda4, I can then mount it. If using 1.6.4 (git version), I can not mount, even after unlock.

[dblaber]

FYI, I looked at how it was being mounted in the initrd script, it is doing something like this:
mount -t auto PARTLABEL=arch /new_root -o ro, which does not look that too out of the ordinary.