udevadm: Sporadic timed out waiting for device on boot from emmc

[forca84]

issue_4gh.log

Sometimes, upon reboot, I see this timeout error.
In particular, if I enter the emergency mode and do "mount -a", the partitions in question gets mounted properly.

The issue is very similar to these both:
#13775
#2257

We have already tried a newer version (255.21) of systemd from poky 5.0.12, result is same same.

As already mentioned in other linked issues, we have checked all needed kernel-configs, required by systemd and other suggestions from there, nothing helped.

Any ideas?

I could already catch the issue with a kernel cmd-line: systemd.log_level=debug and attach the log-file here.

systemd version the issue has been seen with

255.18

Used distribution

yocto scarthgap 5.0.9

Linux kernel version used

6.6.58-ti-01497

CPU architectures issue was seen on

aarch64

Component

udevadm

Expected behaviour you didn't see

No (udevadm: Sporadic timed out waiting for device on boot) error

Unexpected behaviour you saw

Sep 04 17:07:02 target-1 systemd[1]: dev-mmcblk0p1.device: Job dev-mmcblk0p1.device/start timed out.
Sep 04 17:07:02 target-1 systemd[1]: dev-mmcblk0p1.device: Job 75 dev-mmcblk0p1.device/start finished, result=timeout
Sep 04 17:07:02 target-1 systemd[1]: Timed out waiting for device /dev/mmcblk0p1.

Sep 04 17:07:02 target-1 systemd[1]: dev-mmcblk0p4.device: Job dev-mmcblk0p4.device/start timed out.
Sep 04 17:07:02 target-1 systemd[1]: dev-mmcblk0p4.device: Job 69 dev-mmcblk0p4.device/start finished, result=timeout
Sep 04 17:07:02 target-1 systemd[1]: Timed out waiting for device /dev/mmcblk0p4.

Steps to reproduce the problem

Reboot the target in a loop

Additional program output to the terminal or log subsystem illustrating the issue

[forca84]

fstab used on target:

# stock fstab - you probably want to override this with a machine specific one

/dev/root            /                    auto       defaults,noatime              1  1
proc                 /proc                proc       defaults              0  0
devpts               /dev/pts             devpts     mode=0620,ptmxmode=0666,gid=5      0  0
tmpfs                /run                 tmpfs      mode=0755,nodev,nosuid,strictatime 0  0
tmpfs                /var/volatile        tmpfs      defaults              0  0

# uncomment this if your device has a SD/MMC/Transflash slot
#/dev/mmcblk0p1       /media/card          auto       defaults,sync,noauto  0  0

# Where the U-Boot environment resides; for devices with SD card support ONLY!
/dev/mmcblk0p1       /uboot               auto       defaults,sync         0  2
/dev/mmcblk0p4       /data                auto       defaults              0  2

some lines here from attached log-file seem a bit curious for us:

Sep 04 17:04:03 target-1 udevadm[104]: sd-device-enumerator: Failed to scan /sys/class: No such process
..
Sep 04 17:04:03 target-1 udevadm[104]: Failed to scan devices and subsystems: No such process
..
Sep 04 17:04:03 target-1 systemd[1]: Child 104 (udevadm) died (code=exited, status=1/FAILURE)
..

[yuwata]

Huh? /sys seems to be in a spurious state. In the emergency state, please check if sysfs is mounted on /sys and devices are listed in /sys.

[forca84]

it seems to be mounted and having something inside:

sh-5.2# mount
/dev/mmcblk0p2 on / type ext4 (rw,noatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=375600k,nr_inodes=93900,mode=755)
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=176980k,nr_inodes=819200,mode=755)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,nosuid,nodev,relatime,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,nr_inodes=1048576)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /var/volatile type tmpfs (rw,relatime)
sh-5.2# cd /sys/devices/
sh-5.2# ls -l
total 0
drwxr-xr-x  6 root root 0 Sep  7 17:06 armv8_cortex_a53
drwxr-xr-x  3 root root 0 Sep  7 17:06 breakpoint
drwxr-xr-x  3 root root 0 Sep  7 17:06 optee-ta-ab7a617c-b8e7-4d8f-8301-d09b61036b64
drwxr-xr-x 22 root root 0 Sep  7 17:06 platform
drwxr-xr-x  3 root root 0 Sep  7 17:06 soc0
drwxr-xr-x  3 root root 0 Sep  7 17:06 software
drwxr-xr-x  7 root root 0 Sep  7 17:06 system
drwxr-xr-x 19 root root 0 Sep  7 17:06 virtual
sh-5.2# 

additionally, here I do attach the systemd-analyze-plot in this state

[yuwata]

How about /sys/class ?

[yuwata]

Which libc implementation do you use? glibc or musl ?

[forca84]

How about /sys/class ?

looks healthy more or less:

sh-5.2# ls -l /sys/class/
total 0
drwxr-xr-x 2 root root 0 Sep  7 17:06 bdi
drwxr-xr-x 2 root root 0 Sep  7 17:06 block
drwxr-xr-x 2 root root 0 Sep  7 17:06 bsg
drwxr-xr-x 2 root root 0 Sep  7 17:06 chromeos
drwxr-xr-x 2 root root 0 Sep  7 17:06 devcoredump
drwxr-xr-x 2 root root 0 Sep  7 17:06 devfreq
drwxr-xr-x 2 root root 0 Sep  7 17:06 devlink
drwxr-xr-x 2 root root 0 Sep  7 17:06 dma
drwxr-xr-x 2 root root 0 Sep  7 17:06 dma_heap
drwxr-xr-x 2 root root 0 Sep  7 17:06 drm
drwxr-xr-x 2 root root 0 Sep  7 17:06 extcon
drwxr-xr-x 2 root root 0 Sep  7 17:06 fpga_manager
drwxr-xr-x 2 root root 0 Sep  7 17:06 hwmon
drwxr-xr-x 2 root root 0 Sep  7 17:06 i2c-adapter
drwxr-xr-x 2 root root 0 Sep  7 17:06 i2c-dev
drwxr-xr-x 2 root root 0 Sep  7 17:06 input
drwxr-xr-x 2 root root 0 Sep  7 17:06 iommu
drwxr-xr-x 2 root root 0 Sep  7 17:06 k3_udma_glue_chan
drwxr-xr-x 2 root root 0 Sep  7 17:06 leds
drwxr-xr-x 2 root root 0 Sep  7 17:06 mdio_bus
drwxr-xr-x 2 root root 0 Sep  7 17:06 mem
drwxr-xr-x 2 root root 0 Sep  7 17:06 misc
drwxr-xr-x 2 root root 0 Sep  7 17:06 mmc_host
drwxr-xr-x 2 root root 0 Sep  7 17:06 mtd
drwxr-xr-x 2 root root 0 Sep  7 17:06 mux
drwxr-xr-x 2 root root 0 Sep  7 17:06 net
drwxr-xr-x 2 root root 0 Sep  7 17:06 pci_bus
drwxr-xr-x 2 root root 0 Sep  7 17:06 pci_epc
drwxr-xr-x 2 root root 0 Sep  7 17:06 phy
drwxr-xr-x 2 root root 0 Sep  7 17:06 power_supply
drwxr-xr-x 2 root root 0 Sep  7 17:06 ppp
drwxr-xr-x 2 root root 0 Sep  7 17:06 pps
drwxr-xr-x 2 root root 0 Sep  7 17:06 ptp
drwxr-xr-x 2 root root 0 Sep  7 17:06 regulator
drwxr-xr-x 2 root root 0 Sep  7 17:06 remoteproc
drwxr-xr-x 2 root root 0 Sep  7 17:06 rpmsg
drwxr-xr-x 2 root root 0 Sep  7 17:06 rtc
drwxr-xr-x 2 root root 0 Sep  7 17:06 scsi_device
drwxr-xr-x 2 root root 0 Sep  7 17:06 scsi_disk
drwxr-xr-x 2 root root 0 Sep  7 17:06 scsi_host
drwxr-xr-x 2 root root 0 Sep  7 17:06 sound
drwxr-xr-x 2 root root 0 Sep  7 17:06 spi_master
drwxr-xr-x 2 root root 0 Sep  7 17:06 spidev
drwxr-xr-x 2 root root 0 Sep  7 17:06 tee
drwxr-xr-x 2 root root 0 Sep  7 17:06 thermal
drwxr-xr-x 2 root root 0 Sep  7 17:06 tpm
drwxr-xr-x 2 root root 0 Sep  7 17:06 tpmrm
drwxr-xr-x 2 root root 0 Sep  7 17:06 tty
drwxr-xr-x 2 root root 0 Sep  7 17:06 ubi
drwxr-xr-x 2 root root 0 Sep  7 17:06 udc
drwxr-xr-x 2 root root 0 Sep  7 17:06 uio
drwxr-xr-x 2 root root 0 Sep  7 17:06 usb_role
drwxr-xr-x 2 root root 0 Sep  7 17:06 usbmisc
drwxr-xr-x 2 root root 0 Sep  7 17:06 vc
drwxr-xr-x 2 root root 0 Sep  7 17:06 vfio
drwxr-xr-x 2 root root 0 Sep  7 17:06 vfio-dev
drwxr-xr-x 2 root root 0 Sep  7 17:06 virtio-ports
drwxr-xr-x 2 root root 0 Sep  7 17:06 vtconsole
drwxr-xr-x 2 root root 0 Sep  7 17:06 wakeup
drwxr-xr-x 2 root root 0 Sep  7 17:06 watchdog
drwxr-xr-x 2 root root 0 Sep  7 17:06 xt_idletimer
sh-5.2# 

[forca84]

Which libc implementation do you use? glibc or musl ?

glibc

[yuwata]

Hmm, it seems some spurious devices cannot be enumerated.
Is the failure can be reproducible even after boot or in emergency mode?
Please try to run

$ udevadm trigger --type=all --dry-run --verbose

If it fails, then could try to find which devices refuse to be enumerated? E.g. by something like the following:

$ udevadm info --tree /sys/bus
$ udevadm info --tree /sys/module

[forca84]

ok, all 3 calls worked and produced many output. Please find here attached 3 logs for it.

udevadm_trigger_all.log

udevadm_info_sys_bus.log

udevadm_info_sys_module.log

somehow nothing in /dev/disk/ aswell:

sh-5.2#  ls -l /dev/disk/
ls: cannot access '/dev/disk/': No such file or directory
sh-5.2# 

[poettering]

/dev/root / auto defaults,noatime 1 1
proc /proc proc defaults 0 0
devpts /dev/pts devpts mode=0620,ptmxmode=0666,gid=5 0 0
tmpfs /run tmpfs mode=0755,nodev,nosuid,strictatime 0 0

Please remove these four. They are redundant at best on systemd systems, and problematic at worst.