Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FS#2904 - usb-storage fails to load in pre-init #7686

Open
openwrt-bot opened this issue Mar 16, 2020 · 8 comments
Open

FS#2904 - usb-storage fails to load in pre-init #7686

openwrt-bot opened this issue Mar 16, 2020 · 8 comments
Labels
flyspray release/19.07

Comments

@openwrt-bot
Copy link

@openwrt-bot openwrt-bot commented Mar 16, 2020

higuita:

As per forum post:

https://forum.openwrt.org/t/usb-storage-fail-to-load-in-preinit/54903

Basically, installed openwrt, added external root, all looks correctly configured, but after reboot external root fails to mount.
searching in the logs we can see that usb-storage is only loaded after pre-init, but it is listed in /etc/modprobe-boot.d/ ... checking /rom/etc/modules-boot.d/ it is not there.

Either something is missing in the https://openwrt.org/docs/guide-user/additional-software/extroot_configuration to add the usb-storage to the pre-init, or something is broken that makes pre-init fail to load usb-storage when it should.

The same setup worked fine with a older openwrt version (15.01 IIRC)

Supply the following if possible:

  • Device problem occurs on
    asus wl-500w

  • Software versions of OpenWrt/LEDE release, packages, etc.
    OpenWrt 19.07.1, r10911-c155900f66

  • Steps to reproduce
    Install openwrt
    add external root
    manually mount to confirm setup
    reboot
    external root not mounted nor switched to, logs show that usb-storage is only loaded after normal root is mounted
    manual mount still works

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Mar 16, 2020

por:

So /overlay is still on internal storage.

Did you recreate the extroot from the rootfs_data ?
(assuming the external storage is a suitable filesystem on /dev/sda1)

mount /dev/sda1 /mnt
cd /mnt
tar cf - -C /overlay . |tar x

And, assuming the extroot has been added into /etc/config/fstab, did you remove the old marker from the external storage before trying to boot with extroot ?
rm /mnt/etc/.extroot-uuid

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Mar 16, 2020

higuita:

yes, done all those steps, i just redone then and same result
notice the dmesg:

[ 41.505538] block: extroot: cannot find device sda1
(...)
[ 42.888687] procd: - init -
(...)
[ 47.273106] usb-storage 1-2:1.0: USB Mass Storage device detected

extroot fail to work due the missing usb_storage, that is being loaded only on the init step, not in the pre-init

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Mar 16, 2020

por:

Okay, but if your analysis is correct then extroot would not run anywhere, but it does here (ipq806x, OpenWrt 19.07.2). Or d I miss something.

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Mar 17, 2020

higuita:

I agree with you, this is weird.
Either this is simply a documentation bug and i'm missing some step, or something that may be broken on some systems only. I'm using a asus wl-500w, so something related to the image openwrt-19.07.1-brcm47xx-legacy-asus-wl-500w ... maybe "legacy" related?

You are also using a usb-storage device, correct?
what is your output for this:

find /rom/etc/modules-boot.d/ /rom/etc/modules.d/ /etc/modules-boot.d/ /etc/modules.d/ -name usb*

for me, only the /etc/modules*.d/ i have the usb-storage, the /rom do not have anything for usb-storage

and how about the /etc/config/fstab? mine is this one:

config global option anon_swap '0' option anon_mount '0' option auto_swap '1' option auto_mount '1' option delay_root '15' option check_fs '0'

config mount
option target '/mnt/sda1'
option uuid '04f05f45-f506-4465-9ecc-8d836d0588b6'
option enabled '0'

config mount 'rwm'
option device '/dev/mtdblock5'
option target '/rwm'

config mount 'overlay'
option uuid '04f05f45-f506-4465-9ecc-8d836d0588b6'
option target '/overlay'

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Mar 21, 2020

por:

Your has two entries for the external filesystem (same UUID). Removing the first one that's disabled may help.

Also some options seem missing. Assuming your extroot has ext4 fs, these are:
option fstype 'ext4'
option enabled_fsck '1'

The initially installed modules in /rom are determined by the DEVICE_PACKAGES in the makefile for the device, in case of the wl-500w those are:
kmod-b43 kmod-usb-uhci kmod-usb2-pci

Assuming ext4 for the extroot fs, extra packages needed would be:
kmod-usb-storage kmod-usb2 kmod-fs-ext4 block-mount

Using imagebuilder is recommended as it allows to include all needed packages in /rom which saves space in the initial overlay fs (non-extroot) as the fs for /rom (squashfs) is denser than the fs used for the internal overlay (jffs).

Good luck.

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Apr 3, 2020

higuita:

i have removed the first /dev/sda1 config, added the ext4 and fsck to the last entry, confirmed that i have all those packages installed... rebooted and still same result, no extroot mounted, usb_storage is only loaded in init step, after giving up of mounting extroot.

i do not know if there is some bug in pre-init scripts that fail to check the /etc/modules-boot.d/usb-storage or some magic inside kmodloader (i strace and it never even looks to the /etc/modules-boot.d, but i do not know if during boot it does different steps). Can you please check if in your system, usb_storage is loaded in pre-init?

i will probably try the imagebuilder, some years ago i failed to build a working image, but lets try again.

thanks for the help

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Apr 3, 2020

por:

To be sure:

  • the contents of the internal storage mounted on /overlay has been copied to the external fs and
  • a pre-existing etc/.extroot-uuid has been removed from the external fs

Once the external fs is mounted on /overlay, etc/.extroot-uuid is created on that fs and the file contains the UUID of the fs mounted on /rom. If the UUID in etc/.extroot-uuid on the external fs does not match that of the fs mounted on /rom, then the internal overlay storage is mounted.

When still having problems, please show actual contents of /etc/config/fstab, and possibly etc/.extroot-uuid on the external fs and the output of the "block info" command.

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Apr 4, 2020

higuita:

Yes, content was been copied in 2 different attempts, with tar and with cp (the old2 was the first attempt and inside also have another subfolder with my previous openwrt extroot, so i can copy some configs to the new setup, when it works)

no .extroot-uuid is found anywhere

root@submarino:~# df -hT Filesystem Type Size Used Available Use% Mounted on /dev/root squashfs 2.6M 2.6M 0 100% /rom tmpfs tmpfs 13.8M 60.0K 13.7M 0% /tmp /dev/mtdblock5 jffs2 3.8M 1000.0K 2.8M 26% /overlay overlayfs:/overlay overlay 3.8M 1000.0K 2.8M 26% / tmpfs tmpfs 512.0K 0 512.0K 0% /dev /dev/sda1 ext4 233.2M 22.7M 194.3M 10% /mnt root@submarino:~# ls -la /mnt/ /overlay/ /mnt/: drwxr-xr-x 5 root root 1024 Feb 9 02:03 . drwxr-xr-x 1 root root 0 Mar 16 2020 .. lrwxrwxrwx 1 root root 1 Feb 9 02:03 .fs_state -> 2 drwxr-xr-x 10 root root 1024 Feb 9 02:03 old2 drwxr-xr-x 7 root root 1024 Feb 9 02:03 upper drwxr-xr-x 3 root root 1024 Feb 9 02:03 work

/overlay/:
drwxr-xr-x 5 root root 0 Jan 1 1970 .
drwxr-xr-x 1 root root 0 Mar 16 2020 ..
lrwxrwxrwx 1 root root 1 Feb 2 23:35 .fs_state -> 2
drwxr-xr-x 7 root root 0 Mar 16 2020 upper
drwxr-xr-x 3 root root 0 Jan 1 1970 work
root@submarino:# find / -name .extroot-uuid
root@submarino:
#
root@submarino:~# cat /etc/config/fstab

config global
option anon_swap '0'
option anon_mount '0'
option auto_swap '1'
option auto_mount '1'
option delay_root '15'
option check_fs '0'

config mount 'rwm'
option device '/dev/mtdblock5'
option target '/rwm'

config mount 'overlay'
option uuid '04f05f45-f506-4465-9ecc-8d836d0588b6'
option target '/overlay'
option fstype 'ext4'
option enabled_fsck '1'

root@submarino:~# block info
/dev/mtdblock4: UUID="7eeba3f8-70b2ccf1-634a5d1c-24be005d" VERSION="4.0" MOUNT="/rom" TYPE="squashfs"
/dev/mtdblock5: MOUNT="/overlay" TYPE="jffs2"
/dev/sda1: UUID="04f05f45-f506-4465-9ecc-8d836d0588b6" VERSION="1.0" MOUNT="/mnt" TYPE="ext4"

dmesg:
[ 4.263493] hub 3-0:1.0: USB hub found
[ 4.269334] hub 3-0:1.0: 2 ports detected
[ 4.286376] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
[ 4.305484] init: - preinit -
[ 4.584140] usb 1-2: new high-speed USB device number 2 using ehci-pci
[ 4.914701] random: procd: uninitialized urandom read (4 bytes read)
[ 6.148297] random: jshn: uninitialized urandom read (4 bytes read)
[ 6.332918] random: jshn: uninitialized urandom read (4 bytes read)
[ 6.565122] random: jshn: uninitialized urandom read (4 bytes read)
[ 7.195246] b44 ssb0:0 eth0: Link is up at 100 Mbps, half duplex
[ 7.201361] b44 ssb0:0 eth0: Flow control is off for TX and off for RX
[ 7.211354] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 7.220610] IPv6: ADDRCONF(NETDEV_UP): eth0.1: link is not ready
[ 7.227698] IPv6: ADDRCONF(NETDEV_CHANGE): eth0.1: link becomes ready
[ 8.243993] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 8.251716] b44 ssb0:0 eth0: Link is Up - 100Mbps/Full - flow control off
[ 10.930924] jffs2: notice: (407) jffs2_build_xattr_subsystem: complete building xattr subsystem, 49 of xdatum (3 unchecked, 46 orphan) and 107 of xref (47 dead, 0 orphan) found.
[ 10.952986] mount_root: loading kmods from internal overlay
[ 10.982820] kmodloader: failed to open /tmp/overlay/upper/lib/modules/4.14.167/b43.ko
[ 10.991633] kmodloader: failed to open /tmp/overlay/upper/lib/modules/4.14.167/cfg80211.ko
[ 11.000853] kmodloader: failed to open /tmp/overlay/upper/lib/modules/4.14.167/compat.ko
[ 11.090934] kmodloader: failed to open /tmp/overlay/upper/lib/modules/4.14.167/mac80211.ko
[ 11.105972] kmodloader: failed to open /tmp/overlay/upper/lib/modules/4.14.167/ppp_async.ko
[ 11.115301] kmodloader: failed to open /tmp/overlay/upper/lib/modules/4.14.167/ppp_generic.ko
[ 11.124812] kmodloader: failed to open /tmp/overlay/upper/lib/modules/4.14.167/pppoe.ko
[ 11.133953] kmodloader: failed to open /tmp/overlay/upper/lib/modules/4.14.167/pppox.ko
[ 11.235164] mount_root: failed to launch kmodloader from internal overlay
[ 11.391388] block: attempting to load /tmp/jffs_cfg/upper/etc/config/fstab
[ 11.403586] block: extroot: device not present, retrying in 15 seconds
[ 26.480534] block: extroot: cannot find device with UUID 04f05f45-f506-4465-9ecc-8d836d0588b6
[ 26.491973] mount_root: switching to jffs2 overlay
[ 26.554260] overlayfs: upper fs does not support tmpfile.
[ 26.572590] urandom-seed: Seeding with /etc/urandom.seed
[ 26.719184] b44 ssb0:0 eth0: powering down PHY
[ 26.778444] procd: - early -
[ 26.781797] procd: - watchdog -
[ 26.964069] b44 ssb0:0 eth0: Link is Down
[ 27.644848] procd: - watchdog -
[ 27.649449] procd: - ubus -
[ 27.808520] random: ubusd: uninitialized urandom read (4 bytes read)
[ 27.828997] random: ubusd: uninitialized urandom read (4 bytes read)
[ 27.844500] random: ubusd: uninitialized urandom read (4 bytes read)
[ 27.872200] procd: - init -
[ 30.324110] kmodloader: loading kernel modules from /etc/modules.d/*
[ 30.581077] urngd: v1.0.2 started.
[ 31.283602] random: crng init done
[ 31.287233] random: 3 urandom warning(s) missed due to ratelimiting
[ 31.661511] SCSI subsystem initialized
[ 31.838084] ip6_tables: (C) 2000-2006 Netfilter Core Team
[ 31.908246] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 31.947042] nf_conntrack version 0.5.0 (1024 buckets, 4096 max)
[ 32.255561] usb-storage 1-2:1.0: USB Mass Storage device detected
[ 32.285212] scsi host0: usb-storage 1-2:1.0
[ 32.291388] usbcore: registered new interface driver usb-storage
[ 32.393673] xt_time: kernel timezone is -0000
[ 32.444459] kmodloader: done loading kernel modules from /etc/modules.d/*
[ 33.366178] scsi 0:0:0:0: Direct-Access VBTM Store 'n' Go 1.04 PQ: 0 ANSI: 0 CCS
[ 33.384774] sd 0:0:0:0: [sda] 501760 512-byte logical blocks: (257 MB/245 MiB)
[ 33.405928] sd 0:0:0:0: [sda] Write Protect is off
[ 33.410951] sd 0:0:0:0: [sda] Mode Sense: 23 00 00 00
[ 33.424002] sd 0:0:0:0: [sda] No Caching mode page found
[ 33.429533] sd 0:0:0:0: [sda] Assuming drive cache: write through
[ 33.453193] sda: sda1
[ 33.479422] sd 0:0:0:0: [sda] Attached SCSI removable disk
[ 44.944060] b44 ssb0:0 eth0: Link is up at 100 Mbps, full duplex
[ 44.950178] b44 ssb0:0 eth0: Flow control is off for TX and off for RX
[ 44.958684] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 44.964880] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 46.004324] b44 ssb0:0 eth0: Link is Up - 100Mbps/Full - flow control off
[ 49.647074] b44 ssb0:0 eth0: powering down PHY
[ 49.752742] b44 ssb0:0 eth0: Link is up at 100 Mbps, full duplex
[ 49.759018] b44 ssb0:0 eth0: Flow control is off for TX and off for RX

The missing b43 and ppp modules are ok, i removed the wifi card of this router and i do not use adsl

@aparcar aparcar added the release/19.07 label Feb 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
flyspray release/19.07
Projects
None yet
Development

No branches or pull requests

2 participants