Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FS#443 - Error in kmodloader order? #5564

Closed
openwrt-bot opened this issue Jan 29, 2017 · 11 comments
Closed

FS#443 - Error in kmodloader order? #5564

openwrt-bot opened this issue Jan 29, 2017 · 11 comments
Labels

Comments

@openwrt-bot
Copy link

@openwrt-bot openwrt-bot commented Jan 29, 2017

avbohemen:

Supply the following if possible:

  • Device problem occurs on: TP-Link Archer C7 v2
  • Software versions of LEDE release, packages, etc. r3198-74ea99b
  • Steps to reproduce: compile and flash LEDE with Strongswan and pre-requisits included

There seems to be an error in kmod loader not loading every kmod in the correct order. From dmesg:

[ 12.107854] procd: - init -
[ 12.911515] kmodloader: loading kernel modules from /etc/modules.d/*
[ 12.960865] ntfs: driver 2.1.32 [Flags: R/O MODULE].
[ 12.982733] kmodloader: failed to find dependency xfrm_algo
[ 12.988753] kmodloader: failed to find dependency xfrm_algo
[ 12.994601] kmodloader: failed to find dependency xfrm_algo
[ 13.000645] kmodloader: failed to find dependency xfrm_algo
[ 13.006593] kmodloader: failed to find dependency xfrm_algo
[ 13.013877] tun: Universal TUN/TAP device driver, 1.6
[ 13.019046] tun: (C) 1999-2004 Max Krasnyansky maxk@qualcomm.com
[ 13.026621] kmodloader: failed to find dependency xfrm_algo
[ 13.032486] kmodloader: failed to find dependency xfrm_algo
[ 13.040602] kmodloader: failed to find dependency xfrm_algo
[ 13.046662] kmodloader: failed to find dependency xfrm_algo
[ 13.055631] kmodloader: failed to find dependency xfrm_algo
[ 13.061500] kmodloader: failed to find dependency xfrm_algo

This failure continues for some time, even after loading the wifi drivers:

[ 18.425281] kmodloader: failed to find dependency xfrm_algo
[ 18.431036] kmodloader: failed to find dependency xfrm_ipcomp
[ 18.445276] kmodloader: failed to find dependency xfrm_ipcomp
[ 18.451250] kmodloader: failed to find dependency xfrm_algo
[ 18.476726] kmodloader: failed to find dependency xfrm_algo

and finally:

[ 20.628579] kmodloader: 8 modules could not be probed
[ 20.633777] kmodloader: missing dependency xfrm_algo
[ 20.645261] kmodloader: - af_key - 1
[ 20.648960] kmodloader: missing dependency xfrm_algo
[ 20.654050] kmodloader: - ah4 - 1
[ 20.675290] kmodloader: missing dependency xfrm_algo
[ 20.680389] kmodloader: - ah6 - 1
[ 20.683854] kmodloader: missing dependency xfrm_algo
[ 20.705268] kmodloader: - esp4 - 1
[ 20.708782] kmodloader: missing dependency xfrm_algo
[ 20.713871] kmodloader: - esp6 - 1
[ 20.725308] kmodloader: missing dependency xfrm_ipcomp
[ 20.730585] kmodloader: - ipcomp - 1
[ 20.734271] kmodloader: missing dependency xfrm_ipcomp
[ 20.755292] kmodloader: - ipcomp6 - 1
[ 20.759168] kmodloader: missing dependency xfrm_algo
[ 20.764260] kmodloader: - xfrm_user - 1

The end result is that Strongswan will not start. This can be resolved by manually inserting the modules from /etc/modules.d/30-ipsec:

root@router ~# /etc/init.d/ipsec restart
Stopping strongSwan IPsec failed: starter is not running
Starting strongSwan 5.5.1 IPsec [starter]...
no netkey IPsec stack detected
failed to find a module named ipsec
no KLIPS IPsec stack detected
no known IPsec stack detected, ignoring!
root@router ~# insmod xfrm_algo
root@router ~# insmod xfrm_ipcomp
root@router ~# insmod xfrm_user
root@router ~# insmod af_key
root@router ~#
root@router ~# /etc/init.d/ipsec restart
Stopping strongSwan IPsec failed: starter is not running
Starting strongSwan 5.5.1 IPsec [starter]...
root@router ~#

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Feb 3, 2017

yousong:

I cannot reproduce the issue with malta-be target on r3282-50efd40 where ubox was supposed to receive only additional cosmetic changes compared to what's in r3198-74ea99b.

The following are my config before "make defconfig"

CONFIG_TARGET_malta=y CONFIG_TARGET_malta_be=y CONFIG_TARGET_malta_be_Default=y CONFIG_PACKAGE_strongswan-minimal=y

The reason that those "kmodloader: failed to find dependency xxx" repeated multiple times was that kmodloader tried to load modules and their dependencies in an iterative way and xxx could not be found on every pass (it should if xxx.ko existed in /lib/modules/x.x.x/ at kmodloader startup time).

Please try replacing those manual insmod commands with a single "kmodloader /etc/modules.d/" to see if the issue persists.

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Feb 3, 2017

avbohemen:

Today I did a new build (r3285-1a52d11) after doing "make dirclean" first to make sure my build environment was not dirty. However, the bug is still there, exactly the same.

The modules I inserted manually are all in /etc/modules.d/30-ipsec (as it should), and the .ko files of these modules are all in /lib/modules. So no problem there either.

If I do "kmodloader /etc/modules.d/30-ipsec", the following shows in syslog:

Fri Feb 3 20:23:31 2017 user.info kernel: [ 257.899125] kmodloader: loading kernel modules from /etc/modules.d/30-ipsec*
Fri Feb 3 20:23:31 2017 user.err kernel: [ 257.907731] kmodloader: failed to find dependency xfrm_algo
Fri Feb 3 20:23:31 2017 user.err kernel: [ 257.913634] kmodloader: failed to find dependency xfrm_algo
Fri Feb 3 20:23:31 2017 user.err kernel: [ 257.921721] kmodloader: failed to find dependency xfrm_algo
Fri Feb 3 20:23:31 2017 user.err kernel: [ 257.928461] kmodloader: failed to find dependency xfrm_algo
Fri Feb 3 20:23:31 2017 user.err kernel: [ 257.934605] kmodloader: failed to find dependency xfrm_algo
Fri Feb 3 20:23:31 2017 user.err kernel: [ 257.941722] kmodloader: 2 modules could not be probed
Fri Feb 3 20:23:31 2017 user.err kernel: [ 257.947756] kmodloader: missing dependency xfrm_algo
Fri Feb 3 20:23:31 2017 user.err kernel: [ 257.952853] kmodloader: - af_key - 1
Fri Feb 3 20:23:31 2017 user.err kernel: [ 257.957867] kmodloader: missing dependency xfrm_algo
Fri Feb 3 20:23:31 2017 user.err kernel: [ 257.962964] kmodloader: - xfrm_user - 1

So, kmodloader cannot load xfrm_algo, while insmod can? Is there anything else I can test?

By the way, /etc/modules.d/30-ipsec has the entries in the correct order:

root@router ~# cat /etc/modules.d/30-ipsec
xfrm_algo
xfrm_ipcomp
xfrm_user
af_key

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Feb 4, 2017

yousong:

Order is not an issue with kmodloader as it will try to load each modules' dependencies and then the module itself in an iterative way until the end of a final pass where no module was loaded.

Maybe output of "modinfo xfrm_alog" can help. They share the same binary and "modinfo" should also fail if "kmodloader" does not work.

LD_LIBRARY_PATH is another thing to check. It will affect kernel module search path.

kmodloader is part of the ubox package. You can consider adding a few printf statement to the code and compile, scp the built binary to the board and see how it works

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Feb 4, 2017

avbohemen:

Thanks for the info. Apparently, LD_LIBRARY_PATH is empty on my system, and that accounts for the following:

root@router ~# echo $LD_LIBRARY_PATH

root@router ~# modinfo xfrm_algo
cannot find module - xfrm_algo
root@router ~# modinfo xfrm_ipcomp
cannot find module - xfrm_ipcomp
root@router ~# modinfo xfrm_user
module: /lib/modules/4.4.46/xfrm_user.ko
alias: net-pf-16-proto-6
license: GPL
depends: xfrm_algo
root@router ~# modinfo af_key
module: /lib/modules/4.4.46/af_key.ko
alias: net-pf-15
license: GPL
depends: xfrm_algo

Things start to work when I set LD_LIBRARY_PATH to at least /lib:

root@router ~# export LD_LIBRARY_PATH=/lib
root@router ~# modinfo xfrm_algo
module: /lib/modules/4.4.46/xfrm_algo.ko
license: GPL
depends:
root@router ~# modinfo xfrm_ipcomp
module: /lib/modules/4.4.46/xfrm_ipcomp.ko
license: GPL
depends: xfrm_algo

Then the question is, where should LD_LIBRARY_PATH be set so kmodloader picks it up at boot, and why is it not set automatically?

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Feb 4, 2017

yousong:

kmodloader should work out of box without setting LD_LIBRARY_PATH and /lib is supposed to be already included as the last search dir as a fallback.

Please try debuging the kmodloader program itself. The culprit should be easy to find from now on.

btw., an env var set but empty is different from it being unset.

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Feb 12, 2017

bjonglez:

Similar bug report: FS#493

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Feb 12, 2017

avbohemen:

I have been digging through the git rep of ubox, and I have built every version from the last 2 months until the version which is now referenced in the LEDE source.git. This is what I found (the date is the commit date):

ubox_2016-12-04-acc48b55-1_mips_24kc.ipk - xfrm_algo loads fine
ubox_2016-12-19-db070f15-1_mips_24kc.ipk - xfrm_algo loads fine
ubox_2016-12-19-8488bb5b-1_mips_24kc.ipk - xfrm_algo loads fine
ubox_2017-01-15-eacc4261-1_mips_24kc.ipk - xfrm_algo loads fine
ubox_2017-01-15-46a4b5f7-1_mips_24kc.ipk - xfrm_algo loads fine
ubox_2017-01-15-a62c946e-1_mips_24kc.ipk - router failed to boot - needed failsafe and firstboot to get back.
ubox_2017-01-15-93714117-1_mips_24kc.ipk - router failed to boot - needed failsafe and firstboot to get back.
ubox_2017-01-15-6e3c6dcf-1_mips_24kc.ipk - "kmodloader: failed to find dependency xfrm_algo"
ubox_2017-01-15-14839f0a-1_mips_24kc.ipk - "kmodloader: failed to find dependency xfrm_algo"
ubox_2017-02-01-f8d3d167-1_mips_24kc.ipk - "kmodloader: failed to find dependency xfrm_algo"
ubox_2017-02-01-ac2d43e7-1_mips_24kc.ipk - "kmodloader: failed to find dependency xfrm_algo"

So, the bug lies in one of three commits:
a62c946ecdced9468ad16738325ee322029b0476 kmodloader: modprobe: skip possible command line arguments
9371411715c8dd882f1d65a712f1f38b6d38d01f kmodloader: fix out-of-bound access when parsing .modinfo
6e3c6dcf922e30282d2c49a643253e58fffab362 kmodloader: add module alias awareness

That is as close as I can get, I am not a debugger :) I just have some git knowlegde and I like to test LEDE.

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Feb 21, 2017

nbd:

Please try the latest version from my staging tree at
https://git.lede-project.org/?p=lede/nbd/staging.git;a=summary

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Feb 21, 2017

avbohemen:

It looks like it is correct again :) All modules load, and ipsec starts normally.

root@router ~# logread|grep kmodloader
Tue Feb 21 15:59:34 2017 user.info kernel: [ 6.249906] kmodloader: loading kernel modules from /etc/modules-boot.d/*
Tue Feb 21 15:59:34 2017 user.info kernel: [ 6.655337] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
Tue Feb 21 15:59:34 2017 user.info kernel: [ 10.160934] kmodloader: loading kernel modules from //etc/modules-boot.d/*
Tue Feb 21 15:59:34 2017 user.info kernel: [ 10.169871] kmodloader: done loading kernel modules from //etc/modules-boot.d/*
Tue Feb 21 15:59:34 2017 user.info kernel: [ 10.746378] kmodloader: loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Tue Feb 21 15:59:34 2017 user.info kernel: [ 10.759585] kmodloader: done loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Tue Feb 21 15:59:34 2017 user.info kernel: [ 13.372210] kmodloader: loading kernel modules from /etc/modules.d/*
Tue Feb 21 15:59:34 2017 user.info kernel: [ 17.325986] kmodloader: done loading kernel modules from /etc/modules.d/*
root@router ~# opkg list-installed|grep ^ubox
ubox - 2017-02-21-c553354d-1

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Feb 21, 2017

nbd:

Fixed in r3552-2a1ee042cd, thanks for testing.

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Feb 21, 2017

kpv:

I'd also like to happily report that with today's ubox - 2017-02-21-c553354d-1 upgrade, the kmodloader errors under LEDE trunk (x86) seem solved. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant