Weird folder structure after OBP : /etc is empy , files are now in modoverlay ? #957

Dark4Codrutz · 2020-05-08T15:26:51Z

The root file system's /etc folder is now empty. I can see (some) files are present in /modoverla/bank_mod/etc but doesn't seem to be a complete list of all the files I used to see.

Main concern :
For example, I have a samba in modoverlay's init.d that points to /etc/init.d/nqe
The fact that exploring /etc with WinSCP gives an empty folder makes me think that the symlink is broken (and hell went loose) but upon closer inspection by opening the symlink I can see file's content.

Q: Is this by design ?
I would like to be able to explore the contents of /etc , and I don't know what is happening here.
I think same principle applies to other directories that I'm not aware of at this moment.

EDIT
The issue was solved by rebooting once more after OBP

Dark4Codrutz · 2020-05-08T15:28:05Z

This happened after the reboot the GUI asked for regarding OBP

Ansuel · 2020-05-08T15:31:02Z

Sorry can you post a screen of content of /etc /modoverlay/bank_mod/etc and the output of command mount? Also this is on what router?

Dark4Codrutz · 2020-05-08T15:38:09Z

This is on DGA4132

root@modemtim:/etc# mount
/dev/root on /rom type squashfs (ro,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,noatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
/dev/mtdblock2 on /overlay type jffs2 (rw,relatime,compr=zlib)
overlay on /saferoot type overlay (rw,relatime,lowerdir=/,upperdir=/overlay/bank_2,workdir=/overlay/work)
/dev/mtdblock3 on /modoverlay type jffs2 (rw,relatime,compr=zlib)
overlay on / type overlay (rw,relatime,lowerdir=/,upperdir=/modoverlay/bank_mod,workdir=/modoverlay/work)
tmpfs on /dev type tmpfs (rw,nosuid,relatime,mode=0755,size=512K)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)
cgroup on /cgroups/cpumemblk type cgroup (rw,relatime,cpu,blkio,memory)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)
mvfs on /tmp/mvfs type fuse.mvfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
mountd(pid7213) on /tmp/run/mountd type autofs (rw,relatime,fd=5,pgrp=7211,timeout=600,minproto=5,maxproto=5,indirect)
/dev/sda1 on /tmp/run/mountd/sda1 type ufsd (rw,noatime,nls=utf8,uid=0,gid=0,fmask=0,dmask=0,nocase,sparse,force)

root@modemtim:/modoverlay/bank_mod/etc# ls
banner                ethers                lcmd                  ppp                   sysctl.d
board                 fileprofiler          ledfw                 printcap              tod
boards                firewall.user         log                   random-seed           transformer.db
christmas_tree.sh     fwdassist.sh          luci-uploads          rc.d                  transformer.db-wal
clash                 group                 modgui_scripts        samba                 uci-defaults
config                hotplug.d             nginx                 shadow                wansensing
crontabs              init.d                opkg                  shadow-               xupnpd
cups                  inittab               opkg.conf             slicstate
cwmpd.db              ispconfig             parameter_conversion  sms.db
ddns                  lasdb.db              passwd                ssl
dropbear              lastaccess            passwd-               sysctl.conf

some of the above are symlinks, for example directory "board" links to "/etc/boards/VBNT-S", I can explore boards but I can't explore /etc/...

/etc is empty while exploring (ssh and WinSCP, both)

Ansuel · 2020-05-08T15:41:47Z

can you post also a logread (or send me a private email with it at ansuelsmth@gmail.com)
also post content of /overlay/bank_2/etc

Ansuel · 2020-05-08T15:42:39Z

@LuKePicci any idea about this?

Dark4Codrutz · 2020-05-08T15:45:00Z

root@modemtim:/overlay/bank_2/etc# ls
init.d  rc.d
root@modemtim:/overlay/bank_2/etc/init.d# ls
rootdevice
root@modemtim:/overlay/bank_2/etc/rc.d# ls -l
lrwxrwxrwx    1 root     root            20 Jan  1  1970 S94rootdevice -> ../init.d/rootdevice

Dark4Codrutz · 2020-05-08T15:52:29Z

for logread a Debug zip I think will do the same..
OBP_DebugHelper2020-05-08-1745.tar.gz

A note on this: using the option from the gui took a while:
In top I saw find /tmp -type f -name DebugHelper* showing up and disappearing multiple times while it was "Generating debug in background...", looked like it was looping.
The file was there and I managed to copy it with WinSCP, after a while the GUI finished and offered to save the file too. Files are identical, uploaded the latter though.

Dark4Codrutz · 2020-05-08T15:54:20Z

OBP_logread.log

Here is also a full logread
Edit: reupload, sanitized ip

Dark4Codrutz · 2020-05-08T15:59:22Z

Hope it is useful because after I hit reboot , when the gui was suggesting it, it rebooted twice.
I presume after the first reboot OBP was applied then it rebooted a second time... to take effect of the changes?

Ansuel · 2020-05-08T16:00:42Z

touch /etc/test
ls -la /etc
ls -la /modoverlay/bank_mod/etc

@Dark4Codrutz yes it's all correct what is not correct is the fact that you have /etc empty instead of the /rom file + the file present in /modoverlay

Dark4Codrutz · 2020-05-08T16:02:41Z

root@modemtim:~# cd /etc
root@modemtim:/etc# ls -la
root@modemtim:/etc# touch test
root@modemtim:/etc# ls -la
root@modemtim:/etc# ls -la /modoverlay/bank_mod/etc
drwxr-xr-x   30 root     root             0 May  8 18:01 .
drwxr-xr-x   17 root     root             0 Jan  1  1970 ..
-rw-r--r--    1 root     root          2862 Jan  1  1970 banner
lrwxrwxrwx    1 root     root            18 Jan  1  1970 board -> /etc/boards/VBNT-S
drwxr-xr-x    3 root     root             0 Oct 16  2019 boards
-rwxr-xr-x    1 root     root           861 May  7 22:32 christmas_tree.sh
drwxr-xr-x    2 root     root             0 Apr  4 15:54 clash
drwxr-xr-x    2 root     root             0 May  8 17:05 config
drwxr-xr-x    2 root     root             0 May  8 17:05 crontabs
drwxr-xr-x    4 root     nogroup          0 Apr  4 15:54 cups
-rw-r--r--    1 root     root          3072 May  8 17:05 cwmpd.db
drwxr-xr-x    2 root     root             0 May  8 16:50 ddns
drwx------    2 root     root             0 Apr  4 15:54 dropbear
-rw-r--r--    1 root     root             0 Apr  4 15:54 ethers
drwxr-xr-x    2 root     root             0 Oct 16  2019 fileprofiler
-rw-r--r--    1 root     root           572 Apr  4 15:54 firewall.user
-rwxr-xr-x    1 root     root          4222 May  7 22:32 fwdassist.sh
-rw-r--r--    1 root     root           210 Apr 15 21:11 group
drwxr-xr-x    3 root     root             0 May  7 22:32 hotplug.d
drwxr-xr-x    2 root     root             0 Jan  1  1970 init.d
-rw-r--r--    1 root     root            99 Apr 29 03:23 inittab
drwxr-xr-x    2 root     root             0 Apr  4 15:54 ispconfig
-rw-r--r--    1 root     root         15360 May  8 17:05 lasdb.db
-rw-rw-rw-    1 root     root             0 Apr  4 15:54 lastaccess
drwxr-xr-x    3 root     root             0 Apr  4 15:54 lcmd
drwxr-xr-x    2 root     root             0 May  8 16:53 ledfw
drwxr-xr-x    2 root     root             0 Apr  4 15:54 log
drwxr-xr-x    2 root     root             0 Apr 16 00:25 luci-uploads
drwxr-xr-x    2 root     root             0 May  8 16:50 modgui_scripts
drwxr-xr-x    2 root     root             0 May  8 16:50 nginx
drwxr-xr-x    2 root     root             0 May  8 16:53 opkg
-rw-r--r--    1 root     root           724 Apr  4 15:55 opkg.conf
drwxr-xr-x    2 root     root             0 Apr  4 15:54 parameter_conversion
-rw-r--r--    1 root     root           512 Apr 15 21:11 passwd
-rw-r--r--    1 root     root           443 Apr  9 14:08 passwd-
drwxr-xr-x    2 root     root             0 Apr  4 15:56 ppp
-rw-r--r--    1 root     root           135 Apr  4 15:55 printcap
-rw-------    1 root     root           512 Jan  1  1970 random-seed
drwxr-xr-x    2 root     root             0 May  8 17:12 rc.d
drwxr-xr-x    2 root     root             0 Apr 15 23:27 samba
-rw-------    1 root     root           580 Apr 15 21:11 shadow
-rw-------    1 root     root           550 Apr  9 14:08 shadow-
-rw-r--r--    1 root     root            64 Apr  4 15:54 slicstate
-rw-r--r--    1 root     root          2048 Apr  4 15:55 sms.db
drwxr-xr-x    4 root     root             0 May  8 16:50 ssl
-rw-r--r--    1 root     root           978 May  8 16:53 sysctl.conf
drwxr-xr-x    2 root     root             0 Apr 15 21:11 sysctl.d
-rw-r--r--    1 root     root             0 May  8 18:01 test
drwxr-xr-x    2 root     root             0 May  8 16:50 tod
-rw-r--r--    1 root     root        192512 Apr 29 03:25 transformer.db
-rw-r--r--    1 root     root        193912 May  8 17:45 transformer.db-wal
drwxr-xr-x    2 root     root             0 Apr 16 03:20 uci-defaults
drwxr-xr-x    2 root     root             0 May  8 16:53 wansensing
drwxr-xr-x    2 root     root             0 Apr 16 04:08 xupnpd

Dark4Codrutz · 2020-05-08T16:04:03Z

touch /etc/test
ls -la /etc
ls -la /modoverlay/bank_mod/etc

@Dark4Codrutz yes it's all correct what is not correct is the fact that you have /etc empty instead of the /rom file + the file present in /modoverlay

weird indeed, test file ends up only in modoverlay, etc still empty

Dark4Codrutz · 2020-05-08T16:04:30Z

Shall I reboot once again ?

Ansuel · 2020-05-08T16:05:29Z

ls -la /saferoot/etc
@Dark4Codrutz the reboot could fix the problem but i would understand why

Also try
sync
ls -la /etc

Dark4Codrutz · 2020-05-08T16:05:39Z

at least functionally it seems that nothing is broken. files accessed through /etc by scipts/programs are redirected to modoverlay somehow

Ansuel · 2020-05-08T16:06:23Z

also try to restart the ssh session (could be a bug of dropbear? )

Dark4Codrutz · 2020-05-08T16:09:23Z

sync
ls -la /etc

nada

root@modemtim:/etc# ls -la /saferoot/etc
root@modemtim:/etc# sync
root@modemtim:/etc# ls -la

tried ls -la /etc after ssh restart and nothing

Ansuel · 2020-05-08T16:10:32Z

ls -la /rom/etc ?

Dark4Codrutz · 2020-05-08T16:11:20Z

ls -la /rom/etc

root@modemtim:/etc# ls -la /rom/etc
drwxr-xr-x   46 root     root          1748 Oct 25  2019 .
drwxr-xr-x   20 root     root           306 Oct 25  2019 ..
lrwxrwxrwx    1 root     root             7 Oct 25  2019 TZ -> /tmp/TZ
drwxr-xr-x    2 root     root            35 Oct 25  2019 adsl
-rw-r--r--    1 root     root          2862 Oct 25  2019 banner
-rw-r--r--    1 root     root           408 Oct 16  2019 banner.failsafe
drwxr-xr-x    7 root     root            85 Oct 16  2019 boards
-rwxr-xr-x    1 root     root          4442 Oct 16  2019 captive_portal.sh
-rwxr-xr-x    1 root     root           207 Oct 16  2019 cgconfig.conf
-rwxr-xr-x    1 root     root          2280 Oct 16  2019 cgrules.conf
drwxr-xr-x    2 root     root            35 Oct 16  2019 chatscripts
drwxr-xr-x    2 root     root             3 Oct 16  2019 clash
drwxr-xr-x    2 root     root          1312 Oct 25  2019 config
-rw-r--r--    1 root     root           163 Oct 16  2019 config.export
-rw-r--r--    1 root     root           252 Oct 16  2019 config.import
drwxr-xr-x    2 root     root            38 Oct 16  2019 conntrackd
drwxr-xr-x    2 root     root             3 Oct 16  2019 crontabs
drwxr-xr-x    5 root     root           280 Oct 16  2019 cups
drwxr-xr-x    2 root     root            52 Oct 16  2019 ddns
-rw-r--r--    1 root     root            76 Oct 16  2019 device_info
-rw-r--r--    1 root     root            68 Oct 16  2019 diag.sh
-rw-r--r--    1 root     root            10 Oct 16  2019 dnsmasq.conf
drwxr-xr-x    2 root     root           102 Oct 25  2019 dropbear
drwxr-xr-x    2 root     root            79 Oct 16  2019 efu_handler
-rw-r--r--    1 root     root          1362 Oct 16  2019 ethertypes
drwxr-xr-x    2 root     root            40 Oct 16  2019 fileprofiler
-rw-r--r--    1 root     root           352 Oct 16  2019 firewall.user
lrwxrwxrwx    1 root     root            10 Oct 25  2019 fstab -> /tmp/fstab
-rwxr-xr-x    1 root     root          5088 Oct 16  2019 fwdassist.sh
-rw-r--r--    1 root     root           155 Oct 16  2019 group
-rw-r--r--    1 root     root            20 Oct 24  2019 hosts
-rw-r--r--    1 root     root           305 Oct 16  2019 hotplug-preinit.json
drwxr-xr-x   14 root     root           167 Oct 16  2019 hotplug.d
-rw-r--r--    1 root     root          1657 Oct 16  2019 hotplug.json
drwxr-xr-x    2 root     root            36 Oct 16  2019 import
drwxr-xr-x    2 root     root          1875 Oct 16  2019 init.d
-rw-r--r--    1 root     root           100 Oct 24  2019 inittab
drwxr-xr-x    2 root     root            49 Oct 25  2019 iproute2
-rw-r--r--    1 root     root            17 Oct 16  2019 jffs2_options
-rw-r--r--    1 root     root            27 Oct 16  2019 ld.so.preload
drwxr-xr-x    2 root     root            40 Oct 25  2019 ledfw
lrwxrwxrwx    1 root     root            14 Oct 25  2019 localtime -> /tmp/localtime
drwxr-xr-x    2 root     root            51 Oct 16  2019 lxc
drwxr-xr-x    2 root     root            38 Oct 16  2019 mmpbx
drwxr-xr-x    2 root     root           718 Oct 25  2019 mobiled
drwxr-xr-x    2 root     root           364 Oct 16  2019 modules-boot.d
drwxr-xr-x    2 root     root          3112 Oct 16  2019 modules.d
drwxr-xr-x    2 root     root            37 Oct 16  2019 mosquitto
drwxr-xr-x    3 root     root            28 Oct 16  2019 mqttjson_services
lrwxrwxrwx    1 root     root            12 Oct 25  2019 mtab -> /proc/mounts
drwxr-xr-x    2 root     root            47 Oct 16  2019 mud
drwxr-xr-x    2 root     root           106 Oct 25  2019 nginx
drwxr-xr-x    2 root     root             3 Oct 16  2019 nqe
-rw-r--r--    1 root     root           185 Oct 16  2019 nsswitch.conf
-rw-r--r--    1 root     root           236 Oct 16  2019 openwrt_release
-rw-r--r--    1 root     root             8 Oct 16  2019 openwrt_version
drwxr-xr-x    3 root     root            73 Oct 16  2019 opkg
-rw-r--r--    1 root     root            85 Oct 16  2019 opkg.conf
drwxr-xr-x    3 root     root            97 Oct 25  2019 parameter_conversion
-rw-r--r--    1 root     root           317 Oct 16  2019 passwd
drwxr-xr-x    2 root     root           105 Oct 16  2019 ppp
-rwxr-xr-x    1 root     root           952 Oct 16  2019 preinit
-rw-r--r--    1 root     root          1205 Oct 16  2019 profile
-rw-r--r--    1 root     root          2478 Oct 16  2019 protocols
-rw-r--r--    1 root     root           800 Oct 16  2019 public.pem
-rwxr-xr-x    1 root     root          1279 Oct 24  2019 ra_forward.sh
drwxr-xr-x    2 root     root            71 Oct 16  2019 rc.button
-rwxr-xr-x    1 root     root          2405 Oct 16  2019 rc.common
drwxr-xr-x    2 root     root          3023 Oct 16  2019 rc.d
-rw-r--r--    1 root     root           132 Oct 16  2019 rc.local
-rwxr-xr-x    1 root     root           306 Oct 16  2019 rdpa_init.sh
lrwxrwxrwx    1 root     root            16 Oct 25  2019 resolv.conf -> /tmp/resolv.conf
drwxr-xr-x    4 root     root            42 Oct 16  2019 rtfd
-rw-r--r--    1 root     root             0 Oct 16  2019 rtfd_persistent_filelist.conf
-rw-r--r--    1 root     root          3055 Oct 16  2019 services
-rw-------    1 root     root           168 Oct 16  2019 shadow
-rw-r--r--    1 root     root            31 Oct 16  2019 shells
drwxr-xr-x    2 root     root            33 Oct 16  2019 snmp
drwxr-xr-x    3 root     root            28 Oct 25  2019 ssl
-rw-r--r--    1 root     root          3174 Oct 16  2019 sudoers
drwxr-xr-x    2 root     root             3 Oct 16  2019 sudoers.d
-rw-r--r--    1 root     root           500 Oct 16  2019 sysctl-tch.conf
-rw-r--r--    1 root     root           978 Oct 16  2019 sysctl.conf
drwxr-xr-x    2 root     root            85 Oct 16  2019 syslog_fwd
-rw-r--r--    1 root     root           128 Oct 16  2019 sysupgrade.conf
drwxr-xr-x    2 root     root            66 Oct 16  2019 tod
-rw-r--r--    1 root     root          1590 Oct 16  2019 tr143.default
drwxr-xr-x    2 root     root          1413 Oct 25  2019 uci-defaults
-rwxr-xr-x    1 root     root          1443 Oct 16  2019 udhcpc.user
-rw-r--r--    1 root     root         50390 Oct 16  2019 usb-mode.json
drwxr-xr-x    2 root     root           167 Oct 25  2019 wansensing
-rw-r--r--    1 root     root           232 Oct 16  2019 weburl_cfg_tmpl
drwxr-xr-x    2 root     root             3 Oct 16  2019 wifi-doctor-agent
-rw-r--r--    1 root     root          1255 Oct 16  2019 wireless_acs.conf
lrwxrwxrwx    1 root     root            15 Oct 25  2019 wlan -> /etc/board/wlan
drwxr-xr-x    2 root     root            43 Oct 16  2019 wlan_common
-rw-r--r--    1 root     root            40 Oct 16  2019 xinetd.conf
drwxr-xr-x    2 root     root             3 Oct 16  2019 xinetd.d
drwxr-xr-x    2 root     root            55 Oct 16  2019 xl2tpd

Ansuel · 2020-05-08T16:15:41Z

try to reboot and tell me if something change

Dark4Codrutz · 2020-05-08T16:18:50Z

the weirdness keeps growing:

root@modemtim:~# mud


Usage: mud -t <file-type-INI-file> [-n <nice>] [-d] [-h]
Options are as follows:
  -t <path>   path of the INI file mapping file extensions to type/category
  -d          run as a daemon
  -n <nice>   run at non-default nice value (priority)
  -w <delay>  wait for <delay> seconds before initiating cold scan actions
  -h          print this help text and exit

root@modemtim:~# realpath mud
root@modemtim:~# realpath mud
root@modemtim:~# cd /etc
root@modemtim:/etc# realpath mud
/etc/mud
root@modemtim:/etc# cd /rom/etc
root@modemtim:/rom/etc# realpath mud
/rom/etc/mud

with realpath I'm asking for mud directory in /etc , and it says it exists and it is /etc/mud
but in reality it can only be found by the user in /rom/etc

To me it looks like something weird went with overlaying, for the system there is stuff in /etc but cannot be listed

Ansuel · 2020-05-08T16:20:12Z

try /etc/init.d/dropbear restart

Dark4Codrutz · 2020-05-08T16:25:17Z

try to reboot and tell me if something change

root@modemtim:/etc# ls -la
drwxr-xr-x    1 root     root             0 May  8 18:23 .
drwxr-xr-x    1 root     root             0 Jan  1  1970 ..
lrwxrwxrwx    1 root     root             7 Oct 25  2019 TZ -> /tmp/TZ
drwxr-xr-x    2 root     root            35 Oct 25  2019 adsl
-rw-r--r--    1 root     root          2862 Jan  1  1970 banner
-rw-r--r--    1 root     root           408 Oct 16  2019 banner.failsafe
lrwxrwxrwx    1 root     root            18 Jan  1  1970 board -> /etc/boards/VBNT-S
drwxr-xr-x    1 root     root             0 Oct 16  2019 boards
-rwxr-xr-x    1 root     root          4442 Oct 16  2019 captive_portal.sh
-rwxr-xr-x    1 root     root           207 Oct 16  2019 cgconfig.conf
-rwxr-xr-x    1 root     root          2280 Oct 16  2019 cgrules.conf
drwxr-xr-x    2 root     root            35 Oct 16  2019 chatscripts
-rwxr-xr-x    1 root     root           861 May  7 22:32 christmas_tree.sh
drwxr-xr-x    1 root     root             0 Apr  4 15:54 clash
drwxr-xr-x    1 root     root             0 May  8 18:23 config
-rw-r--r--    1 root     root           163 Oct 16  2019 config.export
-rw-r--r--    1 root     root           252 Oct 16  2019 config.import
drwxr-xr-x    2 root     root            38 Oct 16  2019 conntrackd
drwxr-xr-x    1 root     root             0 May  8 18:23 crontabs
drwxr-xr-x    1 root     nogroup          0 Apr  4 15:54 cups
-rw-r--r--    1 root     root          3072 May  8 18:23 cwmpd.db
drwxr-xr-x    1 root     root             0 May  8 16:50 ddns
-rw-r--r--    1 root     root            76 Oct 16  2019 device_info
-rw-r--r--    1 root     root            68 Oct 16  2019 diag.sh
-rw-r--r--    1 root     root            10 Oct 16  2019 dnsmasq.conf
drwx------    1 root     root             0 Apr  4 15:54 dropbear
drwxr-xr-x    2 root     root            79 Oct 16  2019 efu_handler
-rw-r--r--    1 root     root             0 Apr  4 15:54 ethers
-rw-r--r--    1 root     root          1362 Oct 16  2019 ethertypes
drwxr-xr-x    1 root     root             0 Oct 16  2019 fileprofiler
-rw-r--r--    1 root     root           572 Apr  4 15:54 firewall.user
lrwxrwxrwx    1 root     root            10 Oct 25  2019 fstab -> /tmp/fstab
-rwxr-xr-x    1 root     root          4222 May  7 22:32 fwdassist.sh
-rw-r--r--    1 root     root           210 Apr 15 21:11 group
-rw-r--r--    1 root     root            20 Oct 24  2019 hosts
-rw-r--r--    1 root     root           305 Oct 16  2019 hotplug-preinit.json
drwxr-xr-x    1 root     root             0 May  7 22:32 hotplug.d
-rw-r--r--    1 root     root          1657 Oct 16  2019 hotplug.json
drwxr-xr-x    2 root     root            36 Oct 16  2019 import
drwxr-xr-x    1 root     root             0 May  8 18:21 init.d
-rw-r--r--    1 root     root            99 Apr 29 03:23 inittab
drwxr-xr-x    2 root     root            49 Oct 25  2019 iproute2
drwxr-xr-x    2 root     root             0 Apr  4 15:54 ispconfig
-rw-r--r--    1 root     root            17 Oct 16  2019 jffs2_options
-rw-r--r--    1 root     root         15360 May  8 18:23 lasdb.db
-rw-rw-rw-    1 root     root             0 Apr  4 15:54 lastaccess
drwxr-xr-x    3 root     root             0 Apr  4 15:54 lcmd
-rw-r--r--    1 root     root            27 Oct 16  2019 ld.so.preload
drwxr-xr-x    1 root     root             0 May  8 16:53 ledfw
lrwxrwxrwx    1 root     root            14 Oct 25  2019 localtime -> /tmp/localtime
drwxr-xr-x    2 root     root             0 Apr  4 15:54 log
drwxr-xr-x    2 root     root             0 Apr 16 00:25 luci-uploads
drwxr-xr-x    2 root     root            51 Oct 16  2019 lxc
drwxr-xr-x    2 root     root            38 Oct 16  2019 mmpbx
drwxr-xr-x    2 root     root           718 Oct 25  2019 mobiled
drwxr-xr-x    2 root     root             0 May  8 16:50 modgui_scripts
drwxr-xr-x    2 root     root           364 Oct 16  2019 modules-boot.d
drwxr-xr-x    2 root     root          3112 Oct 16  2019 modules.d
drwxr-xr-x    2 root     root            37 Oct 16  2019 mosquitto
drwxr-xr-x    3 root     root            28 Oct 16  2019 mqttjson_services
lrwxrwxrwx    1 root     root            12 Oct 25  2019 mtab -> /proc/mounts
drwxr-xr-x    2 root     root            47 Oct 16  2019 mud
drwxr-xr-x    1 root     root             0 May  8 16:50 nginx
drwxr-xr-x    2 root     root             3 Oct 16  2019 nqe
-rw-r--r--    1 root     root           185 Oct 16  2019 nsswitch.conf
-rw-r--r--    1 root     root           236 Oct 16  2019 openwrt_release
-rw-r--r--    1 root     root             8 Oct 16  2019 openwrt_version
drwxr-xr-x    1 root     root             0 May  8 16:53 opkg
-rw-r--r--    1 root     root           724 Apr  4 15:55 opkg.conf
drwxr-xr-x    1 root     root             0 Apr  4 15:54 parameter_conversion
-rw-r--r--    1 root     root           512 Apr 15 21:11 passwd
-rw-r--r--    1 root     root           443 Apr  9 14:08 passwd-
drwxr-xr-x    1 root     root             0 Apr  4 15:56 ppp
-rwxr-xr-x    1 root     root           952 Oct 16  2019 preinit
-rw-r--r--    1 root     root           135 Apr  4 15:55 printcap
-rw-r--r--    1 root     root          1205 Oct 16  2019 profile
-rw-r--r--    1 root     root          2478 Oct 16  2019 protocols
-rw-r--r--    1 root     root           800 Oct 16  2019 public.pem
-rwxr-xr-x    1 root     root          1279 Oct 24  2019 ra_forward.sh
-rw-------    1 root     root           512 May  8 18:20 random-seed
drwxr-xr-x    2 root     root            71 Oct 16  2019 rc.button
-rwxr-xr-x    1 root     root          2405 Oct 16  2019 rc.common
drwxr-xr-x    1 root     root             0 May  8 18:21 rc.d
-rw-r--r--    1 root     root           132 Oct 16  2019 rc.local
-rwxr-xr-x    1 root     root           306 Oct 16  2019 rdpa_init.sh
lrwxrwxrwx    1 root     root            16 Oct 25  2019 resolv.conf -> /tmp/resolv.conf
drwxr-xr-x    4 root     root            42 Oct 16  2019 rtfd
-rw-r--r--    1 root     root             0 Oct 16  2019 rtfd_persistent_filelist.conf
drwxr-xr-x    2 root     root             0 Apr 15 23:27 samba
-rw-r--r--    1 root     root          3055 Oct 16  2019 services
-rw-------    1 root     root           580 Apr 15 21:11 shadow
-rw-------    1 root     root           550 Apr  9 14:08 shadow-
-rw-r--r--    1 root     root            31 Oct 16  2019 shells
-rw-r--r--    1 root     root            64 Apr  4 15:54 slicstate
-rw-r--r--    1 root     root          2048 Apr  4 15:55 sms.db
drwxr-xr-x    2 root     root            33 Oct 16  2019 snmp
drwxr-xr-x    1 root     root             0 May  8 16:50 ssl
-rw-r--r--    1 root     root          3174 Oct 16  2019 sudoers
drwxr-xr-x    2 root     root             3 Oct 16  2019 sudoers.d
-rw-r--r--    1 root     root           500 Oct 16  2019 sysctl-tch.conf
-rw-r--r--    1 root     root           978 May  8 16:53 sysctl.conf
drwxr-xr-x    2 root     root             0 Apr 15 21:11 sysctl.d
drwxr-xr-x    2 root     root            85 Oct 16  2019 syslog_fwd
-rw-r--r--    1 root     root           128 Oct 16  2019 sysupgrade.conf
-rw-r--r--    1 root     root             0 May  8 18:01 test
drwxr-xr-x    1 root     root             0 May  8 16:50 tod
-rw-r--r--    1 root     root          1590 Oct 16  2019 tr143.default
-rw-r--r--    1 root     root        192512 Apr 29 03:25 transformer.db
-rw-r--r--    1 root     root        193912 May  8 18:21 transformer.db-wal
drwxr-xr-x    1 root     root             0 Apr 16 03:20 uci-defaults
-rwxr-xr-x    1 root     root          1443 Oct 16  2019 udhcpc.user
-rw-r--r--    1 root     root         50390 Oct 16  2019 usb-mode.json
drwxr-xr-x    1 root     root             0 May  8 16:53 wansensing
-rw-r--r--    1 root     root           232 Oct 16  2019 weburl_cfg_tmpl
drwxr-xr-x    2 root     root             3 Oct 16  2019 wifi-doctor-agent
-rw-r--r--    1 root     root          1255 Oct 16  2019 wireless_acs.conf
lrwxrwxrwx    1 root     root            15 Oct 25  2019 wlan -> /etc/board/wlan
drwxr-xr-x    2 root     root            43 Oct 16  2019 wlan_common
-rw-r--r--    1 root     root            40 Oct 16  2019 xinetd.conf
drwxr-xr-x    2 root     root             3 Oct 16  2019 xinetd.d
drwxr-xr-x    2 root     root            55 Oct 16  2019 xl2tpd
drwxr-xr-x    2 root     root             0 Apr 16 04:08 xupnpd

Dark4Codrutz · 2020-05-08T16:25:36Z

a reboot did it

try /etc/init.d/dropbear restart

shall I test ?

Ansuel · 2020-05-08T16:27:04Z

well if you now correctly display the files...
just to make sure... output of mount ?

Dark4Codrutz · 2020-05-08T16:27:32Z

root@modemtim:/etc# mount
/dev/root on /rom type squashfs (ro,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,noatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
/dev/mtdblock2 on /overlay type jffs2 (rw,relatime,compr=zlib)
overlay on /saferoot type overlay (rw,relatime,lowerdir=/,upperdir=/overlay/bank_2,workdir=/overlay/work)
/dev/mtdblock3 on /modoverlay type jffs2 (rw,relatime,compr=zlib)
overlay on / type overlay (rw,relatime,lowerdir=/,upperdir=/modoverlay/bank_mod,workdir=/modoverlay/work)
tmpfs on /dev type tmpfs (rw,nosuid,relatime,mode=0755,size=512K)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)
cgroup on /cgroups/cpumemblk type cgroup (rw,relatime,cpu,blkio,memory)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)
mvfs on /tmp/mvfs type fuse.mvfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
mountd(pid6541) on /tmp/run/mountd type autofs (rw,relatime,fd=5,pgrp=6539,timeout=600,minproto=5,maxproto=5,indirect)
/dev/sda1 on /tmp/run/mountd/sda1 type ufsd (rw,noatime,nls=utf8,uid=0,gid=0,fmask=0,dmask=0,nocase,sparse,force

LuKePicci · 2020-05-08T16:27:35Z

I would expect this behavior if something is written into /modoverlay after mounting the overlay filesystem

Dark4Codrutz · 2020-05-08T16:28:57Z

root@modemtim:/etc# mount
/dev/root on /rom type squashfs (ro,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,noatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
/dev/mtdblock2 on /overlay type jffs2 (rw,relatime,compr=zlib)
overlay on /saferoot type overlay (rw,relatime,lowerdir=/,upperdir=/overlay/bank_2,workdir=/overlay/work)
/dev/mtdblock3 on /modoverlay type jffs2 (rw,relatime,compr=zlib)
overlay on / type overlay (rw,relatime,lowerdir=/,upperdir=/modoverlay/bank_mod,workdir=/modoverlay/work)
tmpfs on /dev type tmpfs (rw,nosuid,relatime,mode=0755,size=512K)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)
cgroup on /cgroups/cpumemblk type cgroup (rw,relatime,cpu,blkio,memory)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)
mvfs on /tmp/mvfs type fuse.mvfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
mountd(pid6541) on /tmp/run/mountd type autofs (rw,relatime,fd=5,pgrp=6539,timeout=600,minproto=5,maxproto=5,indirect)
/dev/sda1 on /tmp/run/mountd/sda1 type ufsd (rw,noatime,nls=utf8,uid=0,gid=0,fmask=0,dmask=0,nocase,sparse,force

only difference with the previous one is mountd pid and pgrp attribute
so not much

Ansuel · 2020-05-08T16:30:04Z

@LuKePicci so if something is written in /modoverlay the entire overlay structure broke up ?

This doesn't happen with the default overlay when we directly write in /overlay

Dark4Codrutz · 2020-05-08T16:30:18Z

a reboot did it

try /etc/init.d/dropbear restart

shall I test ?

I missread when I asked to test...
Was thinking about testing the forced root script

Ansuel · 2020-05-08T16:32:43Z

@LuKePicci I FOUND THE PROBLEM....

on newer firmware init preint is run before the random_seed...
think we need to inject the script in mount_root to be 100% sure

(this was 100% caused by dropbear not beeing restarted after the pivot)

Ansuel · 2020-05-08T16:33:38Z

root@modemtim:~# ls /lib/preinit/
02_default_set_state         95_restore_version
10_indicate_preinit          97_rip
10_sysinfo                   98_boardname
20_mount_essentials          98_efu_handler
50_indicate_regular_preinit  98_procd_debug
70_initramfs_test            99_10_run_init
80_mount_root                99_random_seed

LuKePicci · 2020-05-08T16:34:51Z

It also happens with the default overlay but at lazy timings. A direct write to /modoverlay will appear into / sooner with respect to the standard overlay. That's why you migrate stuff before mounting the overlay. After pivoting nothing should never write to /modoverlay directly

I also expect something will still be written into the old overlay (/saferoot) after modoverlay is stacked on top of it and pivoted Since running processes will keep an handle on the old inodes. That's why we avoid dismounting the old overlay.

LuKePicci · 2020-05-08T16:36:24Z

on newer firmware init preint is run before the random_seed...

I don't think so, hooks are different. The 99_10_run_init belongs to preinit_main I thinm, not preinit_mount_root

Basically those heading numbers in filenames are relative to a sublist of those scripts

Ansuel · 2020-05-08T16:38:01Z

But still... we don't have any process that write to /modoverlay directly if I remember correctly

LuKePicci · 2020-05-08T16:39:13Z

Didn't you on upgrades and settings migration?

Dark4Codrutz · 2020-05-08T16:44:59Z

(this was 100% caused by dropbear not beeing restarted after the pivot)

Though the modem rebooted twice for the OBP setup. Doesn't it do it like:

after gui upgrade set OPB flag up if needed
modem reboots, OPB is set up
modem reboots 2nd time, loads with OPB changes in place

but in fact another reboot was needed here:
4. reboot 3rd time

Don't know the OBP setup script logic so I'm just guessing in the dark here

LuKePicci · 2020-05-08T16:50:40Z

you are mixing OBP and modoverlay stuff

2 router reboots, overlay for bank_1 is migrated to overlay for bank_2, OBP is set up
3 router reboots 2nd time, OBP let it boot from bank_2 firmware, modoverlay is mounted for the very first time and overlay for bank_2 is migrated to modoverlay, then it is mounted as a second overlay

OBP has nothing to do with modoverlay, you may also do modoverlay without OBP from bank_1 but you won't be able to erase the modoverlay partition by TFTP

Dark4Codrutz · 2020-05-08T16:52:25Z

is this normal ?

Active Bank
bank_1
Active Version
Unknown
Passive Bank
bank_2
Passive Version
18.3.k.0451-3161011-20191025181819-3f83315f6d40733756f2a3ef5b192653d99f3c87

LuKePicci · 2020-05-08T16:52:43Z

Sure, bank_1 contains modoverlay

LuKePicci · 2020-05-08T16:53:38Z

@Ansuel I think a reboot is always needed after overlay -> modoverlay migration

Ansuel · 2020-05-08T16:54:46Z

@LuKePicci so we put a reboot after the copy ?

LuKePicci · 2020-05-08T16:55:23Z

after preserve, inside that last { } enclosed line

LuKePicci · 2020-05-08T16:59:02Z

This is because that lazy thing I was mentioning, when you move stuff away from /overlay you need to reboot or remount to make it update the /saferoot structure

The crazy alternative would be to move every file existing in /overlay/bank_2/ from / (/saferoot/) to /modoverlay/

Ansuel · 2020-05-08T17:00:45Z

@LuKePicci could be very slow... lets reboot as this is really for the migration only

Dark4Codrutz · 2020-05-08T18:48:07Z

Guess this issue can be closed now...

Dark4Codrutz · 2020-05-08T18:55:19Z

you are mixing OBP and modoverlay stuff

2 router reboots, overlay for bank_1 is migrated to overlay for bank_2, OBP is set up
3 router reboots 2nd time, OBP let it boot from bank_2 firmware, modoverlay is mounted for the very first time and overlay for bank_2 is migrated to modoverlay, then it is mounted as a second overlay

OBP has nothing to do with modoverlay, you may also do modoverlay without OBP from bank_1 but you won't be able to erase the modoverlay partition by TFTP

Off topic:
If I opt out of this by disabling OBP check in gui is there anything wrong with manual managing bank_2 with occasional cloning of known stable rooted firmware installation from bank_1 ?
Or doing this is not enough and won't save me in case of fatal failure of bank_1 ?

FrancYescO · 2020-05-08T20:17:57Z

basically yes, but is better to just put your stable-and-rooted overlay in /overlay/bank1 than when you need to recover the TFTP uploaded firmware will start with your "safe" overlay and not blank.

LuKePicci · 2020-05-08T20:54:33Z

If I opt out of this by disabling OBP

There is no good reason to opt out of OBP, That is perfectly stable and you loos most chances of recovery.

Or doing this is not enough and won't save me in case of fatal failure of bank_1 ?

Exactly. If you are evaluating to opt out in search of stability in current dev versions, then you might prefer to simply disable modoverlay. But OBP should always be there.

Dark4Codrutz changed the title ~~/etc is empy , files are now in modoverlay ?~~ Weird folder structure after OBP : /etc is empy , files are now in modoverlay ? May 8, 2020

FrancYescO closed this as completed May 15, 2020

Weird folder structure after OBP : /etc is empy , files are now in modoverlay ? #957

Weird folder structure after OBP : /etc is empy , files are now in modoverlay ? #957

Comments

Dark4Codrutz commented May 8, 2020 • edited Loading

Dark4Codrutz commented May 8, 2020

Ansuel commented May 8, 2020

Dark4Codrutz commented May 8, 2020 • edited Loading

Ansuel commented May 8, 2020 • edited Loading

Ansuel commented May 8, 2020

Dark4Codrutz commented May 8, 2020

Dark4Codrutz commented May 8, 2020

Dark4Codrutz commented May 8, 2020 • edited Loading

Dark4Codrutz commented May 8, 2020

Ansuel commented May 8, 2020

Dark4Codrutz commented May 8, 2020

Dark4Codrutz commented May 8, 2020 • edited Loading

Dark4Codrutz commented May 8, 2020

Ansuel commented May 8, 2020

Dark4Codrutz commented May 8, 2020

Ansuel commented May 8, 2020

Dark4Codrutz commented May 8, 2020

Ansuel commented May 8, 2020

Dark4Codrutz commented May 8, 2020

Ansuel commented May 8, 2020

Dark4Codrutz commented May 8, 2020 • edited Loading

Ansuel commented May 8, 2020

Dark4Codrutz commented May 8, 2020

Dark4Codrutz commented May 8, 2020

Ansuel commented May 8, 2020

Dark4Codrutz commented May 8, 2020

LuKePicci commented May 8, 2020

Dark4Codrutz commented May 8, 2020

Ansuel commented May 8, 2020 • edited Loading

Dark4Codrutz commented May 8, 2020

Ansuel commented May 8, 2020

Ansuel commented May 8, 2020

LuKePicci commented May 8, 2020

LuKePicci commented May 8, 2020 • edited Loading

Ansuel commented May 8, 2020

LuKePicci commented May 8, 2020

Dark4Codrutz commented May 8, 2020

LuKePicci commented May 8, 2020 • edited Loading

Dark4Codrutz commented May 8, 2020

LuKePicci commented May 8, 2020

LuKePicci commented May 8, 2020

Ansuel commented May 8, 2020

LuKePicci commented May 8, 2020

LuKePicci commented May 8, 2020 • edited Loading

Ansuel commented May 8, 2020

Dark4Codrutz commented May 8, 2020

Dark4Codrutz commented May 8, 2020 • edited Loading

FrancYescO commented May 8, 2020 • edited Loading

LuKePicci commented May 8, 2020

Dark4Codrutz commented May 8, 2020 •

edited

Loading

Dark4Codrutz commented May 8, 2020 •

edited

Loading

Ansuel commented May 8, 2020 •

edited

Loading

Dark4Codrutz commented May 8, 2020 •

edited

Loading

Dark4Codrutz commented May 8, 2020 •

edited

Loading

Dark4Codrutz commented May 8, 2020 •

edited

Loading

Ansuel commented May 8, 2020 •

edited

Loading

LuKePicci commented May 8, 2020 •

edited

Loading

LuKePicci commented May 8, 2020 •

edited

Loading

LuKePicci commented May 8, 2020 •

edited

Loading

Dark4Codrutz commented May 8, 2020 •

edited

Loading

FrancYescO commented May 8, 2020 •

edited

Loading