Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FS#3730 - resolve_conffiles reports spurious errors #8751

Open
openwrt-bot opened this issue Apr 8, 2021 · 4 comments
Open

FS#3730 - resolve_conffiles reports spurious errors #8751

openwrt-bot opened this issue Apr 8, 2021 · 4 comments
Labels
flyspray release/19.07

Comments

@openwrt-bot
Copy link

@openwrt-bot openwrt-bot commented Apr 8, 2021

kousu:

When I run upgrades root@router:~# opkg update && opkg list-upgradable | awk '{print $1}' | xargs opkg upgrade

I usually end up with some messages like

Collected errors: * resolve_conffiles: Existing conffile /etc/config/luci_statistics is different from the conffile in the new package. The new conffile will be placed at /etc/config/luci_statistics-opkg. * resolve_conffiles: Existing conffile /etc/config/luci is different from the conffile in the new package. The new conffile will be placed at /etc/config/luci-opkg. * resolve_conffiles: Existing conffile /etc/config/ucitrack is different from the conffile in the new package. The new conffile will be placed at /etc/config/ucitrack-opkg.

Sometimes the conflicts are genuine -- because I've configured the router -- but most of them are because uci doesn't force a canonical output format. For example

root@router:~# diff -u /etc/config/luci-opkg /etc/config/luci --- /etc/config/luci-opkg 2021-03-27 02:44:43.000000000 -0400 +++ /etc/config/luci 2021-01-30 05:15:01.000000000 -0500 @@ -1,31 +1,39 @@ -config core main - option lang auto - option mediaurlbase /luci-static/bootstrap - option resourcebase /luci-static/resources - option ubuspath /ubus/ - -config extern flash_keep - option uci "/etc/config/" - option dropbear "/etc/dropbear/" - option openvpn "/etc/openvpn/" - option passwd "/etc/passwd" - option opkg "/etc/opkg.conf" - option firewall "/etc/firewall.user" - option uploads "/lib/uci/upload/" - -config internal languages - -config internal sauth - option sessionpath "/tmp/luci-sessions" - option sessiontime 3600 - -config internal ccache - option enable 1 - -config internal themes - -config internal apply - option rollback 90 - option holdoff 4 - option timeout 5 - option display 1.5 + +config core 'main' + option lang 'auto' + option mediaurlbase '/luci-static/bootstrap' + option resourcebase '/luci-static/resources' + option ubuspath '/ubus/' + +config extern 'flash_keep' + option uci '/etc/config/' + option dropbear '/etc/dropbear/' + option openvpn '/etc/openvpn/' + option passwd '/etc/passwd' + option opkg '/etc/opkg.conf' + option firewall '/etc/firewall.user' + option uploads '/lib/uci/upload/' + +config internal 'languages' + +config internal 'sauth' + option sessionpath '/tmp/luci-sessions' + option sessiontime '3600' + +config internal 'ccache' + option enable '1' + +config internal 'themes' + option Bootstrap '/luci-static/bootstrap' + +config internal 'apply' + option rollback '90' + option holdoff '4' + option timeout '5' + option display '1.5' + +config internal 'diag' + option dns 'openwrt.org' + option ping 'openwrt.org' + option route 'openwrt.org' +

Except for luci.diag and luci.themes.Bootstrap, these are identical config files, and I don't think even configured luci.diag so that's probably just from a previous change.

It can be hard to separate out what I've configured from what the packagers have changed, but it is much more tedious when I also need to separate out what's spurious changes from the quotes and indentation changing.

It would save users a lot of time if either:

  • resolve_conffiles called uci to check differences instead of comparing entire files, or
  • uci had a canonical format that both it and luci agreed to stick to, strictly so that running diff (or whatever resolve_conffiles does) would only report genuine conflicts

Don't get me wrong, I do like openwrt a lot. It's way better than using an unfixable vendor firmware. The fact that I can upgrade packages on it at all is marvelous. Thank you for your work.


Model: TP-Link Archer C50 v4 [[https://openwrt.org/toh/tp-link/archer-c50]]

Build: OpenWrt 19.07.7 r11306-c4a6851c72 / LuCI openwrt-19.07 branch git-21.096.73306-254083c

Kernel: 4.14.221

Packages:

root@router:~# opkg list-installed base-files - 204.2-r11306-c4a6851c72 busybox - 1.30.1-6 cgi-io - 19 collectd - 5.12.0-1 collectd-mod-conntrack - 5.12.0-1 collectd-mod-cpu - 5.12.0-1 collectd-mod-interface - 5.12.0-1 collectd-mod-iwinfo - 5.12.0-1 collectd-mod-load - 5.12.0-1 collectd-mod-memory - 5.12.0-1 collectd-mod-network - 5.12.0-1 collectd-mod-ping - 5.12.0-1 collectd-mod-rrdtool - 5.12.0-1 collectd-mod-uptime - 5.12.0-1 collectd-mod-wireless - 5.12.0-1 conntrack - 2018-05-01-88610abe-1 diffutils - 3.7-2 dnsmasq - 2.80-16.3 dropbear - 2019.78-2 firewall - 2019-11-22-8174814a-3 fstools - 2020-05-12-84269037-1 fwtool - 2 getrandom - 2019-06-16-4df34a4d-3 hostapd-common - 2019-08-08-ca8c2bd2-7 ip6tables - 1.8.3-1 iptables - 1.8.3-1 iptables-mod-conntrack-extra - 1.8.3-1 iptables-mod-ipopt - 1.8.3-1 iw - 5.0.1-1 iwinfo - 2019-10-16-07315b6f-1 jshn - 2020-05-25-66195aee-1 jsonfilter - 2018-02-04-c7e938d6-1 kernel - 4.14.221-1-d92769dc5268e102503ae83fe968a56c kmod-cfg80211 - 4.14.221+4.19.161-1-1 kmod-gpio-button-hotplug - 4.14.221-3 kmod-ifb - 4.14.221-1 kmod-ip6tables - 4.14.221-1 kmod-ipt-conntrack - 4.14.221-1 kmod-ipt-conntrack-extra - 4.14.221-1 kmod-ipt-core - 4.14.221-1 kmod-ipt-ipopt - 4.14.221-1 kmod-ipt-nat - 4.14.221-1 kmod-ipt-offload - 4.14.221-1 kmod-ipt-raw - 4.14.221-1 kmod-leds-gpio - 4.14.221-1 kmod-lib-crc-ccitt - 4.14.221-1 kmod-mac80211 - 4.14.221+4.19.161-1-1 kmod-mt76-core - 4.14.221+2021-02-15-5c768dec-1 kmod-mt7603 - 4.14.221+2021-02-15-5c768dec-1 kmod-mt76x02-common - 4.14.221+2021-02-15-5c768dec-1 kmod-mt76x2 - 4.14.221+2021-02-15-5c768dec-1 kmod-mt76x2-common - 4.14.221+2021-02-15-5c768dec-1 kmod-nf-conntrack - 4.14.221-1 kmod-nf-conntrack-netlink - 4.14.221-1 kmod-nf-conntrack6 - 4.14.221-1 kmod-nf-flow - 4.14.221-1 kmod-nf-ipt - 4.14.221-1 kmod-nf-ipt6 - 4.14.221-1 kmod-nf-nat - 4.14.221-1 kmod-nf-reject - 4.14.221-1 kmod-nf-reject6 - 4.14.221-1 kmod-nfnetlink - 4.14.221-1 kmod-ppp - 4.14.221-1 kmod-pppoe - 4.14.221-1 kmod-pppox - 4.14.221-1 kmod-sched-cake - 4.14.221+2019-03-12-057c7388-1 kmod-sched-core - 4.14.221-1 kmod-slhc - 4.14.221-1 libblobmsg-json - 2020-05-25-66195aee-1 libc - 1.1.24-2 libcap - 2.27-1 libelf1 - 0.177-1 libgcc1 - 7.5.0-2 libip4tc2 - 1.8.3-1 libip6tc2 - 1.8.3-1 libiwinfo-lua - 2019-10-16-07315b6f-1 libiwinfo20181126 - 2019-10-16-07315b6f-1 libjson-c2 - 0.12.1-3.1 libjson-script - 2020-05-25-66195aee-1 libltdl7 - 2.4.6-2 liblua5.1.5 - 5.1.5-3 liblucihttp-lua - 2019-07-05-a34a17d5-1 liblucihttp0 - 2019-07-05-a34a17d5-1 libmbedtls12 - 2.16.10-1 libmnl0 - 1.0.4-2 libnetfilter-conntrack3 - 2018-05-01-3ccae9f5-2 libnetfilter-cthelper0 - 1.0.0-2 libnetfilter-cttimeout1 - 1.0.0-2 libnetfilter-queue1 - 2017-06-27-601abd1c-2 libnfnetlink0 - 1.0.1-3 libnl-tiny - 0.1-5 liboping - 1.10.0-2 libpthread - 1.1.24-2 librrd1 - 1.0.50-3 libubox20191228 - 2020-05-25-66195aee-1 libubus-lua - 2019-12-27-041c9d1c-1 libubus20191227 - 2019-12-27-041c9d1c-1 libuci20130104 - 2019-09-01-415f9e48-4 libuclient20160123 - 2020-06-17-51e16ebf-1 libustream-mbedtls20150806 - 2020-03-13-40b563b1-1 libxtables12 - 1.8.3-1 logd - 2019-06-16-4df34a4d-3 lua - 5.1.5-3 luci - git-21.096.73306-254083c-1 luci-app-firewall - git-21.096.73306-254083c-1 luci-app-openvpn - git-21.096.73306-254083c-1 luci-app-opkg - git-21.096.73306-254083c-1 luci-app-simple-adblock - git-21.050.37945-c33df8f-1 luci-app-sqm - 1.4.0-2 luci-app-statistics - git-21.096.73306-254083c-1 luci-base - git-21.096.73306-254083c-1 luci-compat - git-21.096.73306-254083c-1 luci-lib-ip - git-21.096.73306-254083c-1 luci-lib-iptparser - git-21.096.73306-254083c-1 luci-lib-jsonc - git-21.096.73306-254083c-1 luci-lib-nixio - git-21.096.73306-254083c-1 luci-mod-admin-full - git-21.096.73306-254083c-1 luci-mod-network - git-21.096.73306-254083c-1 luci-mod-status - git-21.096.73306-254083c-1 luci-mod-system - git-21.096.73306-254083c-1 luci-proto-ipv6 - git-21.096.73306-254083c-1 luci-proto-ppp - git-21.096.73306-254083c-1 luci-theme-bootstrap - git-21.096.73306-254083c-1 mtd - 24 netifd - 2021-01-09-753c351b-1 odhcp6c - 2021-01-09-64e1b4e7-16 odhcpd-ipv6only - 2020-05-03-49e4949c-3 openwrt-keyring - 2019-07-25-8080ef34-1 opkg - 2021-01-31-c5dccea9-1 ppp - 2.4.7.git-2019-05-25-3 ppp-mod-pppoe - 2.4.7.git-2019-05-25-3 procd - 2020-03-07-09b9bd82-1 rpcd - 2020-05-26-67c8a3fd-1 rpcd-mod-file - 2020-05-26-67c8a3fd-1 rpcd-mod-iwinfo - 2020-05-26-67c8a3fd-1 rpcd-mod-luci - 20201107 rpcd-mod-rrdns - 20170710 rrdtool1 - 1.0.50-3 simple-adblock - 1.8.4-10 sqm-scripts - 1.4.0-2 swconfig - 12 tc - 5.0.0-2.1 ubox - 2019-06-16-4df34a4d-3 ubus - 2019-12-27-041c9d1c-1 ubusd - 2019-12-27-041c9d1c-1 uci - 2019-09-01-415f9e48-4 uclient-fetch - 2020-06-17-51e16ebf-1 uhttpd - 2020-10-01-3abcc891-1 urandom-seed - 1.0-1 urngd - 2020-01-21-c7f7b6b6-1 usign - 2020-05-23-f1f65026-1 wireless-regdb - 2020.11.20-1 wpad-basic - 2019-08-08-ca8c2bd2-7 zlib - 1.2.11-3
@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Apr 8, 2021

kousu:

Here's a prototype of my first suggestion; given the reported conflicts above, I translated them into uci's cli syntax, which implicitly imposes a canonical format:

root@router:~# TMP=$(mktemp -d) root@router:~# uci show luci > $TMP/luci root@router:~# uci show luci-opkg | sed 's/^luci-opkg/luci/' > $TMP/luci-opkg root@router:~# diff -u $TMP/luci-opkg $TMP/luci --- /tmp/tmp.LNmpnm/luci-opkg 2021-04-08 11:07:26.422562972 -0400 +++ /tmp/tmp.LNmpnm/luci 2021-04-08 11:07:14.892396639 -0400 @@ -18,8 +18,13 @@ luci.ccache=internal luci.ccache.enable='1' luci.themes=internal +luci.themes.Bootstrap='/luci-static/bootstrap' luci.apply=internal luci.apply.rollback='90' luci.apply.holdoff='4' luci.apply.timeout='5' luci.apply.display='1.5' +luci.diag=internal +luci.diag.dns='openwrt.org' +luci.diag.ping='openwrt.org' +luci.diag.route='openwrt.org'

That diff is much easier to read and merge manually.

Here's a prototype of my second idea, by getting uci to regenerate the file:

root@router:~# cp /etc/config/luci-opkg /etc/config/luci-opkg-canonical root@router:~# uci show luci-opkg-canonical | xargs -n 1 uci set && uci commit root@router:~# diff -u /etc/config/luci-opkg-canonical /etc/config/luci-mine --- /etc/config/luci-opkg-canonical 2021-04-08 11:01:48.000000000 -0400 +++ /etc/config/luci-mine 2021-01-30 05:15:01.000000000 -0500 @@ -24,6 +24,7 @@ option enable '1'

config internal 'themes'

  • option Bootstrap '/luci-static/bootstrap'

config internal 'apply'
option rollback '90'
@@ -31,3 +32,8 @@
option timeout '5'
option display '1.5'

+config internal 'diag'

  • option dns 'openwrt.org'
  • option ping 'openwrt.org'
  • option route 'openwrt.org'

You can see that both of these are the same diff, just one is in the cli format and the other is in the config file format. uci took care of replacing tabs and other extra whitespace with single spaces, and double quotes with single quotes, so that the files are directly comparable.

What would you say to making resolve_conffiles run uci show $CONFFILE-opkg | xargs -n 1 uci set && uci commit as a first step?

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Apr 8, 2021

kousu:

Another strategy is to adopt what desktop linux distros do: have /etc/$something.d/ directories, whose contents are merged virtually by uci. That way you can isolate changes made by the user without having to edit the files installed from packages.

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Apr 9, 2021

kousu:

On ucitrack, the first prototype works directly,

root@router:~# uci show ucitrack > /tmp/b root@router:~# uci show ucitrack-opkg | sed 's/-opkg//' > /tmp/a root@router:~# diff -u /tmp/a /tmp/b --- /tmp/a 2021-04-08 11:32:41.281049289 -0400 +++ /tmp/b 2021-04-08 11:32:03.460601813 -0400 @@ -1,11 +1,11 @@ ucitrack.@network[0]=network ucitrack.@network[0].init='network' -ucitrack.@network[0].affects='dhcp' +ucitrack.@network[0].affects='dhcp' 'radvd' ucitrack.@wireless[0]=wireless ucitrack.@wireless[0].affects='network' ucitrack.@firewall[0]=firewall ucitrack.@firewall[0].init='firewall' -ucitrack.@firewall[0].affects='luci-splash' +ucitrack.@firewall[0].affects='luci-splash' 'qos' 'miniupnpd' 'sqm' ucitrack.@olsr[0]=olsr ucitrack.@olsr[0].init='olsrd' ucitrack.@dhcp[0]=dhcp @@ -24,9 +24,9 @@ ucitrack.@system[0]=system ucitrack.@system[0].init='led' ucitrack.@system[0].exec='/etc/init.d/log reload' -ucitrack.@system[0].affects='luci_statistics' +ucitrack.@system[0].affects='luci_statistics' 'dhcp' ucitrack.@luci_splash[0]=luci_splash -ucitrack.@luci_splash[0].init=''\''luci_splash'\''' +ucitrack.@luci_splash[0].init='luci_splash' ucitrack.@upnpd[0]=upnpd ucitrack.@upnpd[0].init='miniupnpd' ucitrack.@ntpclient[0]=ntpclient @@ -35,3 +35,7 @@ ucitrack.@samba[0].init='samba' ucitrack.@tinyproxy[0]=tinyproxy ucitrack.@tinyproxy[0].init='tinyproxy' +ucitrack.@sqm[0]=sqm +ucitrack.@sqm[0].init='sqm' +ucitrack.@luci_statistics[0]=luci_statistics +ucitrack.@luci_statistics[0].init='luci_statistics'

but I is harder to apply back to the actual system because it doesn't give me mergeable config files directly. Unfortunately my second prototype didn't work too well on ucitrack:

root@router:~# uci show ucitrack-opkg | xargs -n 1 uci set && uci commit uci: Entry not found uci: Entry not found uci: Entry not found uci: Invalid argument

I think this is probably because this file uses this syntax which can be misinterpreted by the shell:

root@router:~# uci show ucitrack-opkg ucitrack-opkg.@network[0]=network ucitrack-opkg.@network[0].init='network' ucitrack-opkg.@network[0].affects='dhcp' ucitrack-opkg.@wireless[0]=wireless [...] ucitrack-opkg.@samba[0]=samba ucitrack-opkg.@samba[0].init='samba' ucitrack-opkg.@tinyproxy[0]=tinyproxy ucitrack-opkg.@tinyproxy[0].init='tinyproxy'

So here's a second revision of my second prototype using uci batch:

root@router:~# uci show ucitrack-opkg | sed 's/^/set /' | uci batch root@router:~# uci changes ucitrack-opkg.cfg0102af='network' ucitrack-opkg.cfg02e233='wireless' ucitrack-opkg.cfg03b57b='firewall' ucitrack-opkg.cfg04c845='olsr' ucitrack-opkg.cfg05ad04='dhcp' ucitrack-opkg.cfg06f057='odhcpd' ucitrack-opkg.cfg074dd4='dropbear' ucitrack-opkg.cfg088cc9='httpd' ucitrack-opkg.cfg09cd75='fstab' ucitrack-opkg.cfg0aa258='qos' ucitrack-opkg.cfg0be48a='system' ucitrack-opkg.cfg0c003c='luci_splash' ucitrack-opkg.cfg0d804c='upnpd' ucitrack-opkg.cfg0e8036='ntpclient' ucitrack-opkg.cfg0f0f89='samba' ucitrack-opkg.cfg10822b='tinyproxy' root@router:~# uci commit root@router:~# diff -u /etc/config/ucitrack-opkg /etc/config/ucitrack --- /etc/config/ucitrack-opkg 2021-04-08 11:39:20.000000000 -0400 +++ /etc/config/ucitrack 2021-04-08 10:18:05.000000000 -0400 @@ -1,21 +1,25 @@

config network
option init 'network'

  • option affects 'dhcp'
  • list affects 'dhcp'
  • list affects 'radvd'

config wireless

  • option affects 'network'
  • list affects 'network'

config firewall
option init 'firewall'

  • option affects 'luci-splash'
  • list affects 'luci-splash'
  • list affects 'qos'
  • list affects 'miniupnpd'
  • list affects 'sqm'

config olsr
option init 'olsrd'

config dhcp
option init 'dnsmasq'

  • option affects 'odhcpd'
  • list affects 'odhcpd'

config odhcpd
option init 'odhcpd'
@@ -35,10 +39,11 @@
config system
option init 'led'
option exec '/etc/init.d/log reload'

  • option affects 'luci_statistics'
  • list affects 'luci_statistics'
  • list affects 'dhcp'

config luci_splash

  • option init ''''luci_splash''''
  • option init 'luci_splash'

config upnpd
option init 'miniupnpd'
@@ -52,3 +57,9 @@
config tinyproxy
option init 'tinyproxy'

+config sqm

  • option init 'sqm'

+config luci_statistics

  • option init 'luci_statistics'

I now aspire the patch to be adding

uci show $PKG-opkg | sed 's/^/set /' | uci batch && uci commit

into resolve_conffiles.

@openwrt-bot
Copy link
Author

@openwrt-bot openwrt-bot commented Apr 16, 2021

kousu:

Just answering my own question a bit over here, I saw [[https://forum.openwrt.org/t/is-opkg-upgrade-still-a-bad-idea/93501/|this thread]] and the related [[https://openwrt.org/meta/infobox/upgrade_packages_warning|wiki]] [[https://openwrt.org/docs/guide-user/installation/generic.sysupgrade#savingrestoring_user-installed_packages|pages]] and which explains why this code path hasn't bothered others.

It's because the standard upgrade method is sysupgrade, with optional package list saving and reinstalling via [[https://openwrt.org/docs/guide-user/advanced/opkg_extras|this bonus script]].

This also explains to me why I keep having to do sysctl -w vm.overcommit_memory=1 [[https://bugs.openwrt.org/index.php?do=details&task_id=2734|just to run updates]].

@aparcar aparcar added the release/19.07 label Feb 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
flyspray release/19.07
Projects
None yet
Development

No branches or pull requests

2 participants