RFC: Building 4MB Images

[pmelange]

The new target ar71xx-tiny (issue #514) and Images "default4MB" too big (issue #314) have not really been resolved. It seems that, from the mailing list, there is still interest in supporting 4MB models.

There has recently been some work on the openwrt and luci repos to make things smaller again (like luci-mod-admin-mini). It might also be possible to remove some other dependencies.

For example, I don't know if we need the following:
kmod-gpio-button-hotplug
luci-app-owm
luci-app-owm-cmd
luci-lib-luaneightbl

Other possibilities are:
Reducing the number of community profiles installed by default on the router

Using luci-theme-openwrt instead of bootstrap (saves 4K)
https://openwrt.org/docs/guide-user/luci/luci.themes

Dropping migration for 4MB devices
freifunk-berlin-migration

No longer having the OLSR status pages
luci-app-olsr
luci-app-olsr-services
olsrd-mod-jsoninfo

Using lighthttp instead of uhttp.

Dropping IPv6 support?
ip6tables
kmod-nf-conntrack6
kmod-nf-ipt6
kmod-nf-reject6
kmod-udptunnel6
odhcpd-ipv6only
libip6tc

I'm sure that if we, as a community, put our heads together, we will come up with a solution.

[SvenRoederer]

dropping IPv6-support is really a bad idea, as this is the network-protocol of the time / future.
dropping OWM will make such nodes disappear on the map.
What will be the space gained from dropping luci-olsr? Will the lost comfort be an advantage for the user?

[SvenRoederer]

there is also freifunk-berlin/buildbot#47 pending

[pmelange]

We could also look into using luasrcdiet to reduce the size of the lua source code.

http://luasrcdiet.luaforge.net/

The luci code currently uses this.

[sarumpaet]

See #283 (comment)

Remove/rework qos, then iptables can get removed, which is 100k.

Migration and OWM are tiny and get compressed to almost nothing in the image anyways.

[pmelange]

@sarumpaet
I don't know how we can remove iptables. With the no-tunnel, tunnel-berlin-openvpn and tunel-berlin-tunneldiger versions, we masquerade. Only the deprecated VPN03 version doesn't need to masquerade, and that is because of sven-ola's hack of an old version of OpenVPN on the server.

The tunnel-berlin-openvpn version we could work around by reimplementing the sven-ola hack on the server side (anybody know where to find this hack?). But I'm not sure how to do that with tunneldigger (but it could surely be possible). I don't think it would be possible with no-tunnel.

Additionally, how would it work with OLSRd? The daemon added the following rule on my router (backbone image): -A forwarding_rule -i tunl0 -o tnl_XXXXXX -j ACCEPT. Doesn't that mean that OLSRd is dependent on iptables?

[pmelange]

@SvenRoederer
Yes, I totally agree that IPv6 is the present/future. But on the other hand, 4MB routers are not the present, but the past. If the tiny routers can only do IPv4, then at least they would be able to something.

[SvenRoederer]

Probably olsrd is using netlink directly to setup netfilter-rules.
the qos-package can probably replaced by simple-tc which do not depend on iptables (#283 (comment))

[sarumpaet]

@pmelange NAT is set up via fw3, not via iptables (since a long time). See #283 (comment) .

[pmelange]

I just played around a bit with a test router. I successfully removed qos-scripts, iptables, ip6tables, iptables-mod-ipopt, and iptables-mod-conntrack-extra.

After reboot, everything seems to be working fine with the uplink (tunneldigger).

Then I reconfigued to mesh with the Backbone. This also works fine. I can reach all nodes on the Backbone and also reach the internet.

As another test, I set up two nodes, both without the above packages. The non-uplink node properly forwards to the internet trough the uplink node.

I find this a viable solution.

---

One thing that surprised me is that when I run fw3 print I get lots of commands listed out, and all of them start with iptables. WTF?

I have an issue with simple-tc. Yes, it's in gluon and we could/should use it. But where did it come from? The source code seems to come out of nowhere. It might be better to find the original source. Here is the initial commit in gluon freifunk-gluon/packages@48cddbb#diff-804469766ebbca672fc17773e7f97e67

Also, the copyright states that we need to reproduce it, with all terms and conditions somewhere. Where should the copyright be posted?

[pmelange]

Adding simple-tc to the uplink node was quite painless. The one thing I dislike is that runs via hoplug and doesn't put any log entries out.

It would be nice to be able to reload simple-tc by running something (e.g. /etc/init.d/simple-tc restat). Also, it might be a good idea to have it set up with ucitrack to reload itself.

[pmelange]

I have created a branch which uses simple-tc instead of qos-scripts.

Here are the changes to firmware-packages
https://github.com/freifunk-berlin/firmware-packages/tree/simple-tc

The changes to the firmware repo I still have only locally. I can push them if it interests anybody.

Here are the packages which should be installed (no-tunnel):

Building images for ar71xx - TP-LINK TL-MR3020 v1
Packages: freifunk-berlin-dhcp-defaults freifunk-berlin-firewall-defaults 
freifunk-berlin-freifunk-defaults freifunk-berlin-migration freifunk-berlin-network-defaults 
freifunk-berlin-olsrd-defaults freifunk-berlin-system-defaults community-profiles dnsmasq 
simple-tc firewall iwinfo libiwinfo-lua uhttpd uhttpd-mod-ubus luci-app-ffwizard-berlin 
luci-mod-freifunk-ui luci-app-olsr luci-app-olsr-services luci-app-owm luci-app-owm-cmd 
luci-theme-bootstrap olsrd olsrd-mod-arprefresh olsrd-mod-dyn-gw olsrd-mod-jsoninfo 
olsrd-mod-nameservice kmod-ipip freifunk-berlin-uplink-notunnel-files base-files busybox 
dnsmasq dropbear firewall fstools kernel kmod-ath9k kmod-gpio-button-hotplug 
kmod-usb-ledtrig-usbport libc libgcc logd mtd netifd odhcp6c odhcpd-ipv6only swconfig 
uboot-envtools uci uclient-fetch wpad-mini

but still it's too big:

4821+1 records in
4821+1 records out
2468510 bytes (2.5 MB, 2.4 MiB) copied, 0.0549763 s, 44.9 MB/s
[mktplinkfw] rootfs offset aligned to 0x1241580
[mktplinkfw] *** error: images are too big by 40072 bytes

The entire buildlog is available (for the first device only) if anyone wants it.

---

How can we get the 4MB images working again? Anyone have any other ideas?

[pmelange]

I just realized that since I used an imagebuilder from another build, I am also pulling in opkg. (I origionally added luci-app-opkg as a dependant of luci-mod-freifunk-ui) That is over 50k that would be saved.

I'm going to work on getting this....

[pmelange]

I have built, but not yet tested the no-tunnel and tunneldigger images. The backbone image is still too big (by 181K).

opkg was still in the old 4MB devices. But still, that's not a lot of savings. Should we drop opkg and batman?

[pmelange]

I tried the tunneldigger version of the 4MB image with simple-tc. There isn't enough room leftover to configure the device.

[pmelange]

OLSRd is dependant on iptables
https://github.com/openwrt-routing/packages/blob/3d97690ba39af930b6456ef9901d5037ae0e11eb/olsrd/files/olsrd.init#L702

[SvenRoederer]

I assume, we don't need this section at all in our setup

[sarumpaet]

@pmelange Again, see #283 (comment) . The commit referenced there added NAT for tnl_+ (wildcard!) devices which should cover OLSRd SmartGateway. If it doesn't work (it did back then I think), we could also simply disable SmartGW on 4MB devices (don't quite a few people routinely disable SmartGW anyways as it seems not to be too stable?).

[pmelange]

I think spending time to scrape off more and more just to accommodate a bigger kernel is not time well spent.

https://openwrt.org/supported_devices/432_warning
https://forum.openwrt.org/t/lede-a-bit-over-the-top-with-the-minimal-requirements/2009/4?u=tmomas

I don't plan on submitting the branch to change to simple-tc in the comment above as a PR.

[SvenRoederer]

@pmelange you spoke about interest for support on the ML, but for long time there's no activity.
I suggest to that the developers announce on the ML, that we want to drop support for these devices with the next release. Probably this will give someone the right mood to invest time in these devices. Or they die silently ...

[kls0e]

I think, while it is sustainable, it is also fun to make use of old hardware. And as we know, there is plenty of spare 4/32 routers around. My suggestion for viable 4MB image compilation is

-dropping uhttpd and luci
-creating a cli-based wizard setup script called cliwiz that does the job of the luci wizard

Do you think this is feasible?

[SvenRoederer]

@kls0e Feel free to start with such project.

I suggest to use lua for the code, so we can refactor common-code to reuse it from the web-wizard.

[SvenRoederer]

@kls0e @Akira25 let me mention the wizard2. As far as I remember, only the backend (https://github.com/freifunk-berlin/firmware-packages/tree/master/utils/freifunk-berlin-wizard-backend) needs to be present on the router and the frontend can run on a separate machine.
@andrenarchy @andibraeu @lynxis should be able to tell more

[sarumpaet]

Yes, the new wizard was supposed to open a way to drop uhttpd and luci, among other things. Not sure what's missing to switch over to it at last.

And as we know, there is plenty of spare 4/32 routers around.

I'm all for supporting 4MB ROM devices, but it's really not as urgent as it used to be. Currently there are less than 50 such devices active; only 32MB RAM will always be a problem; and, if you really want to, replacing the flash chip on these devices is relatively easy and quick to do (replacing the RAM is also possible but much more hassle).

But by all means, do play a bit with the source. @kls0e

[andibraeu]

@kls0e @Akira25 let me mention the wizard2. As far as I remember, only the backend (https://github.com/freifunk-berlin/firmware-packages/tree/master/utils/freifunk-berlin-wizard-backend) needs to be present on the router and the frontend can run on a separate machine.
@andrenarchy @andibraeu @lynxis should be able to tell more

At the moment you can consider wizard2 as dead. There was no further development since 2017 and no interest by the community using it.

[SvenRoederer]

Using lighthttp instead of uhttp.

I checked the sizes of the binaries and lighttpd seems much larger than uhttpd (and it's dependencies). The depending packages of uhttpd are used by other packages anyway, so it's only the binaries size added to the image.
In addition I did some tests with an alternative httpd (SvenRoederer/openwrt-packages@ce38f80) which has no other dependencies and has an equal size as uhttpd.
So there is even a chance t make some minimalistic (static) Web-interface.

[sarumpaet]

@SvenRoederer If all you want to serve is one static HTML file you can as well do while true; do { echo -e 'HTTP/1.1 200 OK\r\n'; cat FILENAME; } | nc -l -p 80; done (this needs NC_SERVER turned on in busybox).

@andibraeu I don't quite understand. Clearly there is some interest in the new wizard, visible for example in this thread? I for my part certainly like the idea. The wizard's status, however, is totally unclear to me, i.e., how to get a build with it, what's working, what's not working, what are blockers.

[andibraeu]

To be honest, I don't know anymore what would be needed to get it running. The last commit is more than 2 years ago.

There were a lot of things missing, e.g. status pages. I think the frontend framework will also be outdated, ... I'm not sure, too, if it works with latest OpenWRT releases. And I don't have time to dig deeper into it.

[SvenRoederer]

@sarumpaet please check branch https://github.com/freifunk-berlin/firmware/commits/lede-wizard, which I kept to show the packages to build the new wizard. This branch should still create correct images (of the past state)

[kls0e]

I think, while it is sustainable, it is also fun to make use of old hardware. And as we know, there is plenty of spare 4/32 routers around. My suggestion for viable 4MB image compilation is

-dropping uhttpd and luci
-creating a cli-based wizard setup script called cliwiz that does the job of the luci wizard

Do you think this is feasible?

some Ideas from Freifunk Weimar: https://pretalx.35c3oio.freifunk.space/media/freifunk-4mb.pdf