Lock down of Open Mesh devices
TL;DR (short summary)
- Starting from around the middle of 2016, all dual-band Open Mesh devices are locked down due to FCC requirements.
- For devices used in USA or Canada, the lock down includes not only protection against using custom firmware but also SSH access and requirement for approval of custom.sh scripts.
- IP geolocation seems to be used to recognize in what country device operates. IP address is checked constantly and SSH access is granted only in devices outside USA and Canada.
- U-Boot command line interface in locked down devices is disabled, flashing with ap51-flash tool requires providing fwupgrade.cfg.sig (signature of fwupgrade.cfg). Without providing the signature or if verification fails, upgrade process is cancelled.
- RSA public key is stored in second half of the last 64 KB sector in FLASH (/dev/mtd7 inside the system). It is enough to remove the key or its header to disable the firmware lock down.
- All mtd partitions are writable in vendor's firmware.
- Use of VPN service with IP outside USA or Canada is enough to fool CloudTrax service and enable SSH access.
- SSH password is available only in CloudTrax service after registering the device.
Somewhere around the middle of 2016, Open Mesh announced lock down of their dual-band devices, including MR1750 and OM5P-AC.
The lock down is related with FCC requirements and mostly concerns customers located in USA and Canada, as we can read in the official announcement Dual band changes due to FCC requirements:
Due to new IC and FCC requirements, we were forced to make changes to the way our dual-band devices work. Starting today, any new orders of MR1750 shipped to the United States will include a modified version to comply with the new FCC regulations.
The restrictions are the same as those on the OM5P-AC device, and include:
- Custom firmware versions can no longer be loaded onto the device.
- SSH access is now disabled for networks located within the United States and Canada.
- Custom.sh scripts must be approved and hosted by Open Mesh to ensure they do not take the device out of compliance.
Existing devices are not affected.
For international customers, please order part MR1750-INTL. Once current stock is depleted, the FCC version will be available globally but will not have restrictions 2 and 3 (above) when used outside the US. Note that the hardware of the two models is identical.
We're strong believers in open source software** and did not take this decision lightly.
It is not our role to judge this decision. The most important part of this announcement is that, no matter of what part of the world device is used in, all users of dual-band Open Mesh devices are no longer able to install and use custom firmware like LEDE, OpenWrt or DD-WRT.
Actually, as we are going to show later, this statement is not fully true. During our small research, we found out how the Open Mesh firmware lock down works and how it can be easily disabled. For devices with enabled SSH access it is just a matter of few commands and for fully locked devices, we prepared some kind of automatic tool.
We were wondering how does Open Mesh recognize in what country their devices are located. We found a little hint in their knowledge base: "Unknown access point location" Alert:
In order to comply with local regulatory restrictions, we'll need to detect the region in which your access points currently reside. If the IP geo-location cannot be determined, you'll see this alert when hovering over an AP's icon at Manage -> Access Points: [...]
With a help of several free VPN services, we were able to confirm that they actually must use somehow IP geolocation and enable or disable SSH access based on that. For devices connecting with CloudTrax service from IP located in USA or Canada, SSH access will be deactivated. It is done by stopping and disabling Dropbear service. For devices visible on IP outside USA and Canada, Dropbear is enabled and started:
root@om5p-ac_test:/# ls /etc/rc.d/ -l | grep -i drop lrwxrwxrwx 1 root root 18 Jan 22 16:26 K50dropbear -> ../init.d/dropbear lrwxrwxrwx 1 root root 18 Jan 22 16:26 S50dropbear -> ../init.d/dropbear root@om5p-ac_test:/# pidof dropbear 2580
The bad news here is that the IP is checked all the time and SSH access is not setup just once. We suppose that a custom script which will constantly check if Dropbear process is alive and restarts it if needed should be enough to handle SSH lock.
Except automatic OTA updates in CloudTrax, Open Mesh devices can be also updated locally, with ap51-flash/open-mesh-flash-ng tool. It uses a rather simple update protocol (TFTP based), used also in many other devices. Support for this kind of upgrade process was included in OpenWrt long time ago and is the easiest and fastest way for flashing firmware (not only custom ones) on devices supporting it.
Before the lock down, during the upgrade with ap51-flash or open-mesh-flash-ng tool, Open Mesh router was first asking TFTP server for fwupgrade.cfg which is a simple text file containing information about firmware parts being updated, as in example below:
[vmlinux] filename=kernel md5sum=8fafac44de8c9bd04c695893514ef873 filemd5sum=8fafac44de8c9bd04c695893514ef873 filesha256sum=c0b18205a862c4ef6fe6cd239a67a34d6c9592293307ac11ab0ea68f6bf329af flashaddr=0xb0000 checksize=0x0 cmd_success=setenv bootseq 1,2; setenv kernel_size_1 1152; saveenv cmd_fail=reset [rootfs] filename=rootfs md5sum=7e4211fe0b68d28d70710eaa7e4a543a filemd5sum=3adedcdf148ea0ba72565d224d313623 filesha256sum=80c3e19fff06083666cd4a9740ec0484dd3726cf12f2e96a74be31fe0e93fc47 flashaddr=0x1d0000 checksize=0x340000 cmd_success=setenv bootseq 1,2; setenv kernel_size_1 1152; setenv rootfs_size_1 6656; saveenv cmd_fail=reset
In case of locked devices, before the router starts parsing content of fwupgrade.cfg, it asks TFTP server for another file: fwupgrade.cfg.sig. As you can guess, this one contains encrypted hash (digital signature) of the fwupgrade.cfg. If the signature file is not provided or signature verification fails, upgrade process is canceled and the device just reboots.
Example open-mesh-flash-ng output from a successful update (with embedded NG firmware) on locked Open Mesh OM5P-AC device:
[ac:86:74:66:58:40]: type 'OM5P-AC router' detected [ac:86:74:66:58:40]: OM5P-AC router: tftp client asks for 'fwupgrade.cfg', serving fwupgrade.cfg-OM5PAC portion of: embedded image (2 blocks) ... [ac:86:74:66:58:40]: OM5P-AC router: tftp client asks for 'fwupgrade.cfg.sig', serving fwupgrade.cfg-OM5PAC.sig portion of: embedded image (2 blocks) ... [ac:86:74:66:58:40]: OM5P-AC router: tftp client asks for 'kernel', serving kernel portion of: embedded image (2183 blocks) ... [ac:86:74:66:58:40]: OM5P-AC router: tftp client asks for 'rootfs', serving rootfs portion of: embedded image (7169 blocks) ... [ac:86:74:66:58:40]: OM5P-AC router: image successfully transmitted - writing image to flash ... [ac:86:74:66:58:40]: OM5P-AC router: flash complete. Device ready to unplug.
And log of unsuccessful upgrade to LEDE:
[ac:86:74:66:58:40]: type 'OM5P-AC router' detected [ac:86:74:66:58:40]: OM5P-AC router: tftp client asks for 'fwupgrade.cfg', serving fwupgrade.cfg portion of: lede-ar71xx-generic-om5pac-squashfs-factory.bin (2 blocks) ... [ac:86:74:66:58:40]: OM5P-AC router: tftp client asks for 'fwupgrade.cfg.sig' - file not found ... [ac:86:74:66:58:40]: OM5P-AC router: tftp client asks for 'fwupgrade.cfg.sig' - file not found ... [ac:86:74:66:58:40]: OM5P-AC router: tftp client asks for 'fwupgrade.cfg.sig' - file not found ... [ac:86:74:66:58:40]: OM5P-AC router: tftp client asks for 'fwupgrade.cfg.sig' - file not found ... [ac:86:74:66:58:40]: OM5P-AC router: tftp client asks for 'fwupgrade.cfg.sig' - file not found ... [ac:86:74:66:58:40]: OM5P-AC router: tftp client asks for 'fwupgrade.cfg.sig' - file not found ...
RSA public key, U-Boot sources
It was obvious that the locked device must contain some kind of public key which is used for fwupgrade.cfg signature verification before the upgrade process.
As the OM5P-AC includes only UART header and the U-Boot command line interface is also disabled in locked device, we started analyzing original U-Boot image on similar (from hardware point of view) device: 8devices Rambutan Development Kit.
With the help of JTAG and a bit of luck, we found out that that the RSA public key is stored in the second part of last 64 KB sector in FLASH chip. Moreover, it turned out that removing the key is enough to fully unlock the device. For users of devices with SSH access available it is a straightforward process, because all mtd partitions, including the one with RSA public key inside, are writable in the vendor's firmware.
Roughly the same time, after our request based on the fact that U-Boot is licensed under GPL, Open Mesh provided us with the sources. Surprisingly, even the fact that the code is for Open Mesh MR1750 device, it is complete and includes parts related with update and signature verification process. Code analysis confirmed what we discovered before and allowed us to focus on the most important thing: find an easy way to unlock a device without SSH access.
At this point we also started to ask ourselves why Open Mesh left here two obvious (more or less) security holes:
- Store RSA public key in a separate FLASH sector and lock device based on its presence.
- Setup all mtd partition as writable.
The reason for the first point may be related with backward compatibility as only the dual-band devices sold after middle of 2016 are supposed to be locked. Second point shows rather unusual approach as the general rule is to keep critical data in read-only partitions. RSA public key is stored in the same sector where both WiFi radios calibration data (Atheros Radio Test/ART, unique per every device) resides. Actually, we asked on LEDE-DEV mail list about unlocked mtd partitions and got answer from Sven Eckelmann:
[...] You or someone else for OpenMesh has put so much energy to make the U-Boot use RSA keys for image verification, but on the other hand you keep the ART partition, where you store the RSA key, writable from the system. [....]
If you have full access to the system then you have full access to the system. I don't understand why having full access to the system should mean that you don't have full access to the system.
If you remember Open Mesh announcement from the begin of this page, all dual-band devices should no longer allow to load custom firmware. But, with SSH access available, unlocked all mtd partitions and knowledge where RSA public key is stored (sooner or later someone would find it)... this protection against loading custom firmware is just illusory.
Furthermore, for exchanging firmware, root access (e.g. over SSH) is in most cases enough. Even if interesting mtd partitions are set to be read-only, there are still tools like mtd-rw.
Question here is: is this weak protection intentional?