Bricked device: allow patching of DNS servers and SSH keys from inside the container

[magiconair]

We have several Balena OS devices with default configuration going onto a customer network which blocks DNS to 8.8.8.8 and the IT department cannot change that rule. This is the default DNS server for balenaOS and is not documented on

https://www.balena.io/docs/faq/troubleshooting/faq/#what-network-ports-are-required

My assumption is that this is a non-issue if you are using NetworkManager since it will update the dnsmasq config with the DNS servers from the DHCP response.

However, we need both Client and AP mode for WiFi simultaneously which NetworkManager does not support. We have to unmanage the WiFi interface and use wpa_supplicant directly. Because of this we cannot set the DNS server for the host OS which prevents Balena from establishing the tunnel, basically bricking the device.

Additionally, our devices do not have a removable SD card so patching /mnt/boot/config.json with additional DNS servers was not possible. Switching the device and re-flashing were also not possible or too time consuming because of where the device was installed.

Eventually, I've worked around this is by creating a WiFi access point on an RPi with forwarding and masquerading which also spoofs a DNS server on 8.8.8.8 (configure 8.8.8.8 on the loopback interface and run dnsmasq as proxy on the wlan0 interface).

Having a way to patch the dnsServers variable of the config.json from inside a container would be very helpful. The same goes for adding an ssh key so that someone can login as root on port 22222. This looks like a good place: https://www.balena.io/docs/reference/supervisor/supervisor-api/#patch-v1devicehost-config

However, this only helps if you have thought about this problem beforehand. If the device is already in the field Balena should try very hard to establish a tunnel. I think the current approach can be improved:

1. Update the network FAQ to include that both TCP and DNS must work for Balena to establish a tunnel and that if no DNS server could be acquired Balena uses 8.8.8.8 as a last resort.
2. Add some static ip addresses to /etc/hosts and add them as additional remote options to the OpenVPN configuration. There should be a hard-coded static list in the OS image and the OS should cache the response from host vpn.balena-cloud.com also in /etc/hosts whenever it is online. Best to use names which do not resolve on the internet, e.g. vpn-static1, ... and so forth.

This would increase the chance that the device establishes a tunnel so that it can be managed remotely again.

[lurch]

Also related? balena-io/docs#1122

[ZubairLK]

@magiconair

Setting dns servers, sshkeys and udev rules via supervisor are on the todo list.
See

• Allow setting udev rules from the dashboard, via special config.txt like variables balena-supervisor#998
• Expose the dnsServers config.json field over environment variable balena-supervisor#702

There was a subtle change in the default behaviour at one point. See balena-os/meta-balena@fa3de69
Depending on the OS version you are running, you should have that

The static ip address for vpn is an interesting one. I'll raise it internally.

And I'll raise the docs concern as well. The firewall docs should mention 8.8.8.8. I'll raise that too.

Thanks

[magiconair]

We have that. The problem is that editing the config.json is major open-heart surgery on a remote device since one small mistake will brick the device since it then won't have any DNS server.

[ZubairLK]

@magiconair indeed

Until the supervisor updates, we have a few repos that are slightly safer then open heart surgery
See

• https://github.com/balena-io-playground/ssh-key-insert
• https://github.com/balena-io-playground/udev-rule-insert

Don't have a ready to use one for dnsServer but should be possible to use those as a base

[ZubairLK]

The other day I was thinking of adding a bash script in the OS for config.json updates.

Something that makes a copy of config.json. allows edits. and only saves if it passes some jq sanity checks..

⏳

[lurch]

Something that makes a copy of config.json. allows edits. and only saves if it passes some jq sanity checks.

Interesting idea. Would using JSON Schema for additional validation be overkill and / or too inflexible?

[ZubairLK]

At this point, keeping it lightweight and just using jq should cover most basic mistakes and would be a nice step improvement from the current state..

[ZubairLK]

Made this balena-os/meta-balena#1697

[ZubairLK]

There are some issues that need to be sorted out for the balena tunnel command.
But here is a start for a totally untested and unsafe script that might make it slightly safer to edit config.json

#!/bin/bash

set -e
set -o pipefail

trap "trap - SIGTERM && kill -- -$$" SIGINT SIGTERM EXIT

if ! which balena > /dev/null 2>&1 ; then
	echo "Please install balena-cli"
fi

if balena tunnel > /dev/null 2>&1 ; then
	echo "Please update balena-cli"
fi

if ! balena whoami > /dev/null 2>&1 ; then
	echo "Please run balena login"
fi

if [ -z "$1" ]; then
	echo "Please pass device UUID"
	exit 1
fi

DEVICE_UUID="$1"

LOCAL_PORT=54321

USERNAME=`balena whoami | head -n2 | tail -n1 | cut -d ' ' -f 2`

echo "Setting up tunnel to device using local port $LOCAL_PORT"

balena tunnel "$DEVICE_UUID" -p 22222:$LOCAL_PORT &

sleep 10

echo "Waiting 10 seconds for tunnel to go live"

FILENAME="config.json.$DEVICE_UUID"

# Can use balenaCloudUsername@device ssh after v2.44 if balenaCloud account has ssh key

scp -P $LOCAL_PORT "$USERNAME"@127.0.0.1:/mnt/boot/config.json "$FILENAME"

cat "$FILENAME" > jq . > "$FILENAME.tmp"

vim "$FILENAME.tmp"

diff "$FILENAME" "$FILENAME.tmp"

scp -P $LOCAL_PORT "$FILENAME.tmp" "$USERNAME"@127.0.0.1:/mnt/boot/config.json.tmp

ssh -P $LOCAL_PORT "$USERNAME"@127.0.0.1 'cat /mnt/boot/config.json.tmp | jq .'

ssh -P $LOCAL_PORT "$USERNAME"@127.0.0.1 'cp /mnt/boot/config.json /mnt/boot/config.json.old.DATE'

ssh -P $LOCAL_PORT "$USERNAME"@127.0.0.1 'mv /mnt/boot/config.json.tmp /mnt/boot/config.json'

[lurch]

You probably want the echo "Waiting 10 seconds for tunnel to go live" before the sleep 10 ? 😉

EDIT: And maybe it's worth running jq on the local $FILENAME.tmp (to check that it's valid JSON) before scp-ing it to the remote device?