Node OS Setup Guide Debian 8

SolarNode OS Setup Guide Debian 8

⚠️ WARNING: this is a very old guide, meant for advanced developers. If you are looking to get started with a SolarNode device, there are pre-built OS images for many popular devices.

This guide describes the steps I took to create a "minimal" Debian 8 based system with base SolarNode deployment. The overall goals were these:

1. No X window system.
2. No development tools.
3. No daemons or servers unless required by Debian or SolarNode.
4. SSH daemon for network access.
5. NTP daemon for time synchronization.
6. Boot from SD card.
7. Java 8 JRE.

With these goals in mind, let's dive in. You'll need a Linux-based system to work with.

Note: Binary images for the Node OS are also available here: http://sourceforge.net/projects/solarnetwork/files/solarnode These are great if you are after a quick OS setup (but they do not always contain the latest updates), if you want the latest and greatest you should continue reading below.

Download Debian 8 and copy to USB stick

Download the Debian 8 (jessie) installer CD. This should be the full 650MB CD, not the netboot image, so it can install the base system without needing the network. You can get the image from

http://www.debian.org/CD/http-ftp/#stable

Once you have the .iso image downloaded, copy the image to a USB stick. See https://www.debian.org/releases/stable/i386/ch04s03.html.en for information on copying the ISO image to a USB stick.

Boot into the Debian installer

Insert the Debian installer USB stick into the USB port on the back of the eBox and a 2GB or larger SD card into the SD slot. You'll need a monitor and keyboard (USB or PS/2) plugged in as well, but no mouse is required. Turn the node on, and quickly press the Delete key to enter the BIOS setup utility.

Make sure the BIOS boots from the USB stick first, under the Boot menu. The BIOS should list both the USB and SD disks, as USB: and HDD: options, respectively.

Next go to the Advanced > IDE Configuration screen and make sure the Standard IDE Compatible setting is set to Enabled.

Now you can exit the BIOS configuration - F10 to save and exit - and the Debian boot menu should appear. Choose the Expert install mode, and let the Debian installer load.

Debian install options

First I manually partitioned and formatted an ext4 filesystem, using a guide. I found the estimated block size of my SD card to be 4MB, so I halved all the settings from the guide:

• First sector: 16384 (8M)
• Size: 1600M (divisible by 8M and 4M)
• File system: mkfs.ext4 -O ^has_journal -E stride=2,stripe-width=512 -b 2048 -L SOLARNODE /dev/mmcblk0p1

Then here are some important options I chose during the Debian install:

1. DHCP for network configuration, with solarnode.localdomain as the host name.
2. No root user, with a single solar user.
3. No NTP time synchronization. The installer will install ntpd but we want to use systemd-timesyncd manually later, which is smaller and easier to manage.
4. Use the pre-existing ext4 partition created earlier, with a label SOLARNODE and the noatime option enabled. Also mark this partition bootable.
5. No swap partition. This is to minimize wear on the SD card.
6. Select the most recent available i586 kernel.
7. Configure a local apt mirror, without the non-free and without the contrib repositories used (they are not needed).
8. Install ssh server and sytem utilities but no other software at this time.
9. Install the Grub boot loader (but we will be replaced by extlinux later).

At this point, the installer should complete. You can reboot the node, but un-plug the USB stick before the BIOS RAM check completes, so the node boots from the newly minted SD card.

Post install tasks

1. Edit /etc/fstab to replace UUID=XXX with LABEL=SOLARNODE (to make it easier to share this image) so it should look like this:

    	LABEL=SOLARNODE / ext4 noatime,nodiratime,errors=remount-ro 0 1
   

2. Edit /etc/apt/sources.list to:

3. Remove the cdrom entry (which is really the USB stick).

4. Remove all -src sources.

5. Add deb http://ftp.nz.debian.org/debian jessie-backports main to support Java 8.

6. Run apt-get update.

7. Add solar user to dialout group, to allow use of serial ports.

    usermod -a -G dialout solar
   

8. Make /tmp mounted as a tmpfs (RAM) filesystem with:

    systemctl enable tmp.mount
    rm -rf /tmp/*
    systemctl start tmp.mount
   

9. Disable persistent history in bash, by editing /etc/bash.bashrc and adding

    unset HISTFILE
   

   When also installing the standard base SolarNode framework, the following can also be added:

    if [ -e /usr/share/solarnode/bash-utils.sh ]; then
            . /usr/share/solarnode/bash-utils.sh
    fi
   

10. Make Teletype the default frontend for debconf, because others are not installed, by creating /etc/profile.d/solarnode.sh:

     export DEBIAN_FRONTEND=teletype
    

Software setup

Now I manually removed and added the software I deemed appropriate for the node.

1. Replace rsyslog with busybox-syslogd, to minimize writing to the SD card:

    apt-get remove --purge rsyslog
    apt-get install --no-install-recommends busybox-syslogd
   

2. Install localepurge to remove excess locale data:#!sh

    apt-get install localepurge
   

3. Configure systemd-timesyncd

4. Edit /etc/systemd/timesyncd.conf and uncomment the SERVERS= line, optionally editing the default server list.

5. Run timedatectl set-ntp true

6. Remove nano and docs... any anything else you can, like exim, gcc, perl, python, etc.

    apt-get remove --purge nano info manpages
   

7. Install Java (this pulls in a whopping 250MB+ of stuff):

    apt-get -t jessie-backports install --no-install-recommends openjdk-8-jre-headless
   

8. Install rsync (required for node database backups)

    apt-get install --no-install-recommends rsync
   

9. Install RXTX and JNA Java libraries, to support serial ports in Java:

    apt-get install --no-install-recommends librxtx-java libjna-java
   

   Then follow the RXTX setup guide to configure RXTX with the JVM.

Further savings can be found by installing the deborphan and debfoster packages. Use those to identify non-essential packages and remove them.

WiFi support

If you plan to use WiFi, you'll need to install some firmware for the radio and install the wpasupplicant package.

1. apt-get install --no-install-recommends wpasupplicant

2. You need the rtl8712u.bin firmware file, which is available in git. You can clone this repository via

    git clone git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
   

   Then copy the linux-firmware/rtlwifi/rtl8712u.bin file to /lib/firmware/rtlwifi/rtl8712u.bin.

Use extlinux in place of Grub bootloader

The Debian installer will have configured Grub as the boot loader, but extlinux is better for our purposes here. In order to make this system easier to clone onto other SD cards, I tweaked some of the settings so that the SOLARNODE filesystem label I set up during installation is referenced instead. See http://shallowsky.com/linux/extlinux.html for more info.

1. Install extlinux: apt-get install extlinux

2. Remove grub: apt-get remove --purge grub-pc grub-common grub-pc-bin grub2-common

3. mkdir /boot/extlinux

4. extlinux --install /boot/extlinux

5. dd if=/usr/lib/EXTLINUX/mbr.bin of=/dev/sda bs=440 count=1

6. Create a simple /boot/extlinux/extlinux.conf file with the following content:

    default linux
    label linux
      kernel ../vmlinuz-3.16.0-4-586
      append initrd=../initrd.img-3.16.0-4-586 root=/dev/sda1 ro quiet
   

Add udev rule file to remove MAC address associations

The Debian installer will have set up a udev rule that associates the ethernet and WiFi devices with persistent device names, eth0 and wlan0, based on those devices' hardware MAC addresses. In order to make this system easier to clone onto other SD cards, I disabled the automatic generation of this file:

# Disable the automatic, MAC-based naming rules file generation
rm /etc/udev/rules.d/70-persistent-net.rules
ln -s /dev/null /etc/udev/rules.d/70-persistent-net.rules

Next I added a custom udev rules file /etc/udev/rules.d/a10-solarnode.rules with the following content:

# Rename network interfaces NOT using MAC addresses, so this image can be copied to other devices
SUBSYSTEM=="net", DRIVERS=="?*", KERNEL=="eth*", NAME="eth%n"
SUBSYSTEM=="net", DRIVERS=="?*", KERNEL=="wlan*", NAME="wlan%n"

This will map ethernet devices to ethX and WiFi to wlanX where X starts at 0. This thus provides the eth0 and wlan0 network device names.

Use systemd-networkd for network configuration

Create /etc/systemd/network/eth.network with:

[Match]
Name=eth0

[Network]
DHCP=yes

[DHCP]
RouteMetric=10

Then to support WiFi, which will have lower precedence to the wired network, create /etc/systemd/network/wlan.network with:

[Match]
Name=wlan0

[Network]
DHCP=yes

[DHCP]
RouteMetric=20

Create unit /lib/systemd/system/wpa_supplicant@.service with:

[Unit]
Description=WPA supplicant daemon (interface-specific version)
Requires=sys-subsystem-net-devices-%i.device
After=sys-subsystem-net-devices-%i.device

[Service]
Type=simple
ExecStart=/sbin/wpa_supplicant -c/etc/wpa_supplicant/wpa_supplicant-%I.conf -i%I -Dwext

[Install]
Alias=multi-user.target.wants/wpa_supplicant@%i.service

Create WPA configuration /etc/wpa_supplicant/wpa_supplicant-wlan0.conf with:

network={
	ssid="ssid"
	#psk="passphrase"
	psk=2b1d17284c5410ee5eaae7151290e9744af2182b0eb8af20dd4ebb415928f726

	# if the SSID is hidden, add
	#scan_ssid=1
}

Then,

systemctl enable systemd-networkd
systemctl enable wpa_supplicant\@wlan0
systemctl enable systemd-resolved
systemctl start systemd-networkd
systemctl start wpa_supplicant\@wlan0
systemctl start systemd-resolved

Now map resolve.conf to systemd-resolved:

rm /etc/resolv.conf
ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf

Finally, can remove ifupdown:

apt-get purge ifupdown

Note as the WPA settings are just placeholders, after testing the setup works the service was disabled:

systemctl disable wpa_supplicant\@wlan0

Add firewall and redirect port 80

The SolarNode web application runs on port 8080 by default, but we'd like to be able to access it via the standard HTTP port, 80. We can use iptables to both provide a firewall for the node as well as setup NAT to translate port 80 into 8080 for us. This is adapted from Arch Linux. In addition we will configure support for SSH password brute-force mitigation by dynamically blocking IP addresses after failed password login attempts via SSH.

First create a configuration file suitable for iptables-restore to read, at /etc/iptables/iptables.rules:

*filter

# Create chain for SSH brute-force password cracking mitigation
-N dropBrute

# Allows all loopback (lo0) traffic and drop all traffic to 127/8 that doesn't use lo0
-A INPUT -i lo -j ACCEPT
-A INPUT ! -i lo -d 127.0.0.0/8 -j REJECT

# Allow all established inbound connections
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow outbound traffic
-A OUTPUT -j ACCEPT

# Allows HTTP
-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 8080 -j ACCEPT

# Allow SSH
-A INPUT -p tcp -m state --state NEW --dport 22 -j dropBrute
-A INPUT -p tcp -m state --state NEW --dport 22 -m limit --limit 6/min --limit-burst 6 -j ACCEPT

# Allow ping
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT

# log iptables denied calls (access via 'dmesg' command)
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7

-A INPUT -j REJECT
-A FORWARD -j REJECT

COMMIT

*nat

# Redirect port 80 to 8080 for SolarNode
-A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080

COMMIT

Create /lib/systemd/system/iptables.service with:

[Unit]
Description=Packet Filtering Framework
DefaultDependencies=no
After=systemd-sysctl.service
Before=sysinit.target

[Service]
Type=oneshot
ExecStart=/sbin/iptables-restore /etc/iptables/iptables.rules
ExecReload=/sbin/iptables-restore /etc/iptables/iptables.rules
ExecStop=/lib/systemd/scripts/iptables-flush
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Create /lib/systemd/scripts/iptables-flush with:

#!/bin/bash
#
# Usage: iptables-flush [6]
#

iptables=ip$1tables
if ! type -p "$iptables"; then
  echo "error: invalid argument"
  exit 1
fi

while read -r table; do
  tables+=("/var/lib/$iptables/empty-$table.rules")
done <"/proc/net/ip$1_tables_names"

if (( ${#tables[*]} )); then
  cat "${tables[@]}" | "$iptables-restore"
fi

Make it executable:

chmod 755 /lib/systemd/scripts/iptables-flush

Create /var/lib/iptables/empty-filter.rules with:

# Empty iptables filter table rule file
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT

Create /var/lib/iptables/empty-nat.rules with:

# Empty iptables nat table rules file
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT

Now enable the iptables service:

systemctl enable iptables
systemctl start iptables

You can check the filter and nat tables with:

iptables -L
iptables -L -t nat

Configure SSH password brute-force attack mitigation

Create a /usr/share/solarnode/drop-brute.sh script using the script on GitHub.

Make this executable:

chmod 755 /usr/share/solarnode/drop-brute.sh

Then add a file /etc/cron.d/drop-brute with the following content, to run the drop-brute.sh script every 2 minutes:

# SSH password brute-force mitigation
*/2 * * * * root /usr/share/solarnode/drop-brute.sh >/dev/null 2>&1

Configure systemd-tmpfiles

Create a memory-only location for application files under a /run/solar directory, by adding a /usr/lib/tmpfiles.d/solarnode.conf file with the following content:

# SolarNode tmpfile configuration

# Type Path Mode UID GID Age Argument

# Create primary work area
d /run/solar 0755 solar solar -

# For history compatibility, create symlink
L /run/shm/solar - - - - /run/solar

# Do not clean up any files in these areas
x /run/solar/*
x /run/shm/solar/*

Configure journald, systemd

Edit /etc/systemd/journald.conf to limit the size of the systemd journal:

[Journal]
RuntimeMaxUse=10M

Edit /etc/systemd/system.conf to limit the size of the systemd journal:

[Manager]
DumpCore=no

Configure kernel to reboot on error

If the SolarNode process (or any process) becomes unresponsive, we can make the kernel panic and reboot by adding a file /etc/sysctl.d/solarnode.conf with the following content:

# Reboot 5 seconds after panic
kernel.panic = 5

# Panic if a hung task was found
kernel.hung_task_panic = 1

# Setup timeout for hung task to 300 seconds
kernel.hung_task_timeout_secs = 300

Support expanding root filesystem on boot

Often it is convenient to expand the filesystem SolarNode is running on to take advantage of all the available space on the root device. For example if a 1GB OS image is copied to a 2GB SD card, then 1GB of the SD card will be unused.

The approach taken here is to provide a solarnode-expandfs service unit that expands the filesystem with the SOLARNODE label on it when the OS boots up, but only if a /var/local/solarnode-expandfs file exists. That file is not included by default, however, so it must be explicitly enabled by creating that file and then rebooting or manually running systemctl start solarnode-expandfs.

First, add a /lib/systemd/system/solarnode-expandfs.service unit file with:

[Unit]
Description=SolarNode Filesystem Expand
After=sysinit.target local-fs.target
ConditionPathExists=/var/local/solarnode-expandfs

[Service]
Type=oneshot
ExecStart=/usr/share/solarnode/expandfs.sh -v -o /var/local/solarnode-expandfs.saved
ExecStartPost=/bin/rm -f /var/local/solarnode-expandfs

[Install]
WantedBy=basic.target

This relies on a /usr/share/solarnode/expandfs.sh script, shown below. After creating the script, make it executable with

chmod 700 /usr/share/solarnode/expandfs.sh

An abbreviated example of this script, without error checking, is:

#!/usr/bin/env bash

OLD_GEOM_FILE=/var/local/solarnode-expandfs.saved
SOLARNODE_PART=`lsblk -npo kname,label |grep -i SOLARNODE |cut -d' ' -f 1`

# get the device and partition number with the SOLARNODE label
SOLARNODE_DEV=
SOLARNODE_PART_NUM=
MMC_REGEX='(.*)p([0-9]+)'
if [[ $SOLARNODE_PART =~ $MMC_REGEX ]]; then
	SOLARNODE_DEV=${BASH_REMATCH[1]}
	SOLARNODE_PART_NUM=${BASH_REMATCH[2]}
fi
if [ -z "$SOLARNODE_PART_NUM" ]; then
	echo "Error: SOLARNODE partition number not discovered"
	exit 1
fi

# Expand partition
echo ',+' |sfdisk ${SOLARNODE_DEV} -N${SOLARNODE_PART_NUM} --no-reread \
		-f -uS -q -O "${OLD_GEOM_FILE}" 2>/dev/null

# Inform the kernel of the partition change
partx -u ${SOLARNODE_DEV}

# Resize the filesystem to use the entire partition
resize2fs "${SOLARNODE_PART}"

The full script is available in the solarnetwork-build repository.

You can verify the script ran on boot by checking that:

1. the /var/local/solarnode-expandfs file no longer exists
2. the root filesystem has grown to take over all the available disk space
3. sudo journalctl -u solarnode-expandfs shows output similar to below

Here is example output of the script when the OS boots:

$ sudo journalctl -u solarnode-expandfs
-- Logs begin at Fri 2017-11-03 16:27:06 NZDT, end at Tue 2017-11-07 14:30:52 NZDT. --
Nov 03 16:27:10 solarnode systemd[1]: Starting SolarNode Filesystem Expand...
Nov 03 16:27:10 solarnode expandfs.sh[310]: Discovered SOLARNODE partition /dev/mmcblk0p2...
Nov 03 16:27:10 solarnode expandfs.sh[310]: Expanding /dev/mmcblk0 partition 2
Nov 03 16:27:10 solarnode expandfs.sh[310]: Saving recovery output to /var/local/solarnode-expandfs.saved...
Nov 03 16:27:11 solarnode expandfs.sh[310]: Disk /dev/mmcblk0: 30528 cylinders, 4 heads, 16 sectors/track
Nov 03 16:27:11 solarnode expandfs.sh[310]: Old situation:
Nov 03 16:27:11 solarnode expandfs.sh[310]: Units: sectors of 512 bytes, counting from 0
Nov 03 16:27:11 solarnode expandfs.sh[310]: Device Boot    Start       End   #sectors  Id  System
Nov 03 16:27:11 solarnode expandfs.sh[310]: /dev/mmcblk0p1          8192    122879     114688   c  W95 FAT32 (LBA)
Nov 03 16:27:11 solarnode expandfs.sh[310]: /dev/mmcblk0p2        122880   1949695    1826816  83  Linux
Nov 03 16:27:11 solarnode expandfs.sh[310]: /dev/mmcblk0p3             0         -          0   0  Empty
Nov 03 16:27:11 solarnode expandfs.sh[310]: /dev/mmcblk0p4             0         -          0   0  Empty
Nov 03 16:27:11 solarnode expandfs.sh[310]: New situation:
Nov 03 16:27:11 solarnode expandfs.sh[310]: Units: sectors of 512 bytes, counting from 0
Nov 03 16:27:11 solarnode expandfs.sh[310]: Device Boot    Start       End   #sectors  Id  System
Nov 03 16:27:11 solarnode expandfs.sh[310]: /dev/mmcblk0p1          8192    122879     114688   c  W95 FAT32 (LBA)
Nov 03 16:27:11 solarnode expandfs.sh[310]: /dev/mmcblk0p2        122880   1953791    1830912  83  Linux
Nov 03 16:27:11 solarnode expandfs.sh[310]: /dev/mmcblk0p3             0         -          0   0  Empty
Nov 03 16:27:11 solarnode expandfs.sh[310]: /dev/mmcblk0p4             0         -          0   0  Empty
Nov 03 16:27:11 solarnode expandfs.sh[310]: Successfully wrote the new partition table
Nov 03 16:27:12 solarnode expandfs.sh[310]: Re-reading the partition table ...
Nov 03 16:27:12 solarnode expandfs.sh[310]: Reloading partition table for /dev/mmcblk0...
Nov 03 16:27:12 solarnode expandfs.sh[310]: partx -u /dev/mmcblk0
Nov 03 16:27:12 solarnode expandfs.sh[310]: Expanding filesystem on partition /dev/mmcblk0p2...
Nov 03 16:27:12 solarnode expandfs.sh[310]: resize2fs /dev/mmcblk0p2
Nov 03 16:27:12 solarnode expandfs.sh[310]: resize2fs 1.43.3 (04-Sep-2016)
Nov 03 16:27:12 solarnode expandfs.sh[310]: Filesystem at /dev/mmcblk0p2 is mounted on /; on-line resizing required
Nov 03 16:27:12 solarnode expandfs.sh[310]: old_desc_blocks = 1, new_desc_blocks = 1
Nov 03 16:27:12 solarnode expandfs.sh[310]: The filesystem on /dev/mmcblk0p2 is now 457728 (2k) blocks long.
Nov 03 16:27:12 solarnode systemd[1]: Started SolarNode Filesystem Expand.

Delete SSH key and set up to get created at boot

If you're creating this as an image for many nodes to copy from, then you should delete the SSH key Debian generated for you, and add a small systemd unit to recreate the keys the next time the OS boots. First, delete the keys:

rm -f /etc/ssh/ssh_host_*

Then add a /lib/systemd/system/sshdgenkeys.service unit file with:

[Unit]
Description=SSH Key Generation
ConditionPathExists=|!/etc/ssh/ssh_host_key
ConditionPathExists=|!/etc/ssh/ssh_host_key.pub
ConditionPathExists=|!/etc/ssh/ssh_host_rsa_key
ConditionPathExists=|!/etc/ssh/ssh_host_rsa_key.pub
ConditionPathExists=|!/etc/ssh/ssh_host_dsa_key
ConditionPathExists=|!/etc/ssh/ssh_host_dsa_key.pub
ConditionPathExists=|!/etc/ssh/ssh_host_ecdsa_key
ConditionPathExists=|!/etc/ssh/ssh_host_ecdsa_key.pub
ConditionPathExists=|!/etc/ssh/ssh_host_ed25519_key
ConditionPathExists=|!/etc/ssh/ssh_host_ed25519_key.pub

[Service]
ExecStart=/usr/bin/ssh-keygen -A
Type=oneshot
RemainAfterExit=yes

Create /etc/systemd/system/ssh.service.d/local.conf to add dependency on sshdgenkeys.service:

[Unit]
Wants=sshdgenkeys.service
After=network.target auditd.service sshdgenkeys.service

Note the Wants line was added, while the After line was edited to add sshdgenkeys.service. Finally, reload the configuration:

systemctl daemon-reload

Clean Up

As a final clean up to free space, run apt-get clean. The sfill program, available in the secure-delete package, can help make compressed images created with dd smaller by forcing all unused blocks to 0. Run sfill -f -z -I -ll / to zero out all unused space. This can take some time to run.

Install base SolarNode platform

Now we'll deploy a basic SolarNode platform, and configure it to startup when the node boots. See Deploying the SolarNode application for more information.