Intelligent City Platform production server build and system installation

These steps describe the process of getting from a bare brand new server to a running Rita platform. The specific machine comments, e.g. regarding iTrac relate to a Dell PowerEdge server.

Install Ubuntu on the server

These instructions assume you've downloaded the appropriate Ubuntu Server iso image, e.g. from

https://www.ubuntu.com/download/server

Note that the default download is the 'live' CD, using Subiquity which is incompatible with Dell iDrac.

If this is a problem for you then install the download version available on the 'alternatives' page:

http://cdimage.ubuntu.com/releases/18.04.2/release/

Dell F11 - enter boot manager

One-shot UEFI Boot Disk connected to front USB 1

Installation options:

Install Ubuntu

• Download while installing
• Install 3rd party
• Erase Disk and Install
• Use LVM

Continue in UEFI mode Write changed to disk London timezone

• English (UK) Extended winkeys
• Enter user details, e.g. for Computer Name use 'tfc-appN'

Networking

Usually the IP parameters will be set during the boot process.

Note the IP V4 parameters are as follows, with XXX as issued.

address 128.232.98.XXX (or 128.232.98.XXX/24 if requested)
gateway 128.232.98.1
netmask 255.255.255.0
dns-nameservers 128.232.1.1 128.232.1.2

To add the DNS servers to the config (Ubuntu 22.04):

Create a file:

sudo mkdir /etc/systemd/resolved.conf.d/
sudo nano /etc/systemd/resolved.conf.d/dns_servers.conf

Add dns servers in this file:

[Resolve]
DNS=182.232.1.1 128.232.1.2 8.8.8.8

Then restart systemd-resolved

sudo systemctl restart systemd-resolved

Apply immediate updates

sudo apt-get update
sudo apt-get upgrade

Install openSSH

sudo apt-get install openssh-server

Add or update the following two directives in /etc/ssh/sshd_config:

PermitRootLogin prohibit-password
UseDNS yes

Create local ssh key

ssh-keygen -t rsa -b 4096 -C "username@tfc-appN"

Install emacs

sudo apt-get install emacs

Add emacs/ssh config files

If required, now is an opportunity to sftp .emacs.el and .ssh/authorized_keys from your other servers.

Configure disks as LVM volumes

Run tfc_prod/tools/mkdisks.sh (as below):

#!/bin/bash

sudo pvcreate /dev/sdb
sudo pvcreate /dev/sdc
sudo pvcreate /dev/sdd


sudo vgcreate sdb-vg /dev/sdb
sudo vgcreate sdc-vg /dev/sdc
sudo vgcreate sdd-vg /dev/sdd

sudo lvcreate -L 3.5T -n sdb1 sdb-vg
sudo lvcreate -L 3.5T -n sdc1 sdc-vg
sudo lvcreate -L 3.5T -n sdd1 sdd-vg

sudo mkfs.ext4 /dev/sdb-vg/sdb1
sudo mkfs.ext4 /dev/sdc-vg/sdc1
sudo mkfs.ext4 /dev/sdd-vg/sdd1

sudo mkdir /mnt/sdb1
sudo mkdir /mnt/sdc1
sudo mkdir /mnt/sdd1

These steps are listed in more detail (for a single disk /dev/sdb) below. If you have already run the script above then skip to "Mount drives" section.

E.g. with new 4TB drive as /dev/sdb, check with:

sudo fdisk -l

Create one LV per disk with: Create PV:

sudo pvcreate /dev/sdb

View with:

sudo pvdisplay
sudo pvs

Create VG:

sudo vgcreate sdb-vg /dev/sdb

Create LV:

sudo lvcreate -L 3.5T -n sdb1 sdb-vg

Check with:

sudo fdisk -l /dev/sdb-vg/sdb1

New LV is now accessible as /dev/sdb-vg/sdb1 Add ext4 filesystem to LV:

sudo mkfs.ext4 /dev/sdb-vg/sdb1

Set up permanent mount: Create mount point:

sudo mkdir /mnt/sdb1

Mount Drives

Get LV /dev/mapper location:

ll /dev/mapper

Create fstab entry

sudo emacs /etc/fstab

adding (e.g.):

/dev/mapper/sdb--vg-sdb1 /mnt/sdb1               ext4    defaults 0       2
/dev/mapper/sdc--vg-sdc1 /mnt/sdc1               ext4    defaults 0       2
/dev/mapper/sdd--vg-sdd1 /mnt/sdd1               ext4    defaults 0       2

Re-initialize mounts and check filesystem mounted ok:

sudo mount -a
df -h
ll /mnt/sdb1

Install Java 8 SDK (JRE?)

sudo apt-get install openjdk-8-jdk

test with

java -version

You should see the SDK version 1.8.xxx.

Note that if multiple java versions are to be installed (e.g. on a development server) then the default can be set with

sudo update-alternatives --config java

and checked with

update-java-alternatives --list

Create (non-sudo) tfc_prod user

with tfc-appN as the correct hostname:

sudo adduser tfc_prod
<reply to prompts>

su tfc_prod
<enter password>
cd ~
ssh-keygen -t rsa -b 4096 -C "tfc_prod@tfc-appN"
<reply to prompts>

Install git

sudo apt install git

Get latest tfc_prod build

su tfc_prod
cd ~

git clone https://github.com/SmartCambridge/tfc_prod.git

From another server, sftp the current tfc_prod/secrets/ directory and tfc_prod/secrets.sh file.

Update server /etc/ssh/ssh_known_hosts

As a sudoer, run

~tfc_prod/tfc_prod/scripts/update_known_hosts.sh >ssh_known_hosts
sudo mv ssh_known_hosts /etc/ssh/

If you inspect the ssh_known_hosts file, you should see an entry for each tfc-appX server followed by an identical entry with the smartcambridge.org hostname.

Install Nginx

See nginx/README.md

Install Monit (?)

See (monit/INSTALLATION.md)[monit/INSTALLATION.md]

Add the tfc_server JAR file to the tfc_prod directory

Ideally, as a developer user (not tfc_prod), install the tfc_server source https://github.com/SmartCambridge/tfc_server

Run mvn clean package in the tfc_server directory to create the fat jar.

Copy the fat jar file (such as ~/tfc_server/target/tfc_server-*-fat.jar) to (say) ~/tfc_prod/tfc_2017-09-27.jar.

Alternatively you can simple collect the tfc_prod/tfc_YYYY-MM-DD.jar from another server

In the tfc_prod directory, create a symlink to the jar file (use the actual name, not tfc_YYYY_MM_DD) with:

rm tfc.jar
ln -s tfc_YYYY_MM_DD.jar tfc.jar

Create data directory links

Run tfc_prod/tools/mkdirs.sh as below:

#!/bin/bash

# create log directory for tfc_prod user

sudo mkdir /var/log/tfc_prod
sudo chown tfc_prod:tfc_prod /var/log/tfc_prod
sudo chmod a+w /var/log/tfc_prod

echo Log directory /var/log/tfc_prod is setup

# create basic tfc directories in /mnt/sdb1 and /media

sudo mkdir /media/tfc
sudo chown tfc_prod:tfc_prod /media/tfc

sudo mkdir /mnt/sdb1/tfc
sudo chown tfc_prod:tfc_prod /mnt/sdb1/tfc

echo Data directories /media/tfc and /mnt/sdb1/tfc setup

Create remainder of directories as tfc_prod user

sudo su tfc_prod

Then run tfc_prod/tools/mkdirs_tfc_prod.sh as below:

#!/bin/bash

# run as user tfc_prod

# create tfc/vix directory and sub-dirs on sdb1

mkdir /mnt/sdb1/tfc/vix

mkdir /mnt/sdb1/tfc/vix/data_bin
mkdir /mnt/sdb1/tfc/vix/data_bin_json
mkdir /mnt/sdb1/tfc/vix/data_zone
mkdir /mnt/sdb1/tfc/vix/data_cache
mkdir /mnt/sdb1/tfc/vix/data_monitor
mkdir /mnt/sdb1/tfc/vix/data_monitor_json

# create links to all vix directories from /media/tfc

mkdir /media/tfc/vix

ln -sfn /mnt/sdb1/tfc/vix/data_bin /media/tfc/vix/data_bin
ln -sfn /mnt/sdb1/tfc/vix/data_bin_json /media/tfc/vix/data_bin_json
ln -sfn /mnt/sdb1/tfc/vix/data_zone /media/tfc/vix/data_zone
ln -sfn /mnt/sdb1/tfc/vix/data_cache /media/tfc/vix/data_cache
ln -sfn /mnt/sdb1/tfc/vix/data_monitor /media/tfc/vix/data_monitor
ln -sfn /mnt/sdb1/tfc/vix/data_monitor_json /media/tfc/vix/data_monitor_json

# set up 'tfc/sys' directory

mkdir /mnt/sdb1/tfc/sys
ln -sfn /mnt/sdb1/tfc/sys /media/tfc/sys

# set up 'tfc/cam_park_local' directory

mkdir /mnt/sdb1/tfc/cam_park_local
ln -sfn /mnt/sdb1/tfc/cam_park_local /media/tfc/cam_park_local

# set up 'tfc/cam_park_rss' directory

mkdir /mnt/sdb1/tfc/cam_park_rss
ln -sfn /mnt/sdb1/tfc/cam_park_rss /media/tfc/cam_park_rss

# set up 'tfc/cam_aq' directory

mkdir /mnt/sdb1/tfc/cam_aq
ln -sfn /mnt/sdb1/tfc/cam_aq /media/tfc/cam_aq

# Cambridge Sensor Network - LoraWAN TTN
# set up 'tfc/csn_ttn' directory

mkdir /mnt/sdb1/tfc/csn_ttn
ln -sfn /mnt/sdb1/tfc/csn_ttn /media/tfc/csn_ttn

# copy data into tfc/sys

cp -r /home/tfc_prod/tfc_prod/config/sys/* /media/tfc/sys

# Create tfc_web log directories

mkdir /var/log/tfc_prod/gunicorn
mkdir /var/log/tfc_prod/pocket_log

Create tfc_prod/secrets directory

As tfc_prod user:

mkdir ~/tfc_prod/secrets

Use sftp to populate tfc_prod/secrets contents from another server.

Test run Status Console

java -cp tfc.jar io.vertx.core.Launcher run "service:uk.ac.cam.tfc_server.console.A" -cluster -cluster-port 10081 >/dev/null 2>>/var/log/tfc_prod/tfc_console.A.err &

Test by browsing to

https://servername/backdoor/

and

https://servername/backdoor/system_status.html

(Note for localhost you may use the remote server name if necessary).

Install tfc_web

Download tfc_web

git clone https://github.com/ijl20/tfc_web.git

Setup log rotation

sudo cp logrotate/tfc_prod /etc/logrotate.d/tfc_prod

See tfc_web/README.md

Configure email (for Monit alerts)

Install/configure ssmtp:

sudo apt install ssmtp
sudo cp ssmtp/ssmtp.conf /etc/ssmtp
sudo cp ssmtp/revaliases /etc/ssmtp

Test by sending an email:

ssmtp foo@cam.ac.uk
To: foo@cam.ac.uk
From: bah@cam.ac.uk
Subject: test email from ssmtp

hello world?

Note blank lines above, and finish email with CTRL-D.

This ssmtp configuration does a reasonable job of getting email out of these systems. The envelope FROM address of mail sent by root and tfc_prod is re-written to admin@smartcambridge.org (and more local addresses can be added to this list in revaliases). The FROM address of all other mail has @cam.ac.uk appended. The envelope TO address of all mail for local users with UID < 1000 is rewritten to admin@smartcambridge.org, and for all other users has @cam.ac.uk appended.

This works for almost everything except mail to the 'tfc_prod' local user which fails because tfc_prod@cam.ac.uk doesn't exist. ssmtp explicitly doesn't support aliasing destination addresses for UIDs >= 1000. Th only way around this is to explicitly send such mail to an address that does work, e.e. by including

MAILTO=admin@smartcambridge.org

at the start of tfc_prod's crontab file.

See tfc_prod/monit/README.md