Debian Server setup

Quick setup notes for Debian Servers

Make sure that /data/project/<environment> exists. On WMF servers, this is provided by setting mount_nfs: true in the instance Hiera configuration.

Email

DKIM

Create email signing private (dkim) / public (dkim.pub.pkcs8) keypair

ssh-keygen -t rsa -b 1024 -m PEM -P "" -f dkim
ssh-keygen -f dkim.pub -m PKCS8 -e > dkim.pub.pkcs8

Use the public key in a DKIM DNS TXT record: contents of dikim.pub.pkcs8:

-----BEGIN PUBLIC KEY-----
****************************************************************
****************************************************************
****************************************************************
************************
-----END PUBLIC KEY-----

maps to

<environment>._domainkey.twl.wmflabs.org. 60 IN TXT	"v=DKIM1;t=s;p=************************************************************************************************************************************************************************************************************************;adkim=s;"

note that here needs to match the DKIM_SELECTOR environment variable defined in conf/.twlight.env, which should match the environment name. You'll omit the BEGIN/END markers as well as linebreaks in the DNS record value.

SPF

an SPF dns record is domain-wide and not-server specific. It should already be created, but is documented here for posterity:

twl.wmflabs.org.	60	IN	TXT	"v=spf1 a:mx-out03.wmcloud.org a:mx-out04.wmcloud.org ~all"

Docker Swarm

If you are feeling trustworthy, go ahead and pipe our script directly into a root shell on your server. What's the worst that could happen?

sudo bash -c "$(curl -fsSL https://raw.githubusercontent.com/WikipediaLibrary/TWLight/production/bin/debian_swarm_deploy.sh)"

You should at least check the source at bin/debian_swarm_deploy.sh

Alternatively, you could follow these instructions; the staging environment is used in the following examples.

• Configure the upstream Docker Repository and install the latest version of Docker CE
• Install Docker Compose
• Add yourself (or your shared system user) to the docker group sudo usermod -a -G docker ${USER} then logout and log back in.
• Clone this repository git clone https://github.com/WikipediaLibrary/TWLight.git (ideally into a shared directory like /srv) and checkout appropriate branch
• docker swarm init
• Create secrets, but with real values:

printf "This is a secret" | docker secret create DJANGO_DB_NAME -
printf "This is a secret" | docker secret create DJANGO_DB_USER -
printf "This is a secret" | docker secret create DJANGO_DB_PASSWORD -
printf "'''-----BEGIN RSA PRIVATE KEY-----
****************************************************************
****************************************************************
****************************************************************
****************************************************************
****************************************************************
****************************************************************
****************************************************************
****************************************************************
****************************************************************
****************************************************************
****************************************************************
****************************************************************
************************************************
-----END RSA PRIVATE KEY-----'''" | docker secret create DKIM_PRIVATE_KEY -
printf "This is a secret" | docker secret create MYSQL_ROOT_PASSWORD -
printf "This is a secret" | docker secret create SECRET_KEY -
printf "This is a secret" | docker secret create TWLIGHT_OAUTH_CONSUMER_KEY -
printf "This is a secret" | docker secret create TWLIGHT_OAUTH_CONSUMER_SECRET -
printf "This is a secret" | docker secret create TWLIGHT_EZPROXY_SECRET -
printf "This is a secret" | docker secret create MATOMO_SITEID -
printf "This is a secret" | docker secret create MATOMO_AUTH_TOKEN -

• delete dkim key files: rm dkim dkim.pub dkim.pub.pkcs8
• deploy for your environment docker stack deploy -c docker-compose.yml -c docker-compose.staging.yml staging
  • Repeat this step if you add secrets after deployment or update your docker-compose files.
• Restore state from a backup docker exec -t $(docker ps -q -f name=staging_twlight) /app/bin/virtualenv_restore.sh /app/backup/dd.hh.tar.gz
• Get an interactive shell docker exec -it $(docker ps -q -f name=staging_twlight) bash
• Enable cron tasks for Django tasks and for applying updated Docker images:

> crontab -e
# Run django_cron tasks.
*/5 * * * *  docker exec -t $(docker ps -q -f name=staging_twlight) /app/bin/twlight_docker_entrypoint.sh python manage.py runcrons
# Update the running TWLight service if there is a new image.
*/5 * * * *  /srv/TWLight/bin/./twlight_docker_deploy.sh staging branch_staging
# Reclaim disk space previously used by docker.
*/30 * * * * docker system prune -a -f; docker volume rm $(docker volume ls -qf dangling=true)

• Setup weekly log rotation truncate docker logs $ sudo vim /etc/logrotate.d/twl_docker_container_logs

/var/lib/docker/containers/*/*-json.log {
    rotate 0
    weekly
    missingok
    copytruncate
    postrotate
        rm "${1}.1"
    endscript
}