User Packer to build a pre-built AMI with everything we need (#15)

* initial packer and tf

* packer added files a scripts from Ashs repo

* add new folder structure and terraform

* updateing packer files

* added dependencies file permission and apt source repos

* bootstrap and user data

* prepare packer provisioners and set up all files to be executed

* update tinder

* terraform to create packer roles, starting to fill in packer variables

* packer roles added aws backends, terraform reformed and added iam roles as well as autoscaling cloudwatch alarm and policy

* fixed iam role and removed policy attatchments

* first run of packer_roles, terraform add gitignore for terraform

* update packer code from results of validate

* update runner max size of asg

* packer updated to run and terraform roles for packer updated

* Apply suggestions from code review

* Update for pre-commit checks

Add licenses, and remove trailing whitespace

* archieve lambda before upload

* remove terraform for ci infra

* Make the packer build produce a working image.

Summary of changes:

- Files need to be copied to a "staging" folder and then moved in place
- Use the built-in upload ability of the shell provisioner
- Have shell provisioner run scripts with sudo, rather than using sudo
  10s of times in the scripts
- Don't set up tmpfs mounts in the AMI -- these have to happen at
  instance boot time, not AMI creation
- Preseed the install options for iptables-persistent so that it
  installs without asking questions or replacing the rules we already
  placed.
- Install the runner-supervisor script from local file, not S3.

Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>

.gitignore

@@ -1,2 +1,37 @@
 .cache
 __pycache__/
+
+# Created by https://www.toptal.com/developers/gitignore/api/terraform
+# Edit at https://www.toptal.com/developers/gitignore?templates=terraform
+
+### Terraform ###
+# Local .terraform directories
+**/.terraform/*
+
+# .tfstate files
+*.tfstate
+*.tfstate.*
+
+# Crash log files
+crash.log
+
+# Ignore any .tfvars files that are generated automatically for each Terraform run. Most
+# .tfvars files are managed as part of configuration and so should be included in
+# version control.
+#
+# example.tfvars
+
+# Ignore override files as they are usually used to override resources locally and so
+# are not checked in
+override.tf
+override.tf.json
+*_override.tf
+*_override.tf.json
+
+# Include override files you do wish to add to version control using negated pattern
+# !example_override.tf
+
+# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
+# example: *tfplan*
+
+# End of https://www.toptal.com/developers/gitignore/api/terraform

github-runner-ami/packer/files/actions-runner-ec2-reporting.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+if pgrep -c Runner.Worker >/dev/null; then
+    # Only report metric when we're doing something -- no point paying to submit zeros
+    aws cloudwatch put-metric-data --metric-name jobs-running --value "$(pgrep -c Runner.Worker)" --namespace github.actions
+fi

github-runner-ami/packer/files/actions.runner-supervisor.service

@@ -0,0 +1,30 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+[Unit]
+Description=Fetch credentials and supervise GitHub Actions Runner
+After=network.target
+Before=actions.runner.service
+
+[Service]
+Type=notify
+ExecStart=/opt/runner-supervisor/bin/python /opt/runner-supervisor/bin/runner-supervisor
+# We need to run as root to have the ability to open the netlink connector socket
+User=root
+WorkingDirectory=/home/runner/actions-runner
+Restart=always
+EnvironmentFile=/etc/environment

github-runner-ami/packer/files/actions.runner.service

@@ -0,0 +1,38 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+[Unit]
+Description=GitHub Actions Runner
+After=network.target actions.runner-supervisor.service
+Requires=actions.runner-supervisor.service
+BindsTo=actions.runner-supervisor.service
+
+[Service]
+ExecStartPre=!/usr/local/sbin/runner-cleanup-workdir.sh
+ExecStart=/home/runner/actions-runner/run.sh --once --startuptype service
+ExecStop=/usr/local/bin/stop-runner-if-no-job.sh $MAINPID
+EnvironmentFile=/etc/environment
+Environment=GITHUB_ACTIONS_RUNNER_CHANNEL_TIMEOUT=300
+User=runner
+WorkingDirectory=/home/runner/actions-runner
+KillMode=mixed
+KillSignal=SIGTERM
+TimeoutStopSec=5min
+Restart=on-success
+
+[Install]
+WantedBy=multi-user.target

github-runner-ami/packer/files/cloudwatch-metrics-github-runners

@@ -0,0 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+*/1 * * * * nobody /usr/local/sbin/actions-runner-ec2-reporting

github-runner-ami/packer/files/docker-compose.sh

@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -exu -o pipefail
+
+# https://github.com/actions/virtual-environments/blob/525f79f479cca77aef4e0a680548b65534c64a18/images/linux/scripts/installers/docker-compose.sh
+URL=$(curl -s https://api.github.com/repos/docker/compose/releases/latest | jq -r '.assets[].browser_download_url | select(endswith("docker-compose-Linux-x86_64"))')
+curl --fail -L "$URL" -o /usr/local/bin/docker-compose
+chmod +x /usr/local/bin/docker-compose

github-runner-ami/packer/files/install-dependencies.sh

@@ -0,0 +1,41 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+export DEBIAN_FRONTEND=noninteractive
+
+debconf-set-selections <<EOF
+iptables-persistent iptables-persistent/autosave_done boolean true
+iptables-persistent iptables-persistent/autosave_v4 boolean false
+iptables-persistent iptables-persistent/autosave_v6 boolean false
+EOF
+
+apt-get update
+apt-get install -y --no-install-recommends \
+            awscli \
+            build-essential \
+            docker.io \
+            git \
+            iptables-persistent \
+            jq \
+            parallel \
+            python3-dev \
+            python3-venv \
+            python3-wheel \
+            yarn \
+            vector

github-runner-ami/packer/files/install-files.sh

@@ -0,0 +1,27 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -eu -o pipefail
+
+mkdir /etc/iptables/
+
+install --owner root --mode=0644 --target-directory "/etc/systemd/system/" "/tmp/etc-systemd-system/"*
+install --owner root --mode=0755 --target-directory "/usr/local/sbin" "/tmp/usr-local-sbin/"*
+install --owner root --mode=0644 --target-directory "/etc/iptables" "/tmp/etc-iptables/"*
+install --owner root --mode=0644 --target-directory "/etc/cron.d" "/tmp/etc-cron.d/"*
+install --owner root --mode=0644 --target-directory "/etc/sudoers.d" "/tmp/etc-sudoers.d/"*

github-runner-ami/packer/files/mounts_setup.sh

@@ -0,0 +1,20 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+sudo mount -t tmpfs -o size=10% tmpfs /tmp
+sudo mount -t tmpfs -o size=66% tmpfs /var/lib/docker
+sudo mount -t tmpfs -o tmpfs /home/runner/actions-runner/_work

github-runner-ami/packer/files/rules.v4

@@ -0,0 +1,28 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Generated by iptables-save v1.8.4 on Thu Jan 14 13:59:27 2021
+*filter
+:INPUT ACCEPT [833:75929]
+:FORWARD DROP [0:0]
+:OUTPUT ACCEPT [794:143141]
+:DOCKER-USER - [0:0]
+-A FORWARD -j DOCKER-USER
+# Dis-allow any docker container to access the metadata service
+-A DOCKER-USER -d 169.254.169.254/32 -j REJECT --reject-with icmp-port-unreachable
+-A DOCKER-USER -j RETURN
+COMMIT

github-runner-ami/packer/files/runner

@@ -0,0 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+runner ALL=(ALL) NOPASSWD:/usr/sbin/swapoff -a, /usr/bin/rm -f /swapfile, /usr/bin/apt clean

github-runner-ami/packer/files/runner-cleanup-workdir.sh

@@ -0,0 +1,35 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+echo "Left-over containers:"
+docker ps -a
+docker ps -qa | xargs --verbose --no-run-if-empty docker rm -fv
+
+if [[ -d ~runner/actions-runner/_work/airflow/airflow ]]; then
+    cd ~runner/actions-runner/_work/airflow/airflow
+
+    chown --changes -R runner: .
+    if [[ -e .git ]]; then
+        sudo -u runner bash -c "
+        git reset --hard && \
+        git submodule deinit --all -f && \
+        git submodule foreach git clean -fxd && \
+        git clean -fxd \
+        "
+    fi
+fi

github-runner-ami/packer/files/runner_bootstrap.sh

@@ -0,0 +1,42 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -exu -o pipefail
+
+# Validate params
+: "${RUNNER_VERSION?}"
+
+# Set an env var (that is visible in runners) that will let us know we are on a self-hosted runner
+echo 'AIRFLOW_SELF_HOSTED_RUNNER="[\"self-hosted\"]"' >> /etc/environment
+
+useradd  --create-home runner -G docker
+
+install --owner runner --directory ~runner/actions-runner
+
+cd ~runner/actions-runner
+curl -L "https://github.com/ashb/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz" | tar -zx
+
+python3 -mvenv /opt/runner-supervisor
+/opt/runner-supervisor/bin/pip install -U pip python-dynamodb-lock-whatnick==0.9.3 click==7.1.2 psutil 'tenacity~=6.0'
+
+install --owner root --mode 0755 /tmp/runner-supervisor /opt/runner-supervisor/bin/runner-supervisor
+
+systemctl enable iptables.service
+systemctl enable vector.service
+systemctl enable actions.runner.service