Skip to content
Permalink
Browse files
User Packer to build a pre-built AMI with everything we need (#15)
* initial packer and tf

* packer added files a scripts from Ashs repo

* add new folder structure and terraform

* updateing packer files

* added dependencies file permission and apt source repos

* bootstrap and user data

* prepare packer provisioners and set up all files to be executed

* update tinder

* terraform to create packer roles, starting to fill in packer variables

* packer roles added aws backends, terraform reformed and added iam roles as well as autoscaling cloudwatch alarm and policy

* fixed iam role and removed policy attatchments

* first run of packer_roles, terraform add gitignore for terraform

* update packer code from results of validate

* update runner max size of asg

* packer updated to run and terraform roles for packer updated

* Apply suggestions from code review

* Update for pre-commit checks

Add licenses, and remove trailing whitespace

* archieve lambda before upload

* remove terraform for ci infra

* Make the packer build produce a working image.

Summary of changes:

- Files need to be copied to a "staging" folder and then moved in place
- Use the built-in upload ability of the shell provisioner
- Have shell provisioner run scripts with sudo, rather than using sudo
  10s of times in the scripts
- Don't set up tmpfs mounts in the AMI -- these have to happen at
  instance boot time, not AMI creation
- Preseed the install options for iptables-persistent so that it
  installs without asking questions or replacing the rules we already
  placed.
- Install the runner-supervisor script from local file, not S3.

Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
  • Loading branch information
Mike Hewitt and ashb committed Apr 22, 2021
1 parent 1328951 commit 4ff4feb24e5efbb1938fe4f0a9265fe73c6c2fb2
Showing 23 changed files with 981 additions and 0 deletions.
@@ -1,2 +1,37 @@
.cache
__pycache__/

# Created by https://www.toptal.com/developers/gitignore/api/terraform
# Edit at https://www.toptal.com/developers/gitignore?templates=terraform

### Terraform ###
# Local .terraform directories
**/.terraform/*

# .tfstate files
*.tfstate
*.tfstate.*

# Crash log files
crash.log

# Ignore any .tfvars files that are generated automatically for each Terraform run. Most
# .tfvars files are managed as part of configuration and so should be included in
# version control.
#
# example.tfvars

# Ignore override files as they are usually used to override resources locally and so
# are not checked in
override.tf
override.tf.json
*_override.tf
*_override.tf.json

# Include override files you do wish to add to version control using negated pattern
# !example_override.tf

# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
# example: *tfplan*

# End of https://www.toptal.com/developers/gitignore/api/terraform
@@ -0,0 +1,22 @@
#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

if pgrep -c Runner.Worker >/dev/null; then
# Only report metric when we're doing something -- no point paying to submit zeros
aws cloudwatch put-metric-data --metric-name jobs-running --value "$(pgrep -c Runner.Worker)" --namespace github.actions
fi
@@ -0,0 +1,30 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

[Unit]
Description=Fetch credentials and supervise GitHub Actions Runner
After=network.target
Before=actions.runner.service

[Service]
Type=notify
ExecStart=/opt/runner-supervisor/bin/python /opt/runner-supervisor/bin/runner-supervisor
# We need to run as root to have the ability to open the netlink connector socket
User=root
WorkingDirectory=/home/runner/actions-runner
Restart=always
EnvironmentFile=/etc/environment
@@ -0,0 +1,38 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

[Unit]
Description=GitHub Actions Runner
After=network.target actions.runner-supervisor.service
Requires=actions.runner-supervisor.service
BindsTo=actions.runner-supervisor.service

[Service]
ExecStartPre=!/usr/local/sbin/runner-cleanup-workdir.sh
ExecStart=/home/runner/actions-runner/run.sh --once --startuptype service
ExecStop=/usr/local/bin/stop-runner-if-no-job.sh $MAINPID
EnvironmentFile=/etc/environment
Environment=GITHUB_ACTIONS_RUNNER_CHANNEL_TIMEOUT=300
User=runner
WorkingDirectory=/home/runner/actions-runner
KillMode=mixed
KillSignal=SIGTERM
TimeoutStopSec=5min
Restart=on-success

[Install]
WantedBy=multi-user.target
@@ -0,0 +1,18 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

*/1 * * * * nobody /usr/local/sbin/actions-runner-ec2-reporting
@@ -0,0 +1,25 @@
#!/usr/bin/env bash

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

set -exu -o pipefail

# https://github.com/actions/virtual-environments/blob/525f79f479cca77aef4e0a680548b65534c64a18/images/linux/scripts/installers/docker-compose.sh
URL=$(curl -s https://api.github.com/repos/docker/compose/releases/latest | jq -r '.assets[].browser_download_url | select(endswith("docker-compose-Linux-x86_64"))')
curl --fail -L "$URL" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
@@ -0,0 +1,41 @@
#!/usr/bin/env bash

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

export DEBIAN_FRONTEND=noninteractive

debconf-set-selections <<EOF
iptables-persistent iptables-persistent/autosave_done boolean true
iptables-persistent iptables-persistent/autosave_v4 boolean false
iptables-persistent iptables-persistent/autosave_v6 boolean false
EOF

apt-get update
apt-get install -y --no-install-recommends \
awscli \
build-essential \
docker.io \
git \
iptables-persistent \
jq \
parallel \
python3-dev \
python3-venv \
python3-wheel \
yarn \
vector
@@ -0,0 +1,27 @@
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

set -eu -o pipefail

mkdir /etc/iptables/

install --owner root --mode=0644 --target-directory "/etc/systemd/system/" "/tmp/etc-systemd-system/"*
install --owner root --mode=0755 --target-directory "/usr/local/sbin" "/tmp/usr-local-sbin/"*
install --owner root --mode=0644 --target-directory "/etc/iptables" "/tmp/etc-iptables/"*
install --owner root --mode=0644 --target-directory "/etc/cron.d" "/tmp/etc-cron.d/"*
install --owner root --mode=0644 --target-directory "/etc/sudoers.d" "/tmp/etc-sudoers.d/"*
@@ -0,0 +1,20 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

sudo mount -t tmpfs -o size=10% tmpfs /tmp
sudo mount -t tmpfs -o size=66% tmpfs /var/lib/docker
sudo mount -t tmpfs -o tmpfs /home/runner/actions-runner/_work
@@ -0,0 +1,28 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

# Generated by iptables-save v1.8.4 on Thu Jan 14 13:59:27 2021
*filter
:INPUT ACCEPT [833:75929]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [794:143141]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
# Dis-allow any docker container to access the metadata service
-A DOCKER-USER -d 169.254.169.254/32 -j REJECT --reject-with icmp-port-unreachable
-A DOCKER-USER -j RETURN
COMMIT
@@ -0,0 +1,18 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

runner ALL=(ALL) NOPASSWD:/usr/sbin/swapoff -a, /usr/bin/rm -f /swapfile, /usr/bin/apt clean
@@ -0,0 +1,35 @@
#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

echo "Left-over containers:"
docker ps -a
docker ps -qa | xargs --verbose --no-run-if-empty docker rm -fv

if [[ -d ~runner/actions-runner/_work/airflow/airflow ]]; then
cd ~runner/actions-runner/_work/airflow/airflow

chown --changes -R runner: .
if [[ -e .git ]]; then
sudo -u runner bash -c "
git reset --hard && \
git submodule deinit --all -f && \
git submodule foreach git clean -fxd && \
git clean -fxd \
"
fi
fi
File renamed without changes.
@@ -0,0 +1,42 @@
#!/usr/bin/env bash

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

set -exu -o pipefail

# Validate params
: "${RUNNER_VERSION?}"

# Set an env var (that is visible in runners) that will let us know we are on a self-hosted runner
echo 'AIRFLOW_SELF_HOSTED_RUNNER="[\"self-hosted\"]"' >> /etc/environment

useradd --create-home runner -G docker

install --owner runner --directory ~runner/actions-runner

cd ~runner/actions-runner
curl -L "https://github.com/ashb/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz" | tar -zx

python3 -mvenv /opt/runner-supervisor
/opt/runner-supervisor/bin/pip install -U pip python-dynamodb-lock-whatnick==0.9.3 click==7.1.2 psutil 'tenacity~=6.0'

install --owner root --mode 0755 /tmp/runner-supervisor /opt/runner-supervisor/bin/runner-supervisor

systemctl enable iptables.service
systemctl enable vector.service
systemctl enable actions.runner.service

0 comments on commit 4ff4feb

Please sign in to comment.