Openstack Gitlab Runner Setup

We're now setting up a set of Gitlab runners for vg on the GI's Openstack local cloud.

Runner setup

• Make sure you have an SSH key set up in Openstack
• Get Openstack YAML file from http://gicloud.prism/dashboard/project/api_access/
• Put at ~/config/openstack/clouds.yaml
• Edit in your Openstack password under auth as a password key, since it can't remember a cookie or anything.
• Install the Openstack CLI:

pip install --user python-openstackclient

• Deploy a runner VM and connect to it:

SSH_KEY_NAME=anovak-swords
SERVER_NAME=anovak-gitlab-runner-vgteam-1
FLAVOR=m1.medium

openstack --os-cloud openstack server create --image ubuntu-22.04-LTS-x86_64 --flavor ${FLAVOR} --key-name ${SSH_KEY_NAME} --wait ${SERVER_NAME}
# There is no way to find a free floating IP that already exists without fighting over it.
# Assignment steals the IP if it was already assigned elsewhere.
# See <https://stackoverflow.com/q/36497218>
IP_ID=$(openstack --os-cloud openstack floating ip create ext-net --format value --column id | head -n1)
openstack --os-cloud openstack server add floating ip ${SERVER_NAME} ${IP_ID}
sleep 60
INSTANCE_IP="$(openstack --os-cloud openstack floating ip show ${IP_ID} --column floating_ip_address --format value)"
ssh-keygen -R ${INSTANCE_IP}
ssh ubuntu@${INSTANCE_IP}

• On the VM, become root:

sudo su -

• Go to https://ucsc-ci.com/admin/runners/new and make a new runner that is paused and runs untagged jobs. Name it so you can match it up with the VM you made.
• Set the runner token in your environment on the server:

RUNNER_TOKEN=!!!PASTE!TOKEN!HERE!!!

• Set up the machinery to keep Docker images and scratch space on the instance's ephemeral storage that mounts at /mnt (this code is adapted from Toil and might be Apache licensed):

systemctl stop docker.socket || true
systemctl stop docker.service || true
systemctl stop ephemeral-setup.service || true
rm -Rf /var/lib/docker

cat >/etc/systemd/system/ephemeral-setup.service <<'EOF'
[Unit]
Description=bind mounts ephemeral directories
Before=docker.service
Requires=mnt.mount
After=mnt.mount

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=mkdir -p /mnt/ephemeral/var/lib/docker
ExecStart=mkdir -p /var/lib/docker
ExecStart=mount --bind /mnt/ephemeral/var/lib/docker /var/lib/docker
ExecStop=umount /var/lib/docker

[Install]
RequiredBy=docker.service
EOF

systemctl daemon-reload

systemctl enable ephemeral-setup.service
systemctl start docker.socket || true
systemctl start docker.service || true

Note that some of these commands may complain; Docker probably isn't already installed and running.

• Set up the Gitlab runner on the server:

TASK_MEMORY=25G
TASKS_PER_NODE=1
CPUS_PER_TASK=8
bash -c "export DEBIAN_FRONTEND=noninteractive; curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | bash"
DEBIAN_FRONTEND=noninteractive apt update && sudo DEBIAN_FRONTEND=noninteractive apt upgrade -y
DEBIAN_FRONTEND=noninteractive apt install -y docker.io gitlab-runner
gitlab-runner register --non-interactive --url https://ucsc-ci.com --token "${RUNNER_TOKEN}" --limit "${TASKS_PER_NODE}" --executor docker --docker-privileged --docker-memory "${TASK_MEMORY}" --docker-cpus "${CPUS_PER_TASK}" --docker-image docker:dind
sed -i "s/concurrent = 1/concurrent = ${TASKS_PER_NODE}/g" /etc/gitlab-runner/config.toml
echo "  output_limit = 40960" >>/etc/gitlab-runner/config.toml
gitlab-runner restart

• Unpause the runner in the web UI

Registry proxy setup

On a runner server with free resources, start a persistent Docker registry proxy:

docker run -d -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io -p 5000:5000 -p 80:5000 --restart always --name registry registry:2

You can view the logs with:

docker logs registry

The --restart always registers the container in the Docker daemon to be started when the daemon starts.

If this is very broken, you can run the registry under your terminal instead:

docker run --rm -ti -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io -p 5000:5000 -p 80:5000 --name registry registry:2

Make sure to set DOCKER_HUB_MIRROR for the vg CI jobs to http://server-public-ip so the proxy will be used.