Operations Site Runner

Simple self-hosted git-powered web app automation.

This code sets up a server for launching and testing websites using GitOps tools like GitHub Actions and hosting tools like DDEV.

The goal is to quickly launch running environments on any server for automated and manual testing, integrated directly with GitOps APIs.

It uses GitHub Actions and Workflows with self-hosted runners, providing persistent, accessible environments and logs, fully integrated with GitHub Deployments & Checks APIs for optimal developer experience.

For more details see https://operations-project.gitbook.io/operations-experience-project/operations-site-server

What this does

This Ansible playbook prepares a bare linux server for running websites:

1. Creates users for administrators and grants SSH access using GitHub user SSH keys.
2. Installs Docker, Composer, & DDEV.
3. Creates a platform user for storing site code and running tasks.
4. Installs GitHub Self-hosted Runner as a service, running as platform user.
5. Listens for new jobs remotely. No need to grant GitHub access to post webhooks.
6. Runs the GitHub Actions as defined in the project's .github/workflows config.

Then, the project's codebase takes over using GitHub Actions config.

GitHub Actions

With a self-hosted runner, all activity on the servers is run with GitHub's Actions system.

Through the GitHub interface you can track every deployment, test run, and cron job for pull request and live environments.

Deployment Logic

For now, all deployment logic is stored in the GitHub workflow config:

• Where to clone code.
• Command to launch sites.
• Command to run updates.
• Command to sync data.

See examples folder for working config.

With the right GitHub Actions config, the server will:

1. Clone code to /var/platform/example.com/pr#.
2. Write special DDEV config to set unique URLs for each DDEV site.
3. Launch the site. (ddev start)
4. If a test site, sync data and run tests.

For now, you must copy these templates into your workflows. As the project progresses, we will release shared tooling.

Documentation

This is a very young project still in the proof of concept phase. More information will be added when possible.

Architecture

1. Operations Platform (This repository)

   • An Ansible Collection. Set of roles and configuration for bootstrapping a server to run sites.
   • Installed to /usr/share/operations.
   • Ansible Playbook:
     • geerlingguy.security
     • geerlingguy.github-users
     • geerlingguy.composer
     • geerlingguy.docker
   • Installs system users, Docker, Composer, GitHub Runner, & DDEV. (More engines TBD)
   • @TODO: Self-updating: Control User GitHub runner updates this repo and kicks off playbook when new commits are pushed for changing server config.
   • Users:
     • Special users are created to manage the server.
     • Control user (control) is for server updates. It has sudo NOPASSWD permissions to allow self-configuration. If using multiple servers, this user will have SSH access to those servers in order to configure them remotely.
     • Platform user (platform) is used for all app code. All apps are cloned into the home directory, /var/platform. The platform user has SSH access to clone repositories and access remote servers. The platform user runs the commands needed to launch sites, such as ddev.
     • System Administrators can be set by adding their GitHub username to the ansible variable operations_admin_users. Each user will be granted server access via the SSH keys uploaded to GitHub.com, and sudo NOPASSWD permissions to allow manual maintenance of the server.
   • Service Engines
     • Operations platform relies on other tools for launching sites.
     • "Service Engines" will make it simple to plug in any desired tooling to start, stop, destroy, and restore/sync sites by simply wrapping the tool's commands.
     • DDEV is used as the first service engine as a proof of concept.
     • More will be added whenever time allows.

2. Git Runners

   • This role installs and configures private Git Runners.
   • All needed operations are run through the Git host's "Workflows" or "Pipelines" system, including deployments, cron jobs, backups, etc.
   • Runners listen for events like git pushes or issue creation and then take the actions defined by the git-ops configuration files, without webhooks. This allows privately hosted servers to run sites without forcing users to grant access to the internet.
   • All tasks are logged in the web interfaces of the git hosts. No other task runner (such as jenkins) is needed.
   • Runner Documentation
     • GitHub
     • BitBucket @TODO
     • GitLab @TODO

3. Git Runner config

   • Each project must contain workflow/pipeline configuration files specifically for that git host that works with the private runners.
   • Example config files are located in the (Templates)[./templates] folder (Coming soon).

Links

• GitHub issue templates
• GitHub pull request template
• Contributing guide (Decide about your code of conduct)

TBD

Getting Started

This repo contains everything needed to set up an automated hosting platform.

Prerequisites

1. git
2. ansible

Once the repo is cloned, ansible will set up the rest.

Installing

0. Install Ansible & git.

1. Clone.

    git clone git@github.com:operations-project/site-runner.git /usr/share/operations
   

2. Prepare.

   This playbook creates GitHub runners automatically using a GitHub Token.

   This token is only used to create a runner. No other API interaction is needed because it is all handled by the GitHub Runners. Runner tasks get a unique GITHUB_TOKEN for every job, so users can interact with the GitHub API that way.

   NOTE: If your project is not a personal repo, you need to enable "Personal Access Tokens" in your organization.

   To enable Personal Access Tokens for organizations:

   1. Open your organization's page. For example: https://github.com/organizations/operations-platform/
   2. Click Settings.
   3. Click Personal Access Tokens in the left sidebar.
   4. Go through the wizard to configure how tokens work in your organization.

   Once complete, your organization will be available when you create your own "Personal Access Token".

   To create Personal Access Tokens:

   1. Go to the Create Token Page: https://github.com/settings/personal-access-tokens/new
   2. Under Resource owner, select your organization.
   3. Under Repository Permissions, for Administration, select Read/Write.
   4. Click Generate Token. Save the token for the Configure step, below.

3. Configure Ansible Inventory.

   Ansible inventory is very powerful and flexible. There are many ways to manage your inventory.

   This playbook comes with example inventory and variable files that you can use to manually place in /etc/ansible/hosts and /etc/ansible/host_vars/sites.myhost.com.yml

   See example files in ./ansible/hosts.example and ./ansible/host_vars/host.example.yml

   The result should look something like this:

   • /etc/ansible/hosts

     [operations_host_ddev]
     sites.myhost.com ansible_connection=local
     

   • /etc/ansible/host_vars/sites.myhost.com.yml

     # Add a list of github usernames to grant access to. They will get sudo users and SSH access.
     operations_admin_users:
     - YOUR_GITHUB_USERNAME
     
     operations_github_api_token: YOUR_API_TOKEN
     operations_github_runners:
     - repo_name: YOUR_ORG/YOUR_REPO
     

   For details on additional options, see ./ansible/host_vars/host.example.yml.

   You are free to build up your ansible inventory as you see fit. These are the essential options.

4. Install.

   To install, run ansible-playbook as your user (not with sudo). Ansible knows how to run sudo for the steps that require it.

   As long as your active user can sudo, you can run ansible-playbook as your personal user account.

    ansible-playbook /usr/share/operations/playbook.yml
   

   Once the playbook completes successfully, you will have a new runner present in GitHub. To confirm:

   • Visit your repository's Settings page.
   • Click Actions, then Runners.
   • You should see a runner that matches the hostname and labels you set in your ansible inventory

   NOTE: The playbook will create users from your operations_admin_users variables, and will grant them passwordless SUDO access. After running it the first time, you can run ansible-playbook as your personal user.

5. Implement.

   To use it, you need to add a GitHub workflow config file that has runs-on set to a label that your runner has.

   An example to confirm the runner:

   # .github/workflows/demo.yml
   name: Verify Self-Hosted Runner
   
   # Run on all pushes and manual triggering with "workflow_dispatch"
   on:
       workflow_dispatch:
       push:
   
   jobs:
     verify:
       # Match a label or use the hostname label.
       runs-on: sites.myhost.com
   
       steps:
         - name: Environment
           run: |
             hostname -f
             whoami
             env

   More workflow samples coming soon.

Built With

List significant dependencies that developers of this project will interact with.

• Composer - Dependency Management
• Robo - PHP Task Runner
• Symfony - PHP Framework
• Ansible - System Configuration Platform

Contributing

Please read CONTRIBUTING.md for details on the process for submitting pull requests to us.

Versioning

We use SemVer for versioning. For the versions available, see the releases page.

Authors

• Jon Pugh - created project from template.

See also the list of contributors who participated in this project.

License

This project is licensed under the MIT License - see the LICENSE file for details

Acknowledgments

• @g1a for all that you do.
• aegir, lando, docksal, ddev, etc