GitHub Actions

GitHub Actions master template & GitHub Actions Reusable Workflows library.

• main.yaml - GitHub Actions master workflow template
• .github/workflows/ - GitHub Actions Reusable Workflows Library

See Documentation for how to call these workflows directly from your own GitHub Actions workflow.

Fork this repo to have full control over all updates via Pull Requests. Create environment branches to stage updates across dev/staging/production.

Forked from HariSekhon/Templates, for which this is now a submodule.

To see GitHub Contexts available, including undocumented fields, see HariSekhon/GitHub-Actions-Contexts.

Examples

In your GitHub repo, import these workflows by adding small yaml files to the .github/workflows/ directory.

Scan for Secrets and Security issues

Alerts appear under the GitHub repo's Security tab -> Code scanning alerts.

Alerts appear in the https://semgrep.dev dashboard

Create .github/workflows/semgrep.yaml for local repo alerts:

on: [push]
jobs:
  semgrep:
    uses: HariSekhon/GitHub-Actions/.github/workflows/semgrep.yaml@master

or .github/workflows/semgrep-cloud.yaml for https://semgrep.dev alerts:

on: [push]
jobs:
  semgrep:
    uses: HariSekhon/GitHub-Actions/.github/workflows/semgrep-cloud.yaml@master
    secrets:
      SEMGREP_APP_TOKEN: ${{ secrets.SEMGREP_APP_TOKEN }}

Analyze your Terraform code security & best practices

Alerts appear under Security -> Code scanning alerts.

Create .github/workflows/tfsec.yaml:

on: [push]
jobs:
  tfsec:
    uses: HariSekhon/GitHub-Actions/.github/workflows/tfsec.yaml@master

Terraform Plan & Apply

Plans - updates Pull Requests with the results of validation, format check and full Change Plan outputs

Apply - applies when merged to default branch, eg. master or main

on: [push, pull_request]
jobs:
  terraform:
    uses: HariSekhon/GitHub-Actions/.github/workflows/terraform.yaml@master
    with:
      dir: path/to/terraform/code
    secrets:
      ...

For more sophisticated examples including approvals, secrets, branch and path selection etc. see my Terraform repo's templates for terraform-plan.yaml and terraform-apply.yaml

Docker Build and push to DockerHub

Create .github/workflows/dockerhub_build.yaml:

on: [push]
jobs:
  docker_build:
    uses: HariSekhon/GitHub-Actions/.github/workflows/dockerhub_build.yaml@master
    with:
      repo: user/repo  # your DockerHub user/repo
      tags: latest v1.1
    secrets:
      DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}
      DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}

Docker Build and push to AWS ECR

Create .github/workflows/docker_build_aws_ecr.yaml:

on: [push]
jobs:
  docker_build:
    uses: HariSekhon/GitHub-Actions/.github/workflows/docker_build_aws_ecr.yaml@master
    with:
      repo: MY_ECR_REPO
    secrets:
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }}

Creates several useful tags, supports multi-stage build caching, see README for details.

Check for Broken URL Links

Create .github/workflows/url_links.yaml:

on: [push]
jobs:
  url_links:
    uses: HariSekhon/GitHub-Actions/.github/workflows/url_links.yaml@master

See README for details on ignoring inaccessible / partially constructed links or those containing variables

Auto-Merge Production hotfixes back to Staging

Merges via a Pull Request for full auditing.

Create .github/workflows/merge_production_to_staging.yaml:

on: [push]
jobs:
  merge:
    if: github.ref_name == 'production'
    uses: HariSekhon/GitHub-Actions/.github/workflows/merge-branch.yaml@master
    with:
      head: production  # from
      base: staging     # to

Mirror Repos to GitLab for DR Backups

Mirrors all/given GitHub repos to GitLab - including all branches and tags, and GitHub repo description

on:
  schedule:
    # mirror to GitLab hourly
    - cron: '0 0 * * *'

jobs:
  gitlab_mirror:
    uses: HariSekhon/GitHub-Actions/.github/workflows/gitlab-mirror.yaml@master
    with:
      #organization: my-org    # optional: mirror your company's repos instead of your personal repos
      #repos: repo1 repo2 ...  # list of repos to mirror, space separated, rather than all repos
    secrets:
      GH_TOKEN: ${{ secrets.GH_TOKEN }}
      GITLAB_TOKEN: ${{ secrets.GITLAB_TOKEN }}

AWS CodeArtifact - Publish a Python Package

on:
  tags:
    - v*

jobs:
  aws_codeartifact_python_publish:
    uses: HariSekhon/GitHub-Actions/.github/workflows/codeartifact_python_publish.yaml@master
    with:
      domain: mycompany     # your AWS CodeArtifact service domain name
      repo: mycompany-core  # your CodeArtifact repo name
      #command: make publish_package  # default. Can be any command using CODEARTIFACT_AUTH_TOKEN and CODEARTIFACT_REPO_URL
    secrets:
      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }}

Production

Option 1 - Hashref

Import the reusable workflows from this repo as shown above, replacing @master with @<hashref> to fix to an immutable version (tags are not immutable). This is GitHub Actions Security Best Practice.

Option 2 - Public Fork (fully automated)

Fork this repo for more control and visibility over all updates.

Enable the fork-sync github actions workflow in your fork to keep the master branch sync'd every few hours.

You can then create tags or environment branches in your forked repo to stage updates across dev/staging/production.

If using environment branches enable the fork-update-pr github actions workflow to automatically raise GitHub Pull Requests from master to your environment branches to audit, authorize & control updates.

Option 3 - Private Copy (manual)

Copy .github/workflows to a private repo. Not recommended as it's the most manual legacy approach.

You will be responsible for committing and reconciling any divergences in your local copies.

Related Repositories

• Jenkins - Advanced Jenkinsfile & Jenkins Groovy Shared Library

• Templates - Code & Config templates for many popular DevOps technologies

• DevOps Bash Tools - 1000+ DevOps Bash Scripts, Advanced .bashrc, .vimrc, .screenrc, .tmux.conf, .gitconfig, CI configs & Utility Code Library - AWS, GCP, Kubernetes, Docker, Kafka, Hadoop, SQL, BigQuery, Hive, Impala, PostgreSQL, MySQL, LDAP, DockerHub, Jenkins, Spotify API & MP3 tools, Git tricks, GitHub API, GitLab API, BitBucket API, Code & build linting, package management for Linux / Mac / Python / Perl / Ruby / NodeJS / Golang, and lots more random goodies

• SQL Scripts - 100+ SQL Scripts - PostgreSQL, MySQL, AWS Athena, Google BigQuery

• Kubernetes configs - Kubernetes YAML configs - Best Practices, Tips & Tricks are baked right into the templates for future deployments

• Terraform - Terraform templates for AWS / GCP / Azure / GitHub management

• DevOps Python Tools - 80+ DevOps CLI tools for AWS, GCP, Hadoop, HBase, Spark, Log Anonymizer, Ambari Blueprints, AWS CloudFormation, Linux, Docker, Spark Data Converters & Validators (Avro / Parquet / JSON / CSV / INI / XML / YAML), Elasticsearch, Solr, Travis CI, Pig, IPython

• DevOps Perl Tools - 25+ DevOps CLI tools for Hadoop, HDFS, Hive, Solr/SolrCloud CLI, Log Anonymizer, Nginx stats & HTTP(S) URL watchers for load balanced web farms, Dockerfiles & SQL ReCaser (MySQL, PostgreSQL, AWS Redshift, Snowflake, Apache Drill, Hive, Impala, Cassandra CQL, Microsoft SQL Server, Oracle, Couchbase N1QL, Dockerfiles, Pig Latin, Neo4j, InfluxDB), Ambari FreeIPA Kerberos, Datameer, Linux...

• The Advanced Nagios Plugins Collection - 450+ programs for Nagios monitoring your Hadoop & NoSQL clusters. Covers every Hadoop vendor's management API and every major NoSQL technology (HBase, Cassandra, MongoDB, Elasticsearch, Solr, Riak, Redis etc.) as well as message queues (Kafka, RabbitMQ), continuous integration (Jenkins, Travis CI) and traditional infrastructure (SSL, Whois, DNS, Linux)

• Nagios Plugin Kafka - Kafka API pub/sub Nagios Plugin written in Scala with Kerberos support

• The Advanced Nagios Plugins Collection - 450+ programs for Nagios monitoring your Hadoop & NoSQL clusters. Covers every Hadoop vendor's management API and every major NoSQL technology (HBase, Cassandra, MongoDB, Elasticsearch, Solr, Riak, Redis etc.) as well as message queues (Kafka, RabbitMQ), continuous integration (Jenkins, Travis CI) and traditional infrastructure (SSL, Whois, DNS, Linux)

• HAProxy Configs - 80+ HAProxy Configs for Hadoop, Big Data, NoSQL, Docker, Elasticsearch, SolrCloud, HBase, Cloudera, Hortonworks, MapR, MySQL, PostgreSQL, Apache Drill, Hive, Presto, Impala, ZooKeeper, OpenTSDB, InfluxDB, Prometheus, Kibana, Graphite, SSH, RabbitMQ, Redis, Riak, Rancher etc.

• Dockerfiles - 50+ DockerHub public images for Docker & Kubernetes - Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Mesos, Consul, Riak, OpenTSDB, Jython, Advanced Nagios Plugins & DevOps Tools repos on Alpine, CentOS, Debian, Fedora, Ubuntu, Superset, H2O, Serf, Alluxio / Tachyon, FakeS3

• HashiCorp Packer templates - Linux automated bare-metal installs and portable virtual machines OVA format appliances using HashiCorp Packer, Redhat Kickstart, Debian Preseed and Ubuntu AutoInstaller / Cloud-Init

• Diagrams-as-Code - Cloud & Open Source architecture diagrams with Python & D2 source code provided - automatically regenerated via GitHub Actions CI/CD - AWS, GCP, Kubernetes, Jenkins, ArgoCD, Traefik, Kong API Gateway, Nginx, Redis, PostgreSQL, Kafka, Spark, web farms, event processing...

Stargazers over time