Initialise bash-lib repo

This commit adds the structure of the libraries and a few funcions.
Most of the code in this commit is geared towards making sure this
repo stays tested and documented in future.

The following things are checked:
    * All functions are documented
    * All functions are tested
    * All scripts are linted

See Readme for a list of the functions that are included, and for more
information on testing with BATS.

.gittrees

@@ -0,0 +1,13 @@
+# Git Subtrees
+
+# The advantage of subtrees is that users don't have to care about them - its
+# just a single repo. The disadvantage is that git doesn't track the metadata
+# as it does for submodules.
+
+# This file provides an enumeration of the subtrees in this repo, and the URLs
+# they came from.
+
+# subtree_path remote_url remote_name
+test-utils/bats https://github.com/bats-core/bats bats
+test-utils/bats-support https://github.com/ztombol/bats-support bats-support
+test-utils/bats-assert-1 https://github.com/jasonkarns/bats-assert-1 bats-assert-1

Jenkinsfile

@@ -0,0 +1,47 @@
+#!/usr/bin/env groovy
+
+pipeline {
+  agent { label 'executor-v2' }
+
+  options {
+    timestamps()
+    buildDiscarder(logRotator(numToKeepStr: '30'))
+  }
+
+  triggers {
+    cron(getDailyCronString())
+  }
+
+  environment {
+    BATS_OUTPUT_FORMAT="junit"
+  }
+
+  stages {
+
+    stage('BATS Tests') {
+      steps {
+        sh './tests-for-this-repo/run-bats-tests'
+      }
+    }
+
+    stage('Python Linting') {
+      steps {
+        sh './tests-for-this-repo/run-python-lint'
+      }
+    }
+
+    stage('Secrets Leak Check') {
+      steps {
+        sh './tests-for-this-repo/run-gitleaks'
+      }
+    }
+
+  }
+
+  post {
+    always {
+      junit '*-junit.xml'
+      cleanupAndNotify(currentBuild.currentResult)
+    }
+  }
+}

README.md

@@ -1,2 +1,196 @@
-# bashlib
-Common bash functions for use in test pipelines
+# bash-lib
+```
+                   _______________  _______________
+                 .'               .'               .|
+               .'               .'               .' |
+             .'_______________.'______________ .'   |
+             | ___ _____ ___ || ___ _____ ___ |     |
+             ||_=_|__=__|_=_||||_=_|__=__|_=_||     |
+       ______||_____===_____||||_____===_____||     | __________
+    .'       ||_____===_____||||_____===_____||    .'          .'|
+  .'         ||_____===_____||||_____===_____||  .'          .'  |
+.'___________|_______________||_______________|.'__________.'    |
+|.----------.|.-----___-----.||.-----___-----.||    |_____.----------.
+|]          |||_____________||||_____________|||  .'      [          |
+||          ||.-----___-----.||.-----___-----.||.'        |          |
+||          |||_____________||||_____________|||==========|          |
+||          ||.-----___-----.||.-----___-----.||    |_____|          |
+|]         o|||_____________||||_____________|||  .'      [        'o|
+||          ||.-----___-----.||.-----___-----.||.'        |          |
+||          |||             ||||_____________|||==========|          |
+||          |||             |||.-----___-----.||    |_____|          |
+|]          |||             ||||             |||  .'      [          |
+||__________|||_____________||||_____________|||.'________|__________|
+''----------'''------------------------------'''----------''
+            (o)LGB                           (o)
+```
+
+The place to store functions that are used in pipelines for multiple repos.
+
+Please add whatever is useful to you, but keep it tidy so its still useful to everyone else :)
+
+## Usage
+
+Add bash-lib into your project in the way that best fits your workflow. The only requirement is that you **pin the version of
+bash-lib that you use**. This is important so that changes to bash-lib do not have the power to break all projects that use
+bash-lib. Your project can then test updates to bash-lib and roll forward periodicly.
+
+Options:
+* Add a submodule: they are an easy way to integrate bash-lib and automatically use a single SHA until manually updated. Submodules add a pointer from a mount point in your repo to the external repo (bash-lib), and require workflow changes to ensure that pointer is derferenced during clone, checkout and some other opertaions.
+* Add a subtree: This repo uses subtrees to pull in test dependencies. Subtrees copy an external repo into a subdirectory of the host repo, no workflow changes are required. Subtrees naturally keep a single version of bash-lib until explicitly updated. Note that subtree merge commits do not rebase well :warning:, so best to keep subtree updates in separate PRs from normal commits.
+* Clone bash-lib in your deployment process, bash-lib doesn't have to be within your repo, just needs to be somewhere where your scripts can source [init](init). This is where it's most important that you implement a mechanism to always use the same SHA, as a **clone will track master by default, which is not an allowed use of bash-lib**.
+
+Once you have bash-lib cloned in your project, you source two things:
+
+1. Source `bash-lib/init`. This ensures submodules are initalised and sets the BASH_LIB_DIR env var to the absolute path to the bash-lib dir. This makes it easy to source libraries from other scripts.
+2. Source `${BASH_LIB_DIR}/lib-name/lib` for any libraries you are interested in.
+
+You are now ready to use bash-lib functions :)
+
+## Structure
+The `/init` script sets up everything required to use the library, most
+importantly the `BASH_LIB_DIR` variable which gives the absolute path to the root
+of the library and should be used for sourcing the modules.
+
+The repo is organized into libraries, each library is a directory that has a
+lib file. Sourcing the lib for a library should expose all the functions
+that library offers. The lib file may source or reference other supporting
+files within it's directory.
+
+```
+.
+├── libname
+│   ├── lib
+│   └── supporting-file
+├── init # init script, source this first
+├── run-tests # top level test script, executes all tests
+├── secrets.yml # secrets required for executing tests
+├── test-utils
+│   ├── bats # subtree
+│   ├── bats-assert-1 # subtree
+│   ├── bats-support # subtree
+│   ├── lib
+│   └── tap2junit
+└── tests-for-this-repo
+    ├── filehandling.bats
+    ├── fixtures #
+    │   └── libname # Dir containing test fixtures for a library
+    ├── tap2junit
+    ├── libname.bats # contains tests for libname/lib
+    ├── python-lint # supporting files for python lint
+    ├── run-bats-tests # script to run bats tests
+    ├── run-gitleaks # script to check for leaked secrets
+    └──  run-python-lint # script to run python lint
+```
+## Style Guide
+Follow the [google shell style guide](https://google.github.io/styleguide/shell.xml#Naming_Conventions).
+TL;DR:
+1. Use snake_case function and variable names
+1. Use `function` when declaring functions.
+
+
+## Contents
+
+<!-- html table due to markdown's lack of support for lists within tables -->
+<table>
+  <thead>
+    <tr>
+      <th>Library</th>
+      <th>Description</th>
+      <th>Functions</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><a href="filehandling/lib">filehandling</a></td>
+      <td>Functions relating to file and path handling
+      <td>
+        <ol>
+          <li> <b>abs_path</b>: Ensure a path is absolute</li>
+        </ol>
+      </td>
+    </tr>
+    <tr>
+      <td><a href="git/lib">git</a></td>
+      <td>Git helpers</td>
+      <td>
+        <ol>
+          <li><b>repo_root</b>: Find the root of the current git repo.</li>
+          <li><b>all_files_in_repo</b>: List files tracked by git.</li>
+          <li><b>remote_latest_tag</b>: Returns the symbolic name of the latest tag from a remote.</li>
+          <li><b>remote_latest_tagged_commit</b>: Returns the SHA of the most recently tagged commit in a remote repo (<code>tag^{}</code>).</li>
+          <li><b>remote_sha_for_ref</b>: Returns the SHA for a given ref from a named remote.</li>
+          <li><b>remote_tag_for_sha</b>: Returns the tag corresponding to a SHA from a named remote - if there is one.</li>
+          <li>tracked_files_excluding_subtrees: List files tracked by git, but excluding any files that are in paths listed in <code>.gittrees</code>.</li>
+          <li><b>cat_gittrees</b>: Returns the contents of .gittrees from the top level of the repo, excluding any comments. Fails if .gittrees is not present.</li>
+        </ol>
+      </td>
+    </tr>
+      <td><a href="helpers/lib">helpers</a></td>
+      <td>Bash scripting helpers</td>
+      <td>
+        <ol>
+          <li><b>die</b>: print message and exit 1</li>
+          <li><b>spushd/spopd</b>: Safe verisons of pushd & popd that call die if the push/pop fails, they also drop stdout. </li>
+        </ol>
+      </td>
+    </tr>
+    <tr>
+      <td><a href="k8s/lib">k8s</a></td>
+      <td>Utils for connecting to K8s</td>
+      <td>
+        <ol>
+          <li><b>build_gke_image</b>: Build docker image for running kubectl commands against GKE.</li>
+          <li><b>delete_gke_image</b>: Delete image from GKE.</li>
+          <li><b>run_docker_gke_command</b>: Run command in gke-utils container, already authenticated to k8s cluster.</li>
+        </ol>
+      </td>
+    </tr>
+    <tr>
+      <td><a href="logging/lib">logging</a></td>
+      <td>Helpers related to login</td>
+      <td>
+        <ol>
+          <li><b>announce</b>: Echo message in ascii banner to distinguish it from other log messages.</li>
+        </ol>
+      </td>
+    </tr>
+    <tr>
+      <td><a href="test-utils/lib">test-utils</a></td>
+      <td>Helpers for executing tests</td>
+      <td>
+        <ol>
+          <li><b>shellcheck_script</b>: Execute shellcheck against a script, uses docker.</li>
+          <li><b>find_scripts</b>: Find git tracked files with extension.</li>
+          <li><b>tap2junit</b>: Convert a subset of <a href="http://testanything.org/">TAP</a> to JUnit XML. Retains logs for errors.</li>
+        </ol>
+      </td>
+    </tr>
+  </tbody>
+</table>
+
+## Testing
+Tests are written using [BATS](https://github.com/bats-core/bats). Each libould have a `lib-name.bats` file in [tests-for-this-repo](/tests-for-this-repo).
+Asserts are provided by [bats-assert-1](https://github.com/jasonkarns/bats-assert-1). The value in these is that they provide useful debugging output when the assertion fails, eg expected x got y.
+
+Example:
+```bash
+# source support and assert libraries
+. "${BASH_LIB_DIR}/test-utils/bats-support/load.bash"
+. "${BASH_LIB_DIR}/test-utils/bats-assert-1/load.bash"
+
+# source the library under test
+. "${BASH_LIB_DIR}/git/lib"
+
+# define a test that calls a library function
+@test "it does the thing" {
+  some_prep_work
+  # run is a wrapper that catches failures so that assertsions can be run,
+  # otherwise the test would immediately fail.
+  run does_the_thing
+  assert_success
+  assert_output "thing done"
+}
+```
+
+Test fixtures should go in /tests-for-this-repo/[fixtures](tests-for-this-repo/fixtures)/lib-name.

filehandling/lib

@@ -0,0 +1,24 @@
+#!/bin/bash
+
+: "${BASH_LIB_DIR:?BASH_LIB_DIR must be set. Please source bash-lib/init before other scripts from bash-lib.}"
+. "${BASH_LIB_DIR}/helpers/lib"
+
+#https://stackoverflow.com/a/23002317
+function abs_path() {
+    # generate absolute path from relative path
+    # $1     : relative filename
+    # return : absolute path
+    if [ -d "$1" ]; then
+        # dir
+        (spushd "$1"; pwd)
+    elif [ -f "$1" ]; then
+        # file
+        if [[ $1 = /* ]]; then
+            echo "$1"
+        elif [[ $1 == */* ]]; then
+            echo "$(spushd "${1%/*}"; pwd)/${1##*/}"
+        else
+            echo "$(pwd)/$1"
+        fi
+    fi
+}

git/lib

@@ -0,0 +1,79 @@
+#!/bin/bash
+
+: "${BASH_LIB_DIR:?BASH_LIB_DIR must be set. Please source bash-lib/init before other scripts from bash-lib.}"
+. "${BASH_LIB_DIR}/helpers/lib"
+
+# Get the top level of a git repo
+function repo_root(){
+    git rev-parse --show-toplevel
+}
+
+# List files tracked by git
+function all_files_in_repo(){
+    git ls-tree -r HEAD --name-only
+}
+
+# Find the latest tag available at a repo url
+# Returns tag name, not sha
+function remote_latest_tag(){
+    local -r remote_url="${1}"
+    # In ls-remote the ^{} suffix refers to a peeled/dereferenced object.
+    # eg refs/tags/v0.0.1^{} shows the SHA of the commit that was tagged,
+    # not the SHA of the tag itself.
+    # Adding --refs hides peeled tags
+    git ls-remote --tags --refs --quiet \
+        "${remote_url}" \
+        | tail -n 1 \
+        | cut -f 2 \
+        | sed -e 's+refs/tags/++'
+}
+
+# Find the SHA of the latests commit to be tagged in a remote repo
+function remote_latest_tagged_commit(){
+    local -r remote="${1}"
+    local -r tag="$(remote_latest_tag "${remote}")"
+    git ls-remote "${remote}" | awk "/refs\/tags\/${tag}\^/{print \$1}"
+}
+
+function remote_sha_for_ref(){
+    local -r remote="${1}"
+    local -r ref="${2}"
+
+    # First try adding ^{} to the ref, incase it's a tag
+    # and needs peeling. If nothing is found for that,
+    # try without.
+    peeled_ref=$(
+        git ls-remote "${remote}" \
+            | awk "/${ref}[^$]/{print \$1}"
+    )
+
+    if [[ -n "${peeled_ref}" ]]; then
+        echo "${peeled_ref}"
+    else
+        git ls-remote "${remote}" \
+            | awk "/${ref}/{print \$1}"
+    fi
+}
+
+function remote_tag_for_sha(){
+    local -r remote="${1}"
+    local -r sha="${2}"
+    git ls-remote "${remote}" \
+        | awk -F'/' "/${sha}.*tag/{ gsub(/\^\{\}\$/, \"\"); print \$3 }"
+}
+
+
+## Minimal git subtree functionality required for tests to pass
+# full subtree functionality is not ready for merge.
+function cat_gittrees(){
+    local -r git_trees="$(repo_root)/.gittrees"
+    local -r subtrees_file_format=".gittrees should contain one subtree per line,\
+space seperated with three fields: subtree_path renmote_url remote_name"
+    [[ -e "${git_trees}" ]] || die ".gittrees file ${git_trees} not found. ${subtrees_file_format}"
+    grep -E -v '^\s*$|^\s*#' "$(repo_root)/.gittrees"
+}
+
+function tracked_files_excluding_subtrees(){
+    subtrees="$(cat_gittrees | awk '{print $1}' | paste -sd '|' -)"
+    all_files_in_repo | grep -E -v "${subtrees}"
+}

helpers/lib

@@ -0,0 +1,20 @@
+#!/bin/bash
+
+: "${BASH_LIB_DIR:?BASH_LIB_DIR must be set. Please source bash-lib/init before other scripts from bash-lib.}"
+
+function die(){
+    echo "${@}"
+    exit 1
+}
+
+#safe pushd
+function spushd(){
+    if ! pushd "${1}" >/dev/null; then
+        die "pushd ${1} failed :("
+    fi
+}
+
+#safe popd
+function spopd(){
+    popd >/dev/null || die "popd failed :("
+}