Concourse CI - develop+release+pr

.gitignore

@@ -20,6 +20,7 @@ tmp
 my*.yml
 releases/**/*.tgz
 blobs
+version
 
 /logsearch-config
 /logsearch-config.tgz

README.md

@@ -49,6 +49,20 @@ integration, run the included script...
     SUCCESS
 
 
+### Release Channels
+
+We currently maintain two release channels...
+
+ * **final** - our latest stable, final release (also available on [bosh.io](http://bosh.io/releases/github.com/logsearch/logsearch-boshrelease))
+ * **develop** - the latest build from our `develop` branch which has passed all our tests
+
+You can find the artifacts with the following URI templates...
+
+ * Latest Version: `https://logsearch-boshrelease.s3.amazonaws.com/{channel}/version`
+ * Release Tarball: `https://logsearch-boshrelease.s3.amazonaws.com/{channel}/release/logsearch-{version}.tgz`
+ * Source Tarball: `https://logsearch-boshrelease.s3.amazonaws.com/{channel}/src/logsearch-src-{version}.tgz`
+
+
 ## License
 
 [Apache License 2.0](./LICENSE)

ci/.gitignore

@@ -1 +1,3 @@
-/config/private*.yml
+/config/*-private.yml
+/config/pr/
+/config/pr.txt

ci/README.md

@@ -1,14 +1,50 @@
-If you'll be uploading stuff, be sure to put your credentials in `config/private-main.yml`. Override buckets for testing.
+# CI (via Concourse)
 
-You might want to update your experimental concourse pipeline...
+Continuous Integration is nice when it all works out...
 
-    $ fly configure \
-      --config pipelines/concourse.yml \
-      --vars-from config/default.yml \
-      --vars-from config/private-main.yml \
-      logsearch-boshrelease
 
-You might want to serve your local repository while tweaking scripts...
+## Pipelines
 
-    $ git daemon --base-path=../ -v --listen=172.23.240.4 --port=9191 --export-all
-    $ echo 'git-master-uri: "git://172.23.240.4:9191/"' >> config/private-main.yml
+Unconventionally, we've divided up several pipeline components to make them easier to reuse. Here are the main pipeline pieces...
+
+0. `dev` - this pipeline helps validate the quality of particular branches. It will monitor a particular branch for changes and, upon change, will run through all the appropriate tests. If they pass, the version file will be updated.
+0. `release` - this pipeline helps create final releases. Manually trigger the major, minor, or patch job to kick off creating the final release which will be committed, tagged, uploaded, and pushed.
+0. `pr` - this pipeline locally merges pull requests for testing and includes the `dev` pipeline to run the tests.
+
+To combine them, the `generate` script in their respective directories will dump the resulting pipeline to STDOUT.
+
+
+## Variables
+
+Configuration files with the variables `fly` wants are located in `ci/config/*.yml`. When customizing or adding secrets use a separate file with the convention of `ci/config/*-private.yml` (they have a higher precedence and will not be committed).
+
+
+## Configuration
+
+To re-apply all the pipelines, use the `fly-configure` script...
+
+    $ FLY_TARGET=cilabs-meta ./ci/bin/fly-configure
+
+
+## Pull Requests
+
+If you want to start testing a particularl pull request, append the pull number to the list of approved pulls...
+
+    $ echo 180 >> ci/config/pr.txt
+
+After running `fly-configure` to install the new PR pipeline, you'll want to unpause the pipeline to get it started. Going forward, it will automatically run when new commits are pushed for the PR. It does not currently update commit statuses within GitHub.
+
+
+## Bootstrap
+
+If you're setting up pipelines from scratch or for your own fork, you might need to initialize a few files...
+
+    echo -n "22+dev.4" > version
+    aws s3api put-object --bucket=logsearch-boshrelease --key=develop/version --body=version
+    aws s3api put-object --bucket=logsearch-boshrelease --key=develop/version-wip --body=version
+
+    echo -n "22.0.0" > version
+    aws s3api put-object --bucket=logsearch-boshrelease --key=final/version --body=version
+    aws s3api put-object --bucket=logsearch-boshrelease --key=final/version-wip --body=version
+
+    rm version

ci/bin/fly-configure

@@ -3,9 +3,107 @@
 set -e
 set -u
 
-fly -t "${FLY_TARGET}" \
-  configure \
-  --config ci/pipelines/concourse.yml \
-  --vars-from ci/config/default.yml \
-  --vars-from ci/config/private.yml \
-  "${1:-logsearch-boshrelease}"
+function varsfrom {
+  [ -e ci/config/$1.yml ] && echo "--vars-from ci/config/$1.yml"
+  [ -e ci/config/$1-private.yml ] && echo "--vars-from ci/config/$1-private.yml"
+}
+
+function varslookup {
+  cat ci/config/common.yml $( [ -e ci/config/common-private.yml ] && echo ci/config/common-private.yml ) \
+    | grep "^$1:" \
+    | tail -n1 \
+    | awk '{ print $2 }' \
+    | sed -E 's/"(.+)"/\1/'
+}
+
+PR_BASE_REPO_OWNER=$( varslookup github-user )
+PR_BASE_REPO_NAME=$( varslookup github-repository )
+
+YML=$( mktemp pipeline-XXXXXX )
+
+#
+# develop branch
+#
+
+echo "# develop"
+
+./ci/pipelines/dev/generate > $YML
+
+echo "y" | fly -t "${FLY_TARGET}" \
+  configure --config $YML \
+  $( varsfrom common ) \
+  $( varsfrom develop ) \
+  --var repo-resource=repo-dev \
+  --var release-resource=release-dev \
+  "${1:-logsearch}-develop"
+
+echo ""
+
+#
+# release
+#
+
+echo "# release"
+
+./ci/pipelines/release/generate > $YML
+
+echo "y" | fly -t "${FLY_TARGET}" \
+  configure --config $YML \
+  $( varsfrom common ) \
+  $( varsfrom develop ) \
+  $( varsfrom release ) \
+  "${1:-logsearch}-release"
+
+echo ""
+
+#
+# dynamic prs
+#
+
+if [ -f ci/config/pr.txt ] ; then
+  mkdir -p ci/config/pr
+
+  while read -r PR_NUMBER ; do
+    echo "# pr$PR_NUMBER"
+
+    PR_GITHUB=$( wget -qO- "https://api.github.com/repos/$PR_BASE_REPO_OWNER/$PR_BASE_REPO_NAME/pulls/$PR_NUMBER" )
+
+    PR_BASE_REF=$( echo "${PR_GITHUB}" | jq -r '.base.ref' )
+
+    PR_HEAD_REPO_OWNER=$( echo "${PR_GITHUB}" | jq -r '.head.repo.owner.login' )
+    PR_HEAD_REPO_NAME=$( echo "${PR_GITHUB}" | jq -r '.head.repo.name' )
+    PR_HEAD_REF=$( echo "${PR_GITHUB}" | jq -r '.head.ref' )
+
+    cat > ci/config/pr/$PR_NUMBER.yml << EOF
+repo-branch: "$PR_BASE_REF"
+repo-pr-uri: "https://github.com/$PR_HEAD_REPO_OWNER/$PR_HEAD_REPO_NAME.git"
+repo-pr-branch: "$PR_HEAD_REF"
+EOF
+
+    if [[ "closed" == $( echo "$PR_GITHUB" | jq '.state' ) ]] ; then
+      PAUSED=true
+    else
+      PAUSED=false
+    fi
+
+    ./ci/pipelines/pr/generate $PR_NUMBER > $YML
+
+    echo "y" | fly -t "${FLY_TARGET}" \
+      configure --config $YML \
+      $( varsfrom common ) \
+      $( varsfrom develop ) \
+      $( varsfrom pr ) \
+      $( varsfrom pr/$PR_NUMBER ) \
+      --var repo-resource=repo-pr \
+      --var release-resource=release-pr \
+      "${1:-logsearch}-pr${PR_NUMBER}"
+
+    echo ""
+  done < ci/config/pr.txt
+fi
+
+#
+# cleanup
+#
+
+rm $YML

ci/config/common.yml

@@ -0,0 +1,43 @@
+#
+# Where shall we store our artifacts?
+#
+
+artifact-bucket: "logsearch-boshrelease"
+
+#
+# GitHub Repository where we live
+#
+
+github-user: "logsearch"
+github-repository: "logsearch-boshrelease"
+
+#
+# Git Repository Access
+#
+
+repo-uri: "git@github.com:logsearch/logsearch-boshrelease.git"
+repo-key: ""
+
+
+#
+# GitHub Token for making API calls
+#
+
+github-api-token: ~
+
+
+#
+# AWS Credentials (for uploading to the S3 blobstore)
+#
+
+aws-access-key: ~
+aws-secret-key: ~
+
+
+#
+# BOSH Lite access for running tests
+#
+
+bosh-lite-target: "https://192.0.2.1:25555"
+bosh-lite-username: "admin"
+bosh-lite-password: "admin"

ci/config/develop.yml

@@ -0,0 +1,2 @@
+repo-dev-branch: "develop"
+artifact-dev-prefix: "develop/"

ci/config/pr.yml

@@ -0,0 +1 @@
+repo-pr-key: ~

ci/config/release.yml

@@ -0,0 +1 @@
+artifact-release-prefix: "final/"