Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

convert admin to use same test/lint standard as the main pipelines #682

Merged
merged 3 commits into from Apr 30, 2019
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
6 changes: 3 additions & 3 deletions Makefile
Expand Up @@ -6,19 +6,19 @@ ifdef DEPLOYMENT
endif

ifndef JENKINS_URL
ifndef USE_CONCOURSE_COMPOSE
ifndef ON_CONCOURSE
DOCKER_COMPOSE += -f docker-compose.development.yml
endif
endif

ifdef USE_CONCOURSE_COMPOSE
ifdef ON_CONCOURSE
DOCKER_COMPOSE += -f docker-compose.concourse.yml
endif

DOCKER_BUILD_CMD = BUNDLE_INSTALL_FLAGS="$(BUNDLE_FLAGS)" $(DOCKER_COMPOSE) build

build:
ifndef USE_CONCOURSE_COMPOSE
ifndef ON_CONCOURSE
$(DOCKER_COMPOSE) build
endif

Expand Down
30 changes: 20 additions & 10 deletions ci/pipelines/pr.yml
Expand Up @@ -46,48 +46,58 @@ jobs:
plan:
- aggregate:
- do:
- get: govwifi-admin
- get: src
resource: pull-request
trigger: true
version: every
- <<: *update-status-pending
- get: mysql-image

# fetch each image and place in the docker-cache directory
- get: docker-cache/mysql-image
resource: mysql-image
params: {format: oci}
- get: ruby-image
- get: docker-cache/ruby-image
resource: ruby-image
params: {format: oci}
- get: nginx-image
- get: docker-cache/nginx-image
resource: nginx-image
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need Nginx? Pretty sure we just use Puma.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's used by the fake-register

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes ok

params: {format: oci}

- get: runner

# used to avoid building multiple times down the road
- task: pre-build
privileged: true
image: runner
file: govwifi-admin/ci/tasks/pre-build.yml
file: src/ci/tasks/pre-build.yml

- aggregate:
- task: lint
privileged: true
image: runner
file: govwifi-admin/ci/tasks/lint.yml
file: src/ci/tasks/lint.yml
- task: test
privileged: true
image: runner
file: govwifi-admin/ci/tasks/test.yml
file: src/ci/tasks/test.yml

on_failure:
do:
- <<: *update-status-failure
on_success:
do:
- <<: *update-status-success

common:
update-status-commands:
update-status-base: &update-status-base
put: update-status
resource: pull-request
get_params:
skip_download: true

update-status-params-base: &update-status-params-base
path: govwifi-admin
context: 'concourse/ci'
path: src
context: 'pr'

update-status-pending: &update-status-pending
<<: *update-status-base
Expand Down
19 changes: 7 additions & 12 deletions ci/tasks/lint.yml
@@ -1,20 +1,15 @@
---
platform: linux

image_resource:
type: registry-image
source:
repository: cybojenix/govwifi-concourse-runner

inputs:
- name: govwifi-admin
- name: mysql-image
- name: nginx-image
- name: ruby-image
- name: govwifi-admin-prebuilt
- name: src
- name: docker-cache/mysql-image
- name: docker-cache/nginx-image
- name: docker-cache/ruby-image
- name: docker-cache/app-prebuilt

params:
USE_CONCOURSE_COMPOSE: true
ON_CONCOURSE: true

run:
path: govwifi-admin/ci/tasks/scripts/lint.sh
path: src/ci/tasks/scripts/lint.sh
26 changes: 13 additions & 13 deletions ci/tasks/pre-build.yml
@@ -1,25 +1,25 @@
---
platform: linux

image_resource:
type: registry-image
source:
repository: cybojenix/govwifi-concourse-runner

inputs:
- name: govwifi-admin
- name: mysql-image
- name: nginx-image
- name: ruby-image
- name: src
- name: docker-cache/mysql-image
- name: docker-cache/nginx-image
- name: docker-cache/ruby-image

outputs:
- name: govwifi-admin-prebuilt
# aligns with PREBUILT_TAG
- name: docker-cache/app-prebuilt

caches:
- path: govwifi-admin-prebuilt-cached
- path: docker-cache/app-prebuilt-cached

params:
USE_CONCOURSE_COMPOSE: true
# Expose that we're using Concourse
ON_CONCOURSE: true
# Set this to custom image names for preloading the app image. see `docker-compose.concourse.yml`
PREBUILT_TAG: 'app-prebuilt'
PREBUILT_CACHED_DIR: 'docker-cache/app-prebuilt-cached'

run:
path: govwifi-admin/ci/tasks/scripts/pre-build.sh
path: src/ci/tasks/scripts/pre-build.sh
8 changes: 5 additions & 3 deletions ci/tasks/scripts/lint.sh
@@ -1,10 +1,12 @@
#!/bin/bash

set -v -e -u -o pipefail
set -e -u -o pipefail

./govwifi-admin/ci/tasks/scripts/with-docker.sh
./src/ci/tasks/scripts/with-docker.sh

cd govwifi-admin
workspace_dir="${PWD}"
cd src

make lint

cd "${workspace_dir}"
18 changes: 10 additions & 8 deletions ci/tasks/scripts/pre-build.sh
@@ -1,16 +1,18 @@
#!/bin/bash

set -v -e -u -o pipefail
set -e -u -o pipefail

./govwifi-admin/ci/tasks/scripts/with-docker.sh
./src/ci/tasks/scripts/with-docker.sh

cd govwifi-admin

[[ -f "../govwifi-admin-prebuilt-cached/image.tar" ]] && docker load -qi "../govwifi-admin-prebuilt-cached/image.tar"
workspace_dir="${PWD}"
prebuilt_dir="${workspace_dir}/docker-cache/${PREBUILT_TAG}"
prebuilt_cached_dir="${workspace_dir}/${PREBUILT_CACHED_DIR}"

cd src

make prebuild
docker tag "$(docker-compose images -q app)" "govwifi-admin-app-prebuilt"
docker save "govwifi-admin-app-prebuilt" -o "../govwifi-admin-prebuilt/image.tar"
cp "../govwifi-admin-prebuilt/image.tar" "../govwifi-admin-prebuilt-cached/image.tar"
docker tag "$(docker-compose images -q app)" "${PREBUILT_TAG}"
docker save "${PREBUILT_TAG}" -o "${prebuilt_dir}/image.tar"
cp "${prebuilt_dir}/image.tar" "${prebuilt_cached_dir}/image.tar"

cd "${workspace_dir}"
9 changes: 5 additions & 4 deletions ci/tasks/scripts/test.sh
@@ -1,11 +1,12 @@
#!/bin/bash

set -v -e -u -o pipefail
set -e -u -o pipefail

./govwifi-admin/ci/tasks/scripts/with-docker.sh
./src/ci/tasks/scripts/with-docker.sh

cd govwifi-admin
workspace_dir="${PWD}"
cd src

make test

cd /
cd "${workspace_dir}"
19 changes: 14 additions & 5 deletions ci/tasks/scripts/with-docker.sh
Expand Up @@ -2,19 +2,28 @@

set -e -u -o pipefail

# Launches the docker daemon, and loads our cached images from the `docker-cache` folder.

source /docker-helpers.sh
start_docker

function load_layers() {

# don't do anything if we don't have cache
[[ ! -d 'docker-cache' ]] && return 0;
[[ -z "$(ls docker-cache/*/image.tar)" ]] && return 0;

echo "loading docker layer cache"
pids=
docker load -qi mysql-image/image.tar & pids[0]=$!
docker load -qi nginx-image/image.tar & pids[1]=$!
docker load -qi ruby-image/image.tar & pids[2]=$!
[[ -f "govwifi-admin-prebuilt/image.tar" ]] && docker load -qi "govwifi-admin-prebuilt/image.tar" & pids[3]=$!

index=0
for cached_image in docker-cache/*/image.tar; do
docker load -qi "${cached_image}" & pids[${index}]=$!
index="$(( "${index}" + 1 ))"
done

for pid in ${pids[*]}; do
wait "$pid" || true # we don't care about the return code of docker loading
wait "$pid"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just out of curiosity, we now care about the return code of docker loading?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, explanation for this one.

This occurred when the cache wasn't populated, and relates to the old line 14.

[[ -f "govwifi-admin-prebuilt/image.tar" ]] && docker load -qi "govwifi-admin-prebuilt/image.tar" & pids[3]=$!

The & pids[3]=$! is run on the whole statement, not just the part after the condition check.
This means that for when the file doesn't exist, the process of that statement will return an exit code of 1 due to the [[ ... ]] returning falsely.

adding wait || true was a hack to work around the issue as I didn't fully understand the cause before.

With the move to globbing, the only issues we'll get are ones we want to raise

done
}
load_layers
19 changes: 7 additions & 12 deletions ci/tasks/test.yml
@@ -1,20 +1,15 @@
---
platform: linux

image_resource:
type: registry-image
source:
repository: cybojenix/govwifi-concourse-runner

inputs:
- name: govwifi-admin
- name: mysql-image
- name: nginx-image
- name: ruby-image
- name: govwifi-admin-prebuilt
- name: src
- name: docker-cache/mysql-image
- name: docker-cache/nginx-image
- name: docker-cache/ruby-image
- name: docker-cache/app-prebuilt

params:
USE_CONCOURSE_COMPOSE: true
ON_CONCOURSE: true

run:
path: govwifi-admin/ci/tasks/scripts/test.sh
path: src/ci/tasks/scripts/test.sh
4 changes: 2 additions & 2 deletions docker-compose.concourse.yml
Expand Up @@ -2,9 +2,9 @@ version: '3.4'

services:
app:
image: govwifi-admin-app-prebuilt
image: app-prebuilt
build:
cache_from:
- govwifi-admin-app-prebuilt
- app-prebuilt
args:
RUN_PRECOMPILATION: 'false'