Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chore: Enable PR testing in Drone #26189

Merged
merged 82 commits into from Jul 10, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
82 commits
Select commit Hold shift + click to select a range
e70a9c2
Add Drone configuration
aknuds1 Jun 13, 2020
7ab70b5
Add more steps
aknuds1 Jun 13, 2020
b8395db
Add more steps
aknuds1 Jun 14, 2020
bf835f4
Build front-end before testing it
aknuds1 Jun 14, 2020
a782548
Upgrade grafana/build-container
aknuds1 Jun 14, 2020
d073ed4
Add packaging step
aknuds1 Jun 14, 2020
61269a2
Trigger on push
aknuds1 Jun 14, 2020
bd91311
Remove some steps
aknuds1 Jun 14, 2020
d6e4f80
Enable steps
aknuds1 Jun 14, 2020
471c43e
Install Dockerize
aknuds1 Jun 14, 2020
57a2513
Use node image for test-frontend
aknuds1 Jun 14, 2020
9f316db
Increase number of test workers
aknuds1 Jun 14, 2020
1821e58
Make plugin installation depend on frontend tests
aknuds1 Jun 14, 2020
3c8a652
Make integration tests depend on frontend tests
aknuds1 Jun 14, 2020
4f32fa1
Use grafana/build-container also for front-end tests
aknuds1 Jun 14, 2020
ef9853e
Upgrade dependencies in order to fix front-end tests
aknuds1 Jun 14, 2020
98160b0
Depend on es-check
aknuds1 Jun 14, 2020
8568a8e
Dont' depend on tests before building front-end
aknuds1 Jun 14, 2020
35f8d7f
Add more steps
aknuds1 Jun 14, 2020
63a8cae
Fix packaging
aknuds1 Jun 14, 2020
8e3d5e0
Simplify
aknuds1 Jun 14, 2020
3b834ef
Try to build images
aknuds1 Jun 14, 2020
f810bbc
Fix e2e tests
aknuds1 Jun 14, 2020
960152e
Remove steps
aknuds1 Jun 14, 2020
0ec1efc
Install netcat
aknuds1 Jun 14, 2020
3033d4a
Include golangci-lint with grafana/build-container
aknuds1 Jun 15, 2020
4175c0c
Build storybook and docs website
aknuds1 Jun 17, 2020
ed46e3d
Fix e2e tests
aknuds1 Jun 17, 2020
257ab88
Use build image with root user
aknuds1 Jun 17, 2020
f84edf3
Drop CircleCI dependencies
aknuds1 Jun 18, 2020
753c951
Fix e2e tests
aknuds1 Jun 18, 2020
0b48c5f
Fix e2e under Drone
aknuds1 Jun 18, 2020
9b1755c
Execute e2e server separately
aknuds1 Jun 18, 2020
9fe3d07
Use own plugin for building Docker images
aknuds1 Jun 18, 2020
50c2b10
Use Starlark to configure Drone
aknuds1 Jun 19, 2020
511959e
Add enterprise steps to pipeline
aknuds1 Jun 19, 2020
a37a58a
Add more enterprise steps to pipeline
aknuds1 Jun 19, 2020
d09835a
Maintain Yarn cache
aknuds1 Jun 19, 2020
8f3f581
Build enterprise Docker images
aknuds1 Jun 19, 2020
2e53dc0
Build Ubuntu Docker images
aknuds1 Jun 19, 2020
f19cd91
Refactor
aknuds1 Jun 19, 2020
0eee57a
Add Postgres integration test
aknuds1 Jun 19, 2020
abb5862
Add MySQL integration test
aknuds1 Jun 19, 2020
be20d90
Fix integration tests
aknuds1 Jun 19, 2020
8639721
Parameterize integration test DB connections
aknuds1 Jun 19, 2020
068913f
Categorize integration tests
aknuds1 Jun 19, 2020
de0666f
Use grabpl integration-tests
aknuds1 Jun 19, 2020
943977c
Remove unintended change
aknuds1 Jun 22, 2020
aa313d6
Drone: Disable Ubuntu Docker images for PR pipeline
aknuds1 Jun 22, 2020
d9cea25
Regenerate yarn.lock
aknuds1 Jun 22, 2020
9aa3de9
Upgrade grabpl
aknuds1 Jun 22, 2020
9207d81
Restore Yarn cache before installing in grafana-enterprise
aknuds1 Jun 22, 2020
090789d
Use separate pipelines for OSS and enterprise
aknuds1 Jun 23, 2020
6f17f95
Let OSS builds depend on tests
aknuds1 Jun 23, 2020
932d551
Restore Go cache before building back-end
aknuds1 Jun 23, 2020
9e63ccd
Reduce number of variants built for PRs
aknuds1 Jun 23, 2020
c2109ad
Fix building of Docker images
aknuds1 Jun 23, 2020
55a09a7
Merge remote-tracking branch 'grafana/master' into chore/drone-ci
aknuds1 Jun 24, 2020
444fa69
Drone: Simplify logic
aknuds1 Jun 24, 2020
80aa507
Drone: Use Starlark
aknuds1 Jun 24, 2020
567cc2b
Merge branch 'master' into chore/drone-ci
aknuds1 Jun 24, 2020
c757f26
Merge remote-tracking branch 'grafana/master' into chore/drone-ci
aknuds1 Jun 24, 2020
1b5f956
Drone: Fix syntax error
aknuds1 Jun 24, 2020
ef887af
Merge remote-tracking branch 'grafana/master' into chore/drone-ci
aknuds1 Jun 24, 2020
5fab713
Merge remote-tracking branch 'grafana/master' into chore/drone-ci
aknuds1 Jul 2, 2020
494eed2
Convert .drone.star to YAML
aknuds1 Jul 2, 2020
3c5a23a
Upgrade AWS Go SDK
aknuds1 Jul 2, 2020
c13ed85
Drone: Fix Go linting
aknuds1 Jul 2, 2020
bdfb282
Merge remote-tracking branch 'grafana/master' into chore/drone-ci
aknuds1 Jul 9, 2020
5152f65
Undo irrelevant changes
aknuds1 Jul 9, 2020
a017c55
Revert "Undo irrelevant changes"
aknuds1 Jul 9, 2020
f530dc0
Undo irrelevant changes
aknuds1 Jul 9, 2020
c1eb0e3
e2e: Support Circle
aknuds1 Jul 9, 2020
b46d46e
Remove unused script
aknuds1 Jul 9, 2020
379dddf
TypeScript fixes
aknuds1 Jul 9, 2020
83c17d8
TypeScript fixes
aknuds1 Jul 9, 2020
c113363
Remove unused script
aknuds1 Jul 9, 2020
8eddd42
More Drone support
aknuds1 Jul 9, 2020
b2aaf2d
Remove unused script
aknuds1 Jul 9, 2020
9baa156
Fix build on Circle
aknuds1 Jul 9, 2020
8fbbd24
Merge remote-tracking branch 'grafana/master' into chore/drone-ci
aknuds1 Jul 10, 2020
6db6f3c
Remove TODO comment
aknuds1 Jul 10, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
488 changes: 488 additions & 0 deletions .drone.star

Large diffs are not rendered by default.

351 changes: 351 additions & 0 deletions .drone.yml
@@ -0,0 +1,351 @@
---
kind: pipeline
type: docker
name: test-pr-oss

platform:
os: linux
arch: amd64

steps:
- name: install-deps
image: grafana/build-container:1.2.21
commands:
- curl -fLO https://grafana-downloads.storage.googleapis.com/grafana-build-pipeline/v$${GRABPL_VERSION}/grabpl
- chmod +x grabpl
- mkdir -p bin
- mv grabpl bin
- curl -fLO https://github.com/jwilder/dockerize/releases/download/v$${DOCKERIZE_VERSION}/dockerize-linux-amd64-v$${DOCKERIZE_VERSION}.tar.gz
- tar -C bin -xzvf dockerize-linux-amd64-v$${DOCKERIZE_VERSION}.tar.gz
- rm dockerize-linux-amd64-v$${DOCKERIZE_VERSION}.tar.gz
- yarn install --frozen-lockfile --no-progress
- cp -r $(yarn cache dir) yarn-cache
environment:
DOCKERIZE_VERSION: 0.6.1
GRABPL_VERSION: 0.4.15

- name: lint-go
image: grafana/build-container:1.2.21
commands:
- make lint-go
environment:
CGO_ENABLED: 1
depends_on:
- install-deps

- name: codespell
image: grafana/build-container:1.2.21
commands:
- "echo -e \"unknwon\nreferer\nerrorstring\neror\niam\" > words_to_ignore.txt"
- codespell -I words_to_ignore.txt docs/
depends_on:
- install-deps

- name: shellcheck
image: grafana/build-container:1.2.21
commands:
- curl -fLO http://storage.googleapis.com/grafana-downloads/ci-dependencies/shellcheck-v$${VERSION}.linux.x86_64.tar.xz
- echo $$CHKSUM shellcheck-v$${VERSION}.linux.x86_64.tar.xz | sha512sum --check --strict --status
- tar xf shellcheck-v$${VERSION}.linux.x86_64.tar.xz
- mv shellcheck-v$${VERSION}/shellcheck /usr/local/bin/
- rm -rf shellcheck-v$${VERSION}*
- ./bin/grabpl shellcheck
environment:
CHKSUM: beca3d7819a6bdcfbd044576df4fc284053b48f468b2f03428fe66f4ceb2c05d9b5411357fa15003cb0311406c255084cf7283a3b8fce644c340c2f6aa910b9f
VERSION: 0.7.1
depends_on:
- install-deps

- name: build-backend
image: grafana/build-container:1.2.21
commands:
- rm -rf $(go env GOCACHE) && cp -r go-cache $(go env GOCACHE)
- ./bin/grabpl build-backend --github-token "$${GITHUB_TOKEN}" --edition oss --build-id $DRONE_BUILD_NUMBER --variants linux-x64,linux-x64-musl,osx64,win64
environment:
GITHUB_TOKEN:
from_secret: github_token
depends_on:
- install-deps
- lint-go
- test-backend

- name: build-frontend
image: grafana/build-container:1.2.21
commands:
- rm -rf $(yarn cache dir) && cp -r yarn-cache $(yarn cache dir)
- ./bin/grabpl build-frontend --no-install-deps --github-token "$${GITHUB_TOKEN}" --edition oss --build-id $DRONE_BUILD_NUMBER
environment:
GITHUB_TOKEN:
from_secret: github_token
depends_on:
- install-deps
- test-frontend

- name: test-backend
image: grafana/build-container:1.2.21
commands:
- go test -covermode=atomic ./pkg/...
- ./bin/grabpl integration-tests
- cp -r $(go env GOCACHE) go-cache
depends_on:
- install-deps
- lint-go

- name: test-frontend
image: grafana/build-container:1.2.21
commands:
- rm -rf $(yarn cache dir) && cp -r yarn-cache $(yarn cache dir)
- yarn run prettier:check
- yarn run packages:typecheck
- yarn run typecheck
- yarn run test
depends_on:
- install-deps

- name: build-plugins
image: grafana/build-container:1.2.21
commands:
- rm -rf $(yarn cache dir) && cp -r yarn-cache $(yarn cache dir)
- ./bin/grabpl build-plugins --edition oss --no-install-deps
depends_on:
- install-deps
- lint-go

- name: package
image: grafana/build-container:1.2.21
commands:
- . scripts/build/gpg-test-vars.sh && ./bin/grabpl package --github-token "$${GITHUB_TOKEN}" --edition oss --build-id $DRONE_BUILD_NUMBER --variants linux-x64,linux-x64-musl,osx64,win64
environment:
GITHUB_TOKEN:
from_secret: github_token
depends_on:
- build-backend
- build-frontend
- build-plugins
- test-backend
- test-frontend
- codespell
- shellcheck

- name: end-to-end-tests-server
image: grafana/build-container:1.2.21
detach: true
commands:
- ./e2e/start-server
depends_on:
- package

- name: end-to-end-tests
image: grafana/ci-e2e:12.18-1
commands:
- rm -rf $(yarn cache dir) && cp -r yarn-cache $(yarn cache dir)
- ./node_modules/.bin/cypress install
- ./e2e/wait-for-grafana
- ./e2e/run-suite
environment:
HOST: end-to-end-tests-server
depends_on:
- end-to-end-tests-server

- name: publish-storybook
image: grafana/build-container:1.2.21
commands:
- rm -rf $(yarn cache dir) && cp -r yarn-cache $(yarn cache dir)
- yarn storybook:build
depends_on:
- package

- name: build-docs-website
image: grafana/docs-base:latest
commands:
- mkdir -p /hugo/content/docs/grafana
- cp -r docs/sources /hugo/content/docs/grafana/latest
- cd /hugo && make prod

- name: copy-packages-for-docker
image: grafana/build-container:1.2.21
commands:
- cp dist/*.tar.gz packaging/docker/
depends_on:
- package

- name: build-docker-images
image: grafana/drone-grafana-docker:0.2.0
settings:
archs: amd64
dry_run: true
edition: oss
depends_on:
- copy-packages-for-docker

- name: postgres-integration-test
image: grafana/build-container:1.2.21
commands:
- apt-get update
- apt-get install -yq postgresql-client
- ./bin/dockerize -wait tcp://postgres:5432 -timeout 120s
- psql -p 5432 -h postgres -U grafanatest -d grafanatest -f devenv/docker/blocks/postgres_tests/setup.sql
- rm -rf $(go env GOCACHE) && cp -r go-cache $(go env GOCACHE)
- go clean -testcache
- ./bin/grabpl integration-tests --database postgres
environment:
GRAFANA_TEST_DB: postgres
PGPASSWORD: grafanatest
POSTGRES_HOST: postgres
depends_on:
- test-backend
- test-frontend

- name: mysql-integration-test
image: grafana/build-container:1.2.21
commands:
- apt-get update
- apt-get install -yq default-mysql-client
- ./bin/dockerize -wait tcp://mysql:3306 -timeout 120s
- cat devenv/docker/blocks/mysql_tests/setup.sql | mysql -h mysql -P 3306 -u root -prootpass
- rm -rf $(go env GOCACHE) && cp -r go-cache $(go env GOCACHE)
- go clean -testcache
- ./bin/grabpl integration-tests --database mysql
environment:
GRAFANA_TEST_DB: mysql
MYSQL_HOST: mysql
depends_on:
- test-backend
- test-frontend

services:
- name: postgres
image: postgres:12.3-alpine
environment:
POSTGRES_DB: grafanatest
POSTGRES_PASSWORD: grafanatest
POSTGRES_USER: grafanatest

- name: mysql
image: mysql:5.6.48
environment:
MYSQL_DATABASE: grafana_tests
MYSQL_PASSWORD: password
MYSQL_ROOT_PASSWORD: rootpass
MYSQL_USER: grafana

trigger:
event:
- pull_request

---
kind: pipeline
type: docker
name: test-pr-enterprise

platform:
os: linux
arch: amd64

steps:
- name: install-deps
image: grafana/build-container:1.2.21
commands:
- curl -fLO https://grafana-downloads.storage.googleapis.com/grafana-build-pipeline/v$${GRABPL_VERSION}/grabpl
- chmod +x grabpl
- mkdir -p bin
- mv grabpl bin
- curl -fLO https://github.com/jwilder/dockerize/releases/download/v$${DOCKERIZE_VERSION}/dockerize-linux-amd64-v$${DOCKERIZE_VERSION}.tar.gz
- tar -C bin -xzvf dockerize-linux-amd64-v$${DOCKERIZE_VERSION}.tar.gz
- rm dockerize-linux-amd64-v$${DOCKERIZE_VERSION}.tar.gz
- yarn install --frozen-lockfile --no-progress
- cp -r $(yarn cache dir) yarn-cache
environment:
DOCKERIZE_VERSION: 0.6.1
GRABPL_VERSION: 0.4.15
when:
repo:
- grafana/grafana
- aknuds1/grafana

- name: build-backend
image: grafana/build-container:1.2.21
commands:
- ./bin/grabpl build-backend --github-token "$${GITHUB_TOKEN}" --edition enterprise --build-id $DRONE_BUILD_NUMBER --variants linux-x64,linux-x64-musl,osx64,win64
environment:
GITHUB_TOKEN:
from_secret: github_token
when:
repo:
- grafana/grafana
- aknuds1/grafana
depends_on:
- install-deps

- name: build-frontend
image: grafana/build-container:1.2.21
commands:
- rm -rf $(yarn cache dir) && cp -r yarn-cache $(yarn cache dir)
- ./bin/grabpl build-frontend --no-install-deps --github-token "$${GITHUB_TOKEN}" --edition enterprise --build-id $DRONE_BUILD_NUMBER
environment:
GITHUB_TOKEN:
from_secret: github_token
when:
repo:
- grafana/grafana
- aknuds1/grafana
depends_on:
- install-deps

- name: build-plugins
image: grafana/build-container:1.2.21
commands:
- rm -rf $(yarn cache dir) && cp -r yarn-cache $(yarn cache dir)
- ./bin/grabpl build-plugins --edition enterprise --no-install-deps
when:
repo:
- grafana/grafana
- aknuds1/grafana
depends_on:
- install-deps

- name: package
image: grafana/build-container:1.2.21
commands:
- . scripts/build/gpg-test-vars.sh && ./bin/grabpl package --github-token "$${GITHUB_TOKEN}" --edition enterprise --build-id $DRONE_BUILD_NUMBER --variants linux-x64,linux-x64-musl,osx64,win64
environment:
GITHUB_TOKEN:
from_secret: github_token
when:
repo:
- grafana/grafana
- aknuds1/grafana
depends_on:
- build-backend
- build-frontend
- build-plugins

- name: copy-packages-for-docker
image: grafana/build-container:1.2.21
commands:
- cp dist/*.tar.gz packaging/docker/
when:
repo:
- grafana/grafana
- aknuds1/grafana
depends_on:
- package

- name: build-docker-images
image: grafana/drone-grafana-docker:0.2.0
settings:
archs: amd64
dry_run: true
edition: enterprise
when:
repo:
- grafana/grafana
- aknuds1/grafana
depends_on:
- copy-packages-for-docker

trigger:
event:
- pull_request

...
3 changes: 3 additions & 0 deletions .editorconfig
Expand Up @@ -25,3 +25,6 @@ trim_trailing_whitespace = false
[Makefile]
indent_style = tab
indent_size = 2

[*.star]
indent_size = 4
8 changes: 6 additions & 2 deletions e2e/run-suite
@@ -1,13 +1,17 @@
#!/bin/bash
set -xeo pipefail

. e2e/variables

HOST=${HOST:-$DEFAULT_HOST}
PORT=${PORT:-$DEFAULT_PORT}

echo -e "Starting Cypress scenarios"

CMD="start"
PARAMS=""
SLOWMO=0
URL=${BASE_URL:-"http://$DEFAULT_HOST:$DEFAULT_PORT"}
URL=${BASE_URL:-"http://$HOST:$PORT"}
SUITE=${SUITE:-$DEFAULT_SUITE}

if [ "$1" == "debug" ]; then
Expand All @@ -23,6 +27,6 @@ fi

cd packages/grafana-e2e

yarn $CMD --env BASE_URL=$URL,CIRCLE_SHA1=$CIRCLE_SHA1,SLOWMO=$SLOWMO \
yarn $CMD --env BASE_URL=$URL,SLOWMO=$SLOWMO \
--config integrationFolder=../../e2e/$SUITE/specs,screenshotsFolder=../../e2e/$SUITE/screenshots,videosFolder=../../e2e/$SUITE/videos,fileServerFolder=./cypress,viewportWidth=1920,viewportHeight=1080,trashAssetsBeforeRuns=false \
$PARAMS
2 changes: 1 addition & 1 deletion package.json
Expand Up @@ -13,7 +13,7 @@
"e2e:debug": "./e2e/start-and-run-suite debug",
"e2e:dev": "./e2e/start-and-run-suite dev",
"jest": "jest --notify --watch",
"jest-ci": "mkdir -p reports/junit && export JEST_JUNIT_OUTPUT_DIR=reports/junit && jest --ci --reporters=default --reporters=jest-junit --maxWorkers 2",
"jest-ci": "mkdir -p reports/junit && export JEST_JUNIT_OUTPUT_DIR=reports/junit && jest --ci --reporters=default --reporters=jest-junit -w ${TEST_MAX_WORKERS:-100%}",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"jest-ci": "mkdir -p reports/junit && export JEST_JUNIT_OUTPUT_DIR=reports/junit && jest --ci --reporters=default --reporters=jest-junit -w ${TEST_MAX_WORKERS:-100%}",

What does the -100% mean?

Copy link
Contributor Author

@aknuds1 aknuds1 Jul 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dprokop The bash syntax with ${SOMETHING:-<default>} means to use the default value if the environment variable isn't set. So it defaults to "100%" for max workers. Basically, 100% parallelization works on Drone, but not on Circle (runs out of memory).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kk, gotcha. Thanks!

"lint": "eslint public/app e2e/suite1 public/test --ext=.js,.ts,.tsx",
"lint:fix": "yarn lint --fix",
"packages:build": "lerna run clean && lerna run build --ignore @grafana-plugins/input-datasource",
Expand Down