Chore: Enable PR testing in Drone (#26189)

* Add Drone configuration

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Add more steps

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Add more steps

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Build front-end before testing it

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Upgrade grafana/build-container

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Add packaging step

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Trigger on push

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Remove some steps

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Enable steps

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Install Dockerize

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Use node image for test-frontend

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Increase number of test workers

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Make plugin installation depend on frontend tests

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Make integration tests depend on frontend tests

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Use grafana/build-container also for front-end tests

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Upgrade dependencies in order to fix front-end tests

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Depend on es-check

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Dont' depend on tests before building front-end

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Add more steps

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Fix packaging

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Simplify

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Try to build images

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Fix e2e tests

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Remove steps

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Install netcat

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Include golangci-lint with grafana/build-container

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Build storybook and docs website

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Fix e2e tests

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Use build image with root user

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Drop CircleCI dependencies

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Fix e2e tests

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Fix e2e under Drone

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Execute e2e server separately

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Use own plugin for building Docker images

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Use Starlark to configure Drone

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Add enterprise steps to pipeline

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Add more enterprise steps to pipeline

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Maintain Yarn cache

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Build enterprise Docker images

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Build Ubuntu Docker images

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Refactor

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Add Postgres integration test

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Add MySQL integration test

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Fix integration tests

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Parameterize integration test DB connections

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Categorize integration tests

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Use grabpl integration-tests

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Remove unintended change

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Drone: Disable Ubuntu Docker images for PR pipeline

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Regenerate yarn.lock

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Upgrade grabpl

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Restore Yarn cache before installing in grafana-enterprise

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Use separate pipelines for OSS and enterprise

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Let OSS builds depend on tests

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Restore Go cache before building back-end

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Reduce number of variants built for PRs

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Fix building of Docker images

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Drone: Simplify logic

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Drone: Use Starlark

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Drone: Fix syntax error

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Convert .drone.star to YAML

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Upgrade AWS Go SDK

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Drone: Fix Go linting

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Undo irrelevant changes

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Revert "Undo irrelevant changes"

This reverts commit 5152f65.

* Undo irrelevant changes

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* e2e: Support Circle

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Remove unused script

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* TypeScript fixes

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* TypeScript fixes

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Remove unused script

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* More Drone support

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Remove unused script

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Fix build on Circle

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

* Remove TODO comment

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

.drone.yml

@@ -0,0 +1,351 @@
+---
+kind: pipeline
+type: docker
+name: test-pr-oss
+
+platform:
+  os: linux
+  arch: amd64
+
+steps:
+- name: install-deps
+  image: grafana/build-container:1.2.21
+  commands:
+  - curl -fLO https://grafana-downloads.storage.googleapis.com/grafana-build-pipeline/v$${GRABPL_VERSION}/grabpl
+  - chmod +x grabpl
+  - mkdir -p bin
+  - mv grabpl bin
+  - curl -fLO https://github.com/jwilder/dockerize/releases/download/v$${DOCKERIZE_VERSION}/dockerize-linux-amd64-v$${DOCKERIZE_VERSION}.tar.gz
+  - tar -C bin -xzvf dockerize-linux-amd64-v$${DOCKERIZE_VERSION}.tar.gz
+  - rm dockerize-linux-amd64-v$${DOCKERIZE_VERSION}.tar.gz
+  - yarn install --frozen-lockfile --no-progress
+  - cp -r $(yarn cache dir) yarn-cache
+  environment:
+    DOCKERIZE_VERSION: 0.6.1
+    GRABPL_VERSION: 0.4.15
+
+- name: lint-go
+  image: grafana/build-container:1.2.21
+  commands:
+  - make lint-go
+  environment:
+    CGO_ENABLED: 1
+  depends_on:
+  - install-deps
+
+- name: codespell
+  image: grafana/build-container:1.2.21
+  commands:
+  - "echo -e \"unknwon\nreferer\nerrorstring\neror\niam\" > words_to_ignore.txt"
+  - codespell -I words_to_ignore.txt docs/
+  depends_on:
+  - install-deps
+
+- name: shellcheck
+  image: grafana/build-container:1.2.21
+  commands:
+  - curl -fLO http://storage.googleapis.com/grafana-downloads/ci-dependencies/shellcheck-v$${VERSION}.linux.x86_64.tar.xz
+  - echo $$CHKSUM shellcheck-v$${VERSION}.linux.x86_64.tar.xz | sha512sum --check --strict --status
+  - tar xf shellcheck-v$${VERSION}.linux.x86_64.tar.xz
+  - mv shellcheck-v$${VERSION}/shellcheck /usr/local/bin/
+  - rm -rf shellcheck-v$${VERSION}*
+  - ./bin/grabpl shellcheck
+  environment:
+    CHKSUM: beca3d7819a6bdcfbd044576df4fc284053b48f468b2f03428fe66f4ceb2c05d9b5411357fa15003cb0311406c255084cf7283a3b8fce644c340c2f6aa910b9f
+    VERSION: 0.7.1
+  depends_on:
+  - install-deps
+
+- name: build-backend
+  image: grafana/build-container:1.2.21
+  commands:
+  - rm -rf $(go env GOCACHE) && cp -r go-cache $(go env GOCACHE)
+  - ./bin/grabpl build-backend --github-token "$${GITHUB_TOKEN}" --edition oss --build-id $DRONE_BUILD_NUMBER --variants linux-x64,linux-x64-musl,osx64,win64
+  environment:
+    GITHUB_TOKEN:
+      from_secret: github_token
+  depends_on:
+  - install-deps
+  - lint-go
+  - test-backend
+
+- name: build-frontend
+  image: grafana/build-container:1.2.21
+  commands:
+  - rm -rf $(yarn cache dir) && cp -r yarn-cache $(yarn cache dir)
+  - ./bin/grabpl build-frontend --no-install-deps --github-token "$${GITHUB_TOKEN}" --edition oss --build-id $DRONE_BUILD_NUMBER
+  environment:
+    GITHUB_TOKEN:
+      from_secret: github_token
+  depends_on:
+  - install-deps
+  - test-frontend
+
+- name: test-backend
+  image: grafana/build-container:1.2.21
+  commands:
+  - go test -covermode=atomic ./pkg/...
+  - ./bin/grabpl integration-tests
+  - cp -r $(go env GOCACHE) go-cache
+  depends_on:
+  - install-deps
+  - lint-go
+
+- name: test-frontend
+  image: grafana/build-container:1.2.21
+  commands:
+  - rm -rf $(yarn cache dir) && cp -r yarn-cache $(yarn cache dir)
+  - yarn run prettier:check
+  - yarn run packages:typecheck
+  - yarn run typecheck
+  - yarn run test
+  depends_on:
+  - install-deps
+
+- name: build-plugins
+  image: grafana/build-container:1.2.21
+  commands:
+  - rm -rf $(yarn cache dir) && cp -r yarn-cache $(yarn cache dir)
+  - ./bin/grabpl build-plugins --edition oss --no-install-deps
+  depends_on:
+  - install-deps
+  - lint-go
+
+- name: package
+  image: grafana/build-container:1.2.21
+  commands:
+  - . scripts/build/gpg-test-vars.sh && ./bin/grabpl package --github-token "$${GITHUB_TOKEN}" --edition oss --build-id $DRONE_BUILD_NUMBER --variants linux-x64,linux-x64-musl,osx64,win64
+  environment:
+    GITHUB_TOKEN:
+      from_secret: github_token
+  depends_on:
+  - build-backend
+  - build-frontend
+  - build-plugins
+  - test-backend
+  - test-frontend
+  - codespell
+  - shellcheck
+
+- name: end-to-end-tests-server
+  image: grafana/build-container:1.2.21
+  detach: true
+  commands:
+  - ./e2e/start-server
+  depends_on:
+  - package
+
+- name: end-to-end-tests
+  image: grafana/ci-e2e:12.18-1
+  commands:
+  - rm -rf $(yarn cache dir) && cp -r yarn-cache $(yarn cache dir)
+  - ./node_modules/.bin/cypress install
+  - ./e2e/wait-for-grafana
+  - ./e2e/run-suite
+  environment:
+    HOST: end-to-end-tests-server
+  depends_on:
+  - end-to-end-tests-server
+
+- name: publish-storybook
+  image: grafana/build-container:1.2.21
+  commands:
+  - rm -rf $(yarn cache dir) && cp -r yarn-cache $(yarn cache dir)
+  - yarn storybook:build
+  depends_on:
+  - package
+
+- name: build-docs-website
+  image: grafana/docs-base:latest
+  commands:
+  - mkdir -p /hugo/content/docs/grafana
+  - cp -r docs/sources /hugo/content/docs/grafana/latest
+  - cd /hugo && make prod
+
+- name: copy-packages-for-docker
+  image: grafana/build-container:1.2.21
+  commands:
+  - cp dist/*.tar.gz packaging/docker/
+  depends_on:
+  - package
+
+- name: build-docker-images
+  image: grafana/drone-grafana-docker:0.2.0
+  settings:
+    archs: amd64
+    dry_run: true
+    edition: oss
+  depends_on:
+  - copy-packages-for-docker
+
+- name: postgres-integration-test
+  image: grafana/build-container:1.2.21
+  commands:
+  - apt-get update
+  - apt-get install -yq postgresql-client
+  - ./bin/dockerize -wait tcp://postgres:5432 -timeout 120s
+  - psql -p 5432 -h postgres -U grafanatest -d grafanatest -f devenv/docker/blocks/postgres_tests/setup.sql
+  - rm -rf $(go env GOCACHE) && cp -r go-cache $(go env GOCACHE)
+  - go clean -testcache
+  - ./bin/grabpl integration-tests --database postgres
+  environment:
+    GRAFANA_TEST_DB: postgres
+    PGPASSWORD: grafanatest
+    POSTGRES_HOST: postgres
+  depends_on:
+  - test-backend
+  - test-frontend
+
+- name: mysql-integration-test
+  image: grafana/build-container:1.2.21
+  commands:
+  - apt-get update
+  - apt-get install -yq default-mysql-client
+  - ./bin/dockerize -wait tcp://mysql:3306 -timeout 120s
+  - cat devenv/docker/blocks/mysql_tests/setup.sql | mysql -h mysql -P 3306 -u root -prootpass
+  - rm -rf $(go env GOCACHE) && cp -r go-cache $(go env GOCACHE)
+  - go clean -testcache
+  - ./bin/grabpl integration-tests --database mysql
+  environment:
+    GRAFANA_TEST_DB: mysql
+    MYSQL_HOST: mysql
+  depends_on:
+  - test-backend
+  - test-frontend
+
+services:
+- name: postgres
+  image: postgres:12.3-alpine
+  environment:
+    POSTGRES_DB: grafanatest
+    POSTGRES_PASSWORD: grafanatest
+    POSTGRES_USER: grafanatest
+
+- name: mysql
+  image: mysql:5.6.48
+  environment:
+    MYSQL_DATABASE: grafana_tests
+    MYSQL_PASSWORD: password
+    MYSQL_ROOT_PASSWORD: rootpass
+    MYSQL_USER: grafana
+
+trigger:
+  event:
+  - pull_request
+
+---
+kind: pipeline
+type: docker
+name: test-pr-enterprise
+
+platform:
+  os: linux
+  arch: amd64
+
+steps:
+- name: install-deps
+  image: grafana/build-container:1.2.21
+  commands:
+  - curl -fLO https://grafana-downloads.storage.googleapis.com/grafana-build-pipeline/v$${GRABPL_VERSION}/grabpl
+  - chmod +x grabpl
+  - mkdir -p bin
+  - mv grabpl bin
+  - curl -fLO https://github.com/jwilder/dockerize/releases/download/v$${DOCKERIZE_VERSION}/dockerize-linux-amd64-v$${DOCKERIZE_VERSION}.tar.gz
+  - tar -C bin -xzvf dockerize-linux-amd64-v$${DOCKERIZE_VERSION}.tar.gz
+  - rm dockerize-linux-amd64-v$${DOCKERIZE_VERSION}.tar.gz
+  - yarn install --frozen-lockfile --no-progress
+  - cp -r $(yarn cache dir) yarn-cache
+  environment:
+    DOCKERIZE_VERSION: 0.6.1
+    GRABPL_VERSION: 0.4.15
+  when:
+    repo:
+    - grafana/grafana
+    - aknuds1/grafana
+
+- name: build-backend
+  image: grafana/build-container:1.2.21
+  commands:
+  - ./bin/grabpl build-backend --github-token "$${GITHUB_TOKEN}" --edition enterprise --build-id $DRONE_BUILD_NUMBER --variants linux-x64,linux-x64-musl,osx64,win64
+  environment:
+    GITHUB_TOKEN:
+      from_secret: github_token
+  when:
+    repo:
+    - grafana/grafana
+    - aknuds1/grafana
+  depends_on:
+  - install-deps
+
+- name: build-frontend
+  image: grafana/build-container:1.2.21
+  commands:
+  - rm -rf $(yarn cache dir) && cp -r yarn-cache $(yarn cache dir)
+  - ./bin/grabpl build-frontend --no-install-deps --github-token "$${GITHUB_TOKEN}" --edition enterprise --build-id $DRONE_BUILD_NUMBER
+  environment:
+    GITHUB_TOKEN:
+      from_secret: github_token
+  when:
+    repo:
+    - grafana/grafana
+    - aknuds1/grafana
+  depends_on:
+  - install-deps
+
+- name: build-plugins
+  image: grafana/build-container:1.2.21
+  commands:
+  - rm -rf $(yarn cache dir) && cp -r yarn-cache $(yarn cache dir)
+  - ./bin/grabpl build-plugins --edition enterprise --no-install-deps
+  when:
+    repo:
+    - grafana/grafana
+    - aknuds1/grafana
+  depends_on:
+  - install-deps
+
+- name: package
+  image: grafana/build-container:1.2.21
+  commands:
+  - . scripts/build/gpg-test-vars.sh && ./bin/grabpl package --github-token "$${GITHUB_TOKEN}" --edition enterprise --build-id $DRONE_BUILD_NUMBER --variants linux-x64,linux-x64-musl,osx64,win64
+  environment:
+    GITHUB_TOKEN:
+      from_secret: github_token
+  when:
+    repo:
+    - grafana/grafana
+    - aknuds1/grafana
+  depends_on:
+  - build-backend
+  - build-frontend
+  - build-plugins
+
+- name: copy-packages-for-docker
+  image: grafana/build-container:1.2.21
+  commands:
+  - cp dist/*.tar.gz packaging/docker/
+  when:
+    repo:
+    - grafana/grafana
+    - aknuds1/grafana
+  depends_on:
+  - package
+
+- name: build-docker-images
+  image: grafana/drone-grafana-docker:0.2.0
+  settings:
+    archs: amd64
+    dry_run: true
+    edition: enterprise
+  when:
+    repo:
+    - grafana/grafana
+    - aknuds1/grafana
+  depends_on:
+  - copy-packages-for-docker
+
+trigger:
+  event:
+  - pull_request
+
+...

.editorconfig

@@ -25,3 +25,6 @@ trim_trailing_whitespace = false
 [Makefile]
 indent_style = tab
 indent_size = 2
+
+[*.star]
+indent_size = 4

e2e/run-suite

@@ -1,13 +1,17 @@
 #!/bin/bash
+set -xeo pipefail
 
 . e2e/variables
 
+HOST=${HOST:-$DEFAULT_HOST}
+PORT=${PORT:-$DEFAULT_PORT}
+
 echo -e "Starting Cypress scenarios"
 
 CMD="start"
 PARAMS=""
 SLOWMO=0
-URL=${BASE_URL:-"http://$DEFAULT_HOST:$DEFAULT_PORT"}
+URL=${BASE_URL:-"http://$HOST:$PORT"}
 SUITE=${SUITE:-$DEFAULT_SUITE}
 
 if [ "$1" == "debug" ]; then
@@ -23,6 +27,6 @@ fi
 
 cd packages/grafana-e2e
 
-yarn $CMD --env BASE_URL=$URL,CIRCLE_SHA1=$CIRCLE_SHA1,SLOWMO=$SLOWMO \
+yarn $CMD --env BASE_URL=$URL,SLOWMO=$SLOWMO \
   --config integrationFolder=../../e2e/$SUITE/specs,screenshotsFolder=../../e2e/$SUITE/screenshots,videosFolder=../../e2e/$SUITE/videos,fileServerFolder=./cypress,viewportWidth=1920,viewportHeight=1080,trashAssetsBeforeRuns=false \
   $PARAMS

package.json

@@ -13,7 +13,7 @@
     "e2e:debug": "./e2e/start-and-run-suite debug",
     "e2e:dev": "./e2e/start-and-run-suite dev",
     "jest": "jest --notify --watch",
-    "jest-ci": "mkdir -p reports/junit && export JEST_JUNIT_OUTPUT_DIR=reports/junit && jest --ci --reporters=default --reporters=jest-junit --maxWorkers 2",
+    "jest-ci": "mkdir -p reports/junit && export JEST_JUNIT_OUTPUT_DIR=reports/junit && jest --ci --reporters=default --reporters=jest-junit -w ${TEST_MAX_WORKERS:-100%}",
     "lint": "eslint public/app e2e/suite1 public/test --ext=.js,.ts,.tsx",
     "lint:fix": "yarn lint --fix",
     "packages:build": "lerna run clean && lerna run build --ignore @grafana-plugins/input-datasource",