[buildkite] Add new Buildkite pipelines to test packages with Serverless #8072

mrodm · 2023-10-03T16:12:30Z

Proposed commit message

Add new pipelines to test packages in serverless.
Example of output:

https://buildkite.com/elastic/integrations/builds/5531

This PR adds a schedule-daily and a integrations-serverless pipelines to test packages with Serverless projects. Tests performed in this script/pipeline:

Perform linting and formatting (elastic-package check -v)
Try to install the package (elastic-package install)
Run all tests except system tests
- IMPORTANT Pipeline tests do not compare the documents with the sample ones.

This pipeline is different from the one running in Pull Requests:

Benchmarks are not triggered/run for now.
Coverage reports have not been migrated.
All packages are test in the same step using the same Serverless project (Observability or Security). Just one Serverless project is created/started in each step (to test all packages).
- In Jenkins, using docker-compose locally, there is an step for each package running its own Elastic stack.
Artifacts are also saved in an internal bucket as in Jenkins
This pipeline does not run system tests
- They are so time consuming, and to avoid creating more than 200 serverless projects everything is run in the same step.
- Tests performed:
  - elastic-package check
  - elastic-package install
  - elastic-package test static
  - elastic-package test asset
  - elastic-package test pipeline

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

elastic-package v0.91.0 updated in integrations Bump github.com/elastic/elastic-package from 0.90.0 to 0.91.0 #8272
allow to skip compare results in pipeline tests Skip by default compareResults in pipeline tests for serverless elastic-package#1521, should it be released before merging?
Remove serverless step from PR pipeline

How to test this PR locally

Related issues

Closes Automate tests in CI for serverless packages elastic-package#1451

elasticmachine · 2023-10-03T17:11:33Z

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-10-26T17:53:07.354+0000
Duration: 112 min 59 sec

Test stats 🧪

Test	Results
Failed	2
Passed	5055
Skipped	9
Total	5066

Test errors

Expand to view the tests failures

`Check integrations / entityanalytics_entra_id / entityanalytics_entra_id: check / system test: (elastic-agent logs) – entityanalytics_entra_id.entity`

Expand to view the error details

 null

Expand to view the stacktrace

 test case failed: one or more errors found while examining elastic-agent.log: [0] found error "Component state changed entity-analytics-default (HEALTHY->DEGRADED): Degraded: pid '280' missed 1 check-in"

`Check integrations / sql_input / sql_input: check / system test: mysql – sql_input.`

Expand to view the error details

 null

Expand to view the stacktrace

 one or more errors found in documents stored in metrics-sql.sql-ep data stream: [0] found error.message in event: could not open connection: testing connection: dial tcp 172.18.0.6:3306: connect: connection refused

Steps errors

Expand to view the steps failures

`Test integration: dga`

Took 0 min 6 sec . View more details here
Description: eval "$(../../build/elastic-package stack shellinit)" ../../build/elastic-package test -v --report-format xUnit --report-output file --test-coverage

`Test integration: entityanalytics_entra_id`

Took 1 min 22 sec . View more details here
Description: eval "$(../../build/elastic-package stack shellinit)" ../../build/elastic-package test -v --report-format xUnit --report-output file --test-coverage

`Test integration: sql_input`

Took 2 min 22 sec . View more details here
Description: eval "$(../../build/elastic-package stack shellinit)" ../../build/elastic-package test -v --report-format xUnit --report-output file --test-coverage

`Google Storage Download`

Took 0 min 0 sec . View more details here

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-10-03T18:26:28Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`470/470`)	💚
Files	96.163% (`802/834`)	👍 1.426
Classes	96.163% (`802/834`)	👍 1.426
Methods	92.11% (`7798/8466`)	👍 7.261
Lines	88.593% (`183950/207635`)	👍 2.285
Conditionals	100.0% (`0/0`)	💚

mrodm · 2023-10-24T15:21:43Z

.buildkite/pull-requests.json

+      "pipelineSlug": "integrations",
+      "allow_org_users": true,
+      "allowed_repo_permissions": ["admin", "write"],
+      "allowed_list": ["dependabot[bot]", "mergify[bot]"],


Allow dependabot PRs to run buildkite pull requests.

.buildkite/pipeline.schedule-daily.yml

.buildkite/scripts/publish_packages.sh

sharbuz · 2023-10-25T12:25:20Z

.buildkite/scripts/publish_packages.sh

+
+echo "Checking gsutil command..."
+if ! command -v gsutil &> /dev/null ; then
+    echo "⚠️  gsutil is not installed"


Probably we can add this message to the buildkite annotation, @mrodm WDYT?

I prefer just to keep it in the output. I think there is no need to add more information in the annotation for this.

sharbuz · 2023-10-25T12:31:58Z

.buildkite/scripts/test_integrations_with_serverless.sh

+fi
+
+if [ $any_package_failing -eq 1 ] ; then
+    echo "These packages have failed:"


Probably we can add this message to the buildkite annotation, @mrodm WDYT?

Failed packages are already in one annotation. This is to be also in the log output.

.buildkite/scripts/common.sh

sharbuz · 2023-10-25T12:53:11Z

.buildkite/scripts/common.sh

+}
+
+is_pr() {
+    if [ "${BUILDKITE_PULL_REQUEST}" == "false" ]; then


Suggested change

if [ "${BUILDKITE_PULL_REQUEST}" == "false" ]; then

if [[ "${BUILDKITE_PULL_REQUEST}" == "false" && "${BUILDKITE_TAG}" == "" ]]; then

This repository is configured to not run builds when pushing tags (catalog.yml) build_tags: false

But to be sure I'll add this (with OR condition):

if it is not a PR

or, it is a tag

if [[ "${BUILDKITE_PULL_REQUEST}" == "false" || "${BUILDKITE_TAG}" == "" ]]; then

This reverts commit 3030df2.

jsoriano

👍

.buildkite/scripts/common.sh

sharbuz · 2023-10-26T16:00:21Z

catalog-info.yaml

+      # env:
+      #   ELASTIC_SLACK_NOTIFICATIONS_ENABLED: 'true'
+#     #     SLACK_NOTIFICATIONS_CHANNEL: '#beats-build' as the current jenkins schedule-daily?
+      #   SLACK_NOTIFICATIONS_CHANNEL: '#ingest-test-notifications'   #TODO: will be changed before GO-PROD
+      #   SLACK_NOTIFICATIONS_ALL_BRANCHES: 'true'
+      #   SLACK_NOTIFICATIONS_ON_SUCCESS: 'true'


I think we can enable it and use out test channel for tests: #ingest-test-notifications
WDYT?

Ok! I'll uncomment those lines

Co-authored-by: sharbuz <87968844+sharbuz@users.noreply.github.com>

…ests)"" This reverts commit 94fa971.

sharbuz

LGTM!
Thanks!

mrodm · 2023-10-27T08:22:28Z

Merging since failures in the Jenkins CI job are unrelated to this change.

Add an empty schedule-daily job

ef44515

mrodm self-assigned this Oct 3, 2023

mrodm force-pushed the add_schedule_daily branch from 7733215 to 43d279d Compare October 3, 2023 16:56

Add first steps

554d5a0

mrodm force-pushed the add_schedule_daily branch from 43d279d to 554d5a0 Compare October 3, 2023 17:03

Avoid running all packages in Jenkins

dcabe5b

Fix mage and gvm installation

ce9e0f1

mrodm force-pushed the add_schedule_daily branch from 3469da8 to ce9e0f1 Compare October 3, 2023 17:15

test

e9af47e

mrodm force-pushed the add_schedule_daily branch from 9f35c13 to e9af47e Compare October 3, 2023 17:27

mrodm added 18 commits October 4, 2023 16:28

Add check-git-diff to check sources

884bed7

Add check command

a1882c1

Show links table info

6dd38bd

test

81cd4f4

Test with relative path from root repository directory

24f2596

Set links file path in pre-command hook

5234eaa

Set links file full path just if exists var

7b36121

Install yq and test spec version

17ff5d1

add missing double quotes

22e414c

Use variable for elastic-package binary path

dea5662

Add verbose to stack commands

87f110e

Added a draft version of publish package storage step

14d5ed8

Change step key

130e672

Add dependencies

93a4104

Change agent for publish packages

a3a07ab

Add debug

5b54f1c

Add go in publish_packages

4dee0a9

Check python

d11957b

mrodm added 7 commits October 24, 2023 10:43

Add context for info annotation

c0fbc57

Move to a function code run in each iteration

aae1db9

Rename

1737413

Add debug

41af02c

Change continue for return - not in a loop anymore

8b3ddf4

Update elastic-package

e06e6dc

Add environment variable to avoid comapre results in pipeline tests

3b751a8

mrodm commented Oct 24, 2023

View reviewed changes

mrodm added 3 commits October 24, 2023 17:56

Merge remote upstream/main into add_schedule_daily

2c1a09b

Remove elastic-package replacement dep

74bd61c

Add TODOs and revisted scripts and hooks for new serverless pipeline

8162b7d

mrodm requested a review from jsoriano October 24, 2023 17:28

Remove serverless steps from main pipeline (Pull Requests)

3030df2

sharbuz reviewed Oct 25, 2023

View reviewed changes

Changes from code review

bf44534

mrodm requested a review from sharbuz October 25, 2023 14:06

mrodm added 2 commits October 25, 2023 16:11

Revert "Remove serverless steps from main pipeline (Pull Requests)"

94fa971

This reverts commit 3030df2.

Fix target docker-compose path

22fc1ad

jsoriano approved these changes Oct 25, 2023

View reviewed changes

mrodm changed the title ~~[buildkite] Add new Buildkite pipelines to test packages with Serverless WIP~~ [buildkite] Add new Buildkite pipelines to test packages with Serverless Oct 26, 2023

sharbuz reviewed Oct 26, 2023

View reviewed changes

mrodm and others added 4 commits October 26, 2023 18:11

Apply suggestions from code review

5f95599

Co-authored-by: sharbuz <87968844+sharbuz@users.noreply.github.com>

Enable slack notifications to ingest-test-notifications

7e0172a

Revert "Revert "Remove serverless steps from main pipeline (Pull Requ…

f7b74ae

…ests)"" This reverts commit 94fa971.

Change permissions of publish script

a9c3fa0

sharbuz approved these changes Oct 26, 2023

View reviewed changes

mrodm merged commit ab11988 into elastic:main Oct 27, 2023

mrodm deleted the add_schedule_daily branch October 27, 2023 08:31

mrodm mentioned this pull request Dec 4, 2023

[Buildkite] Update condition to check whether or not a build is a PR #8640

Merged

	if [ "${BUILDKITE_PULL_REQUEST}" == "false" ]; then
	if [[ "${BUILDKITE_PULL_REQUEST}" == "false" && "${BUILDKITE_TAG}" == "" ]]; then

[buildkite] Add new Buildkite pipelines to test packages with Serverless #8072

[buildkite] Add new Buildkite pipelines to test packages with Serverless #8072

Uh oh!

Conversation

mrodm commented Oct 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Uh oh!

elasticmachine commented Oct 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Tests Failed

Build stats

Test stats 🧪

Test errors

Check integrations / entityanalytics_entra_id / entityanalytics_entra_id: check / system test: (elastic-agent logs) – entityanalytics_entra_id.entity

Check integrations / sql_input / sql_input: check / system test: mysql – sql_input.

Steps errors

Test integration: dga

Test integration: entityanalytics_entra_id

Test integration: sql_input

Google Storage Download

🤖 GitHub comments

Uh oh!

elasticmachine commented Oct 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sharbuz left a comment

Choose a reason for hiding this comment

Uh oh!

mrodm commented Oct 27, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mrodm commented Oct 3, 2023 •

edited

Loading

elasticmachine commented Oct 3, 2023 •

edited

Loading

`Check integrations / entityanalytics_entra_id / entityanalytics_entra_id: check / system test: (elastic-agent logs) – entityanalytics_entra_id.entity`

`Check integrations / sql_input / sql_input: check / system test: mysql – sql_input.`

`Test integration: dga`

`Test integration: entityanalytics_entra_id`

`Test integration: sql_input`

`Google Storage Download`

elasticmachine commented Oct 3, 2023 •

edited

Loading