Enhance Kibana CI pull-request tests to include new Elastic Agent & Kibana end-to-end test

[EricDavisX]

Describe the enhancement:
Enhance Kibana CI to call a new e2e Agent test when Kibana files change that relate to Ingest Manager

• the Observability Automation Team was kind enough to help us set up a small initial test for Elastic Agent and Kibana Ingest Manager! The next step is to have it run when specific file directories change.

Files from this repo location should trigger the noted CI:
https://github.com/elastic/kibana/tree/master/x-pack/plugins/ingest_manager

@ph @ruflin are there any other locations we want to trigger this e2e test?

The e2e test is here:
https://github.com/elastic/e2e-testing/tree/master/e2e/_suites/ingest-manager

It can be called as, from the elastic/e2e-testing repo e2e/_suites/ingest-manager
$ godog -t stand_alone_mode

• you'll also need dependencies of course. specifics are being Doc'ed for that immediately so we can use them here.

Describe a specific use case for the enhancement or feature:
Lets say a developer somehow puts a bug into the code, during PR CI this could catch it so it isn't merged, which saves the company time overall.

@mdelapenya thanks for helping guide this

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[EricDavisX]

Manu says we probably need to ping the Kibana powers that be to get help pushing this thru. @LeeDr who is a person to talk to... @spalger maybe?

[spalger]

You should be able to copy the implementation used to conditionally run the tests that the security solution team uses to run their cypress tests

kibana/Jenkinsfile

Lines 45 to 49 in 4d82a76

	'xpack-securitySolutionCypress': { processNumber ->
	whenChanged(['x-pack/plugins/security_solution/', 'x-pack/test/security_solution_cypress/']) {
	kibanaPipeline.functionalTestProcess('xpack-securitySolutionCypress', './test/scripts/jenkins_security_solution_cypress.sh')(processNumber)
	}
	},

As for checking out the e2e-testing repo and running those tests, you should be able to do whatever you need if you add a script to the test/scripts directory. Checkout the script we use for siem-cypress for the basics but you'll probably need substantially more logic to get go all setup:

kibana/test/scripts/jenkins_security_solution_cypress.sh

Line 1 in 4d82a76

#!/usr/bin/env bash

Can you get JUnit output from the end-to-end tests? I hope so or debugging is going to be quite a chore. Write/copy the junit output to target/junit at the root of the repo to feed them into jest as necessary.

Let us know how that goes!

cc @brianseeders

[ruflin]

@EricDavisX I wonder if we should also run these when the package registry changes?

[EricDavisX]

Ruflin, hi - indeed, we have a ticket open for Integrations repo, here: elastic/integrations#102

• Manu says its within their purview so he can easily help get it set up.

Thanks Spencer for the pointers. I can look at this... sometime. And maybe get help from some Elastic Security testers talent pool, especially when we can show them it will help test Elastic Endpoint end to end (lets build a prototype after we get integration done and show them!). @MindyRS @charlie-pichette just fyi, heads up.

[ph]

@EricDavisX I would say we should trigger a CI also when with changes for https://github.com/elastic/kibana/tree/master/x-pack/plugins/security_solution.

as @ruflin commented, We should also trigger them for the package registry changes too.

[EricDavisX]

the Registry issue is here, FYI: elastic/package-registry#630

• I still plan to integrate the Elastic-Agent / Beats CI first, then Kibana as noted here, then the new Registry ticket. All of those are waiting on a robust enough set of tests to put the test creation on hold so they can be integrated in various repo CI as noted.

[EricDavisX]

We merged in some Agent & Endpoint tests recently, I think I'll change my mind and we can target this sooner... the Beats PR CI work hopefully will be done soon so we can model / borrow code as appropriate.

[EricDavisX]

The e2e test is running in Beats CI when the /package call is run now and tests are green for 7.10 cycle, the time is ripe to pull this in to Kibana execution.

[EricDavisX]

we have updated the e2e-testing repo, the call to execute it is more advanced now.

and I'm wondering if Kibana CI includes the building of a docker image (from the pull request under test) which the test will consume? if not, that is another step.

We think the e2e- test here is valuable, and even still valuable is a simplified version that just runs Kibana + ES to validate the pair are viable before building artifacts, like #89482 which caused some downtime and noise for cloud, Obs, Fleet, Security teams.

[mdelapenya]

We added a pipeline in the Beats Ci that is listening for a webhook on Kibana repo. This pipeline will:

• be triggered by commenting in a kibana PR with /run-fleet-e2e-tests
• build the kibana Docker image for that PR, pushing it under observability-ci namespace
• trigger the e2e tests using that new Docker image for the PR

At this moment, there is no response after tests run (something like commenting on the PR, contributing a check, or a slack notification), but we'd like to open a discussion for it.

Said that, I think we can close this one.

[mdelapenya]

Closing as my previous comment.

@EricDavisX please reopen it if you have any concern about the above implementation

Thanks!

[EricDavisX]

Thanks so much @mdelapenya - sorry for the lack of quicker reply, I will ask... should we reopen this? The bulk of the work is done, but the final bits that bring the value to the Engineer writing the Kibana code won't be realized if it isn't:

• run automatically for them
• run for them based on a specific well documented desired subset of Kibana files
• reporting back the results in the PR as a pass / fail, with a link to the job / test results
  I think we can leave the tests non-blocking, but that is debatable.

@jen-huang your team commits Fleet Kibana code the most, do you have a preference on reopening this or a new discussion ticket to review the requirements? For now I'm reopening it, we can easily close it back (again!) if we don't want to continue tracking the need here.

[jen-huang]

@EricDavisX I think it's a good idea to run these e2e tests automatically but have them as non-blocking. So that a developer can open a PR, have it pass the main CI suite, and be able to merge it. The notification whether the e2e tests passed or not can happen after merging. I'm thinking this way because we often have much smaller PRs that won't affect what's covered by the e2e tests, so I don't want to have those PRs be blocked by that.

[EricDavisX]

Thanks. I agree on leaving them non-blocking. I like having them run automatically, as we'll want to know if we break something. :)