[tests] split up serverless and resource leaks integration tests #8396

pkoutsovasilis · 2025-06-09T07:20:37Z

What does this PR do?

This PR refactors the Elastic Agent integration test structure by introducing dedicated sub-packages for specific test categories. The changes include:

New serverless and leak test packages under testing/integration/, each with their own TestMain.
Moved long-running resource leak tests to the leak package and removed the dependency on the TEST_LONG_RUNNING env var.
Moved TestLogIngestionFleetManaged into a reusable function and invoked it from both integration and serverless packages.
Normalized test entrypoints across Windows (PowerShell) and Unix (Bash) runners.
Made PreinstalledPackages public and importable by other test packages.
Renamed groups_test.go to groups.go to support shared usage.
Enhanced mage integration:check to validate all test packages (integration/, integration/serverless, integration/leak).
Updated Buildkite test scripts and pipeline definitions to accommodate the split.

Why is it important?

The testing/integration package had grown into a monolithic and hard-to-maintain suite containing various unrelated test types. This structural refactor:

Improves clarity and maintainability by grouping tests by setup type (e.g., serverless, extended runtime).
Reduces reliance on complex filtering logic using env vars or CLI args.
Enables tailored TestMain implementations per package for better control of setup/teardown behavior.
Builds a solid baseline for better alignment of Buildkite pipelines and runtime configurations.

Checklist

I have read and understood the pull request guidelines of this project.
My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in ./changelog/fragments using the changelog tool
I have added an integration test or an E2E test

Disruptive User Impact

None expected for end-users. CI and internal contributors may need to adjust test invocation to use the correct package path or mage targets (e.g., mage integration:testServerless, integration:testForResourceLeaks).

How to test this PR locally

# Run standard integration tests
mage integration:test

# Run serverless smoke tests
mage integration:testServerless

# Run extended leak tests
mage integration:testForResourceLeaks

Related issues

… invocation flow

pkoutsovasilis · 2025-06-09T07:21:55Z

@pchila @ycombinator please have a look and let's sync 🙂

elasticmachine · 2025-06-09T09:34:10Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

pchila

LGTM
Thank you for splitting out runtime leak and serverless tests into dedicated packages.
The work on the serverless test could be used to simplify #8263 change to a more manageable size.

v1v

LGTM, just added a very tiny nits

.buildkite/scripts/steps/integration_tests_tf.ps1

.buildkite/scripts/buildkite-integration-tests.sh

.buildkite/scripts/buildkite-integration-tests.ps1

…gration-tests.ps1

…ourceLeaks in test-framework-dev-guide.md

v1v

LGTM, I reviewed the .buildkite folder, again my nits are super-nit.

.buildkite/scripts/steps/integration_tests_tf.ps1

.buildkite/scripts/buildkite-integration-tests.ps1

elastic-sonarqube · 2025-06-11T15:55:15Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2025-06-11T16:23:41Z

💚 Build Succeeded

Buildkite Build
Commit: f088a5f

History

💔 Build #22202 failed ee6bd63
💔 Build #22198 failed fa9657c
💛 Build #22172 was flaky 907167b
💛 Build #22071 was flaky 2e7352b
💔 Build #22064 failed 7f12158

cc @pkoutsovasilis

v1v

LGTM, I reviewed the recent changes in the .buldkite folder after reviewing them earlier too :)
Thank you for your collaboration on the various review comments 🙇

github-actions · 2025-06-11T17:44:03Z

@Mergifyio backport 8.17 8.18 8.19 9.0

mergify · 2025-06-11T17:44:11Z

backport 8.17 8.18 8.19 9.0

✅ Backports have been created

#8455 [8.17] (backport #8396) [tests] split up serverless and resource leaks integration tests has been created for branch 8.17 but encountered conflicts
#8456 [8.18] (backport #8396) [tests] split up serverless and resource leaks integration tests has been created for branch 8.18 but encountered conflicts
#8457 [8.19] (backport #8396) [tests] split up serverless and resource leaks integration tests has been created for branch 8.19 but encountered conflicts
#8458 [9.0] (backport #8396) [tests] split up serverless and resource leaks integration tests has been created for branch 9.0 but encountered conflicts

* fix: expose PreinstalledPackages * fix: make groups.go importable * feat: introduce new leak package for integration tests * feat: introduce new serverless package for integration tests * feat: incorporate serverless and extended runtime leak tests in mage * fix: normalise powershell (windows) and bash (unix) integration tests invocation flow * fix: use -test prefixed args for test-wise argument in buildkite-integration-tests.ps1 * fix: add comment about test timeout 2h0m0s * fix: correct wording * doc: add integration:TestServerless and update integration:TestForResourceLeaks in test-framework-dev-guide.md * fix: return directly the test exit code for ps1 scripts * fix: rename file that host TestMain to main_test.go * fix: correct build tags for serverless integration tests package * fix: leverage log.Fatalf in serverless/main_test.go * feat: expand the BK log group on failure * tmp: introduce artificial test failure to test buildkite logging group expansion * Revert "tmp: introduce artificial test failure to test buildkite logging group expansion" This reverts commit ee6bd63. * fix: typo * fix: remove redundant BK log group expansion (cherry picked from commit 92e139c) # Conflicts: # .buildkite/scripts/buildkite-integration-tests.sh # .buildkite/scripts/integration-tests.ps1 # testing/integration/auditd_monitoring_test.go # testing/integration/fleetserver_test.go # testing/integration/logs_ingestion_test.go # testing/integration/network_traffic_monitoring_test.go # testing/integration/osquery_monitoring_test.go

* fix: expose PreinstalledPackages * fix: make groups.go importable * feat: introduce new leak package for integration tests * feat: introduce new serverless package for integration tests * feat: incorporate serverless and extended runtime leak tests in mage * fix: normalise powershell (windows) and bash (unix) integration tests invocation flow * fix: use -test prefixed args for test-wise argument in buildkite-integration-tests.ps1 * fix: add comment about test timeout 2h0m0s * fix: correct wording * doc: add integration:TestServerless and update integration:TestForResourceLeaks in test-framework-dev-guide.md * fix: return directly the test exit code for ps1 scripts * fix: rename file that host TestMain to main_test.go * fix: correct build tags for serverless integration tests package * fix: leverage log.Fatalf in serverless/main_test.go * feat: expand the BK log group on failure * tmp: introduce artificial test failure to test buildkite logging group expansion * Revert "tmp: introduce artificial test failure to test buildkite logging group expansion" This reverts commit ee6bd63. * fix: typo * fix: remove redundant BK log group expansion (cherry picked from commit 92e139c) # Conflicts: # .buildkite/scripts/buildkite-integration-tests.sh # .buildkite/scripts/integration-tests.ps1 # testing/integration/auditd_monitoring_test.go # testing/integration/fleetserver_test.go # testing/integration/network_traffic_monitoring_test.go # testing/integration/osquery_monitoring_test.go

* fix: expose PreinstalledPackages * fix: make groups.go importable * feat: introduce new leak package for integration tests * feat: introduce new serverless package for integration tests * feat: incorporate serverless and extended runtime leak tests in mage * fix: normalise powershell (windows) and bash (unix) integration tests invocation flow * fix: use -test prefixed args for test-wise argument in buildkite-integration-tests.ps1 * fix: add comment about test timeout 2h0m0s * fix: correct wording * doc: add integration:TestServerless and update integration:TestForResourceLeaks in test-framework-dev-guide.md * fix: return directly the test exit code for ps1 scripts * fix: rename file that host TestMain to main_test.go * fix: correct build tags for serverless integration tests package * fix: leverage log.Fatalf in serverless/main_test.go * feat: expand the BK log group on failure * tmp: introduce artificial test failure to test buildkite logging group expansion * Revert "tmp: introduce artificial test failure to test buildkite logging group expansion" This reverts commit ee6bd63. * fix: typo * fix: remove redundant BK log group expansion (cherry picked from commit 92e139c) # Conflicts: # .buildkite/scripts/integration-tests.ps1 # testing/integration/auditd_monitoring_test.go # testing/integration/fleetserver_test.go # testing/integration/network_traffic_monitoring_test.go # testing/integration/osquery_monitoring_test.go

* fix: expose PreinstalledPackages * fix: make groups.go importable * feat: introduce new leak package for integration tests * feat: introduce new serverless package for integration tests * feat: incorporate serverless and extended runtime leak tests in mage * fix: normalise powershell (windows) and bash (unix) integration tests invocation flow * fix: use -test prefixed args for test-wise argument in buildkite-integration-tests.ps1 * fix: add comment about test timeout 2h0m0s * fix: correct wording * doc: add integration:TestServerless and update integration:TestForResourceLeaks in test-framework-dev-guide.md * fix: return directly the test exit code for ps1 scripts * fix: rename file that host TestMain to main_test.go * fix: correct build tags for serverless integration tests package * fix: leverage log.Fatalf in serverless/main_test.go * feat: expand the BK log group on failure * tmp: introduce artificial test failure to test buildkite logging group expansion * Revert "tmp: introduce artificial test failure to test buildkite logging group expansion" This reverts commit ee6bd63. * fix: typo * fix: remove redundant BK log group expansion (cherry picked from commit 92e139c) # Conflicts: # .buildkite/scripts/buildkite-integration-tests.sh # .buildkite/scripts/integration-tests.ps1 # testing/integration/auditd_monitoring_test.go # testing/integration/network_traffic_monitoring_test.go # testing/integration/osquery_monitoring_test.go

… integration tests (#8458) * [tests] split up serverless and resource leaks integration tests (#8396) * fix: expose PreinstalledPackages * fix: make groups.go importable * feat: introduce new leak package for integration tests * feat: introduce new serverless package for integration tests * feat: incorporate serverless and extended runtime leak tests in mage * fix: normalise powershell (windows) and bash (unix) integration tests invocation flow * fix: use -test prefixed args for test-wise argument in buildkite-integration-tests.ps1 * fix: add comment about test timeout 2h0m0s * fix: correct wording * doc: add integration:TestServerless and update integration:TestForResourceLeaks in test-framework-dev-guide.md * fix: return directly the test exit code for ps1 scripts * fix: rename file that host TestMain to main_test.go * fix: correct build tags for serverless integration tests package * fix: leverage log.Fatalf in serverless/main_test.go * feat: expand the BK log group on failure * tmp: introduce artificial test failure to test buildkite logging group expansion * Revert "tmp: introduce artificial test failure to test buildkite logging group expansion" This reverts commit ee6bd63. * fix: typo * fix: remove redundant BK log group expansion (cherry picked from commit 92e139c) # Conflicts: # .buildkite/scripts/buildkite-integration-tests.sh # .buildkite/scripts/integration-tests.ps1 # testing/integration/auditd_monitoring_test.go # testing/integration/network_traffic_monitoring_test.go # testing/integration/osquery_monitoring_test.go * fix: resolve conflicts * fix: remove -integration.fips flag --------- Co-authored-by: Panos Koutsovasilis <panos.koutsovasilis@elastic.co>

…s integration tests (#8457) * [tests] split up serverless and resource leaks integration tests (#8396) * fix: expose PreinstalledPackages * fix: make groups.go importable * feat: introduce new leak package for integration tests * feat: introduce new serverless package for integration tests * feat: incorporate serverless and extended runtime leak tests in mage * fix: normalise powershell (windows) and bash (unix) integration tests invocation flow * fix: use -test prefixed args for test-wise argument in buildkite-integration-tests.ps1 * fix: add comment about test timeout 2h0m0s * fix: correct wording * doc: add integration:TestServerless and update integration:TestForResourceLeaks in test-framework-dev-guide.md * fix: return directly the test exit code for ps1 scripts * fix: rename file that host TestMain to main_test.go * fix: correct build tags for serverless integration tests package * fix: leverage log.Fatalf in serverless/main_test.go * feat: expand the BK log group on failure * tmp: introduce artificial test failure to test buildkite logging group expansion * Revert "tmp: introduce artificial test failure to test buildkite logging group expansion" This reverts commit ee6bd63. * fix: typo * fix: remove redundant BK log group expansion (cherry picked from commit 92e139c) # Conflicts: # .buildkite/scripts/integration-tests.ps1 # testing/integration/auditd_monitoring_test.go # testing/integration/fleetserver_test.go # testing/integration/network_traffic_monitoring_test.go # testing/integration/osquery_monitoring_test.go * fix: resolve conflicts * fix: remove irrelevant files * fix: point to the correct path for "fleet-server.json" in TestInstallFleetServerBootstrap --------- Co-authored-by: Panos Koutsovasilis <panos.koutsovasilis@elastic.co>

…s integration tests (#8456) * [tests] split up serverless and resource leaks integration tests (#8396) * fix: expose PreinstalledPackages * fix: make groups.go importable * feat: introduce new leak package for integration tests * feat: introduce new serverless package for integration tests * feat: incorporate serverless and extended runtime leak tests in mage * fix: normalise powershell (windows) and bash (unix) integration tests invocation flow * fix: use -test prefixed args for test-wise argument in buildkite-integration-tests.ps1 * fix: add comment about test timeout 2h0m0s * fix: correct wording * doc: add integration:TestServerless and update integration:TestForResourceLeaks in test-framework-dev-guide.md * fix: return directly the test exit code for ps1 scripts * fix: rename file that host TestMain to main_test.go * fix: correct build tags for serverless integration tests package * fix: leverage log.Fatalf in serverless/main_test.go * feat: expand the BK log group on failure * tmp: introduce artificial test failure to test buildkite logging group expansion * Revert "tmp: introduce artificial test failure to test buildkite logging group expansion" This reverts commit ee6bd63. * fix: typo * fix: remove redundant BK log group expansion (cherry picked from commit 92e139c) # Conflicts: # .buildkite/scripts/buildkite-integration-tests.sh # .buildkite/scripts/integration-tests.ps1 # testing/integration/auditd_monitoring_test.go # testing/integration/fleetserver_test.go # testing/integration/network_traffic_monitoring_test.go # testing/integration/osquery_monitoring_test.go * fix: resolve conflicts * fix: remove irrelevant files * fix: remove -integration.fips flag * fix: point to the correct path for "fleet-server.json" in TestInstallFleetServerBootstrap --------- Co-authored-by: Panos Koutsovasilis <panos.koutsovasilis@elastic.co>

…ts-oblt-cli * feature/serverless-its-oblt-cli: (51 commits) as agreed let's move the group to the kb.integration pipeline Update .github/workflows/serverless-project.yml ci: invoke serverless integration tests package from BK github-actions: create a serverless project daily [tests] split up serverless and resource leaks integration tests (#8396) chore: Update to elastic/beats@dfdc12e33de0 (#8446) Lock upgrade marker (#8254) build(deps): bump github.com/elastic/elastic-agent-system-metrics from 0.11.12 to 0.11.13 (#8420) Add docker image name template and renamed fips cloud specs (#8429) buildkite(scripts): refactor common scripts (#8365) Use require.Eventually to try and address flakiness (#8421) Fix pre-command to support extended testing (#8418) [Automation] Bump Golang version to 1.24.4 (#8384) chore: Update to elastic/beats@aed2a8b768bd (#8423) [main][Automation] Update versions (#8425) Add Pipeline to deploy custom agent image for FIPS testing (#8035) ci: automatically update OTel components (#8288) [Automation] Bump VM Image version to 1749258065 (#8390) fix: increase context timeout to accommodate for slower machines in Test_checkForUnprivilegedVault (#8374) otel: add cumulativetodeltaprocessor to EDOT collector (#8372) ...

…s integration tests (#8455) * [tests] split up serverless and resource leaks integration tests (#8396) * fix: expose PreinstalledPackages * fix: make groups.go importable * feat: introduce new leak package for integration tests * feat: introduce new serverless package for integration tests * feat: incorporate serverless and extended runtime leak tests in mage * fix: normalise powershell (windows) and bash (unix) integration tests invocation flow * fix: use -test prefixed args for test-wise argument in buildkite-integration-tests.ps1 * fix: add comment about test timeout 2h0m0s * fix: correct wording * doc: add integration:TestServerless and update integration:TestForResourceLeaks in test-framework-dev-guide.md * fix: return directly the test exit code for ps1 scripts * fix: rename file that host TestMain to main_test.go * fix: correct build tags for serverless integration tests package * fix: leverage log.Fatalf in serverless/main_test.go * feat: expand the BK log group on failure * tmp: introduce artificial test failure to test buildkite logging group expansion * Revert "tmp: introduce artificial test failure to test buildkite logging group expansion" This reverts commit ee6bd63. * fix: typo * fix: remove redundant BK log group expansion (cherry picked from commit 92e139c) # Conflicts: # .buildkite/scripts/buildkite-integration-tests.sh # .buildkite/scripts/integration-tests.ps1 # testing/integration/auditd_monitoring_test.go # testing/integration/fleetserver_test.go # testing/integration/logs_ingestion_test.go # testing/integration/network_traffic_monitoring_test.go # testing/integration/osquery_monitoring_test.go * fix: resolve conflicts * fix: go fmt * fix: remove irrelevant files * fix: remove -integration.fips flag * fix: point to the correct path for "fleet-server.json" in TestInstallFleetServerBootstrap --------- Co-authored-by: Panos Koutsovasilis <panos.koutsovasilis@elastic.co>

pkoutsovasilis added 6 commits June 9, 2025 10:10

fix: expose PreinstalledPackages

b6a782b

fix: make groups.go importable

5cb57a7

feat: introduce new leak package for integration tests

35869d9

feat: introduce new serverless package for integration tests

f91b367

feat: incorporate serverless and extended runtime leak tests in mage

4058895

fix: normalise powershell (windows) and bash (unix) integration tests…

8484c1e

… invocation flow

pkoutsovasilis requested review from ycombinator and pchila June 9, 2025 07:20

pkoutsovasilis self-assigned this Jun 9, 2025

pkoutsovasilis added Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team skip-changelog backport-active-all Automated backport with mergify to all the active branches labels Jun 9, 2025

pkoutsovasilis marked this pull request as ready for review June 9, 2025 09:34

pkoutsovasilis requested review from a team as code owners June 9, 2025 09:34

pkoutsovasilis mentioned this pull request Jun 9, 2025

ci(serverless): use OIDC for running the serverless ITs #8263

Merged

8 tasks

pchila previously approved these changes Jun 9, 2025

View reviewed changes

v1v previously approved these changes Jun 9, 2025

View reviewed changes

.buildkite/scripts/steps/integration_tests_tf.ps1 Outdated Show resolved Hide resolved

.buildkite/scripts/buildkite-integration-tests.sh Outdated Show resolved Hide resolved

.buildkite/scripts/buildkite-integration-tests.ps1 Outdated Show resolved Hide resolved

pkoutsovasilis added 4 commits June 9, 2025 13:16

fix: use -test prefixed args for test-wise argument in buildkite-inte…

11f6cf1

…gration-tests.ps1

fix: add comment about test timeout 2h0m0s

a0ed6f1

fix: correct wording

4dbfd5f

doc: add integration:TestServerless and update integration:TestForRes…

f4f395c

…ourceLeaks in test-framework-dev-guide.md

pkoutsovasilis dismissed stale reviews from v1v and pchila via f4f395c June 9, 2025 10:26

v1v previously approved these changes Jun 9, 2025

View reviewed changes

.buildkite/scripts/steps/integration_tests_tf.ps1 Outdated Show resolved Hide resolved

.buildkite/scripts/buildkite-integration-tests.ps1 Outdated Show resolved Hide resolved

fix: return directly the test exit code for ps1 scripts

a2a072d

pkoutsovasilis dismissed v1v’s stale review via a2a072d June 9, 2025 10:41

fix: remove redundant BK log group expansion

f088a5f

pkoutsovasilis dismissed stale reviews from v1v, pchila, and dliappis via f088a5f June 11, 2025 14:48

pkoutsovasilis requested review from dliappis, v1v, pchila and ycombinator June 11, 2025 15:28

ycombinator approved these changes Jun 11, 2025

View reviewed changes

v1v approved these changes Jun 11, 2025

View reviewed changes

pkoutsovasilis merged commit 92e139c into elastic:main Jun 11, 2025
11 checks passed

pkoutsovasilis deleted the feat/split_up_integration_tests branch June 11, 2025 17:44

mergify bot mentioned this pull request Jun 11, 2025

[8.17] (backport #8396) [tests] split up serverless and resource leaks integration tests #8455

Merged

8 tasks

This was referenced Jun 11, 2025

[8.18] (backport #8396) [tests] split up serverless and resource leaks integration tests #8456

Merged

[8.19] (backport #8396) [tests] split up serverless and resource leaks integration tests #8457

Merged

mergify bot mentioned this pull request Jun 11, 2025

[9.0] (backport #8396) [tests] split up serverless and resource leaks integration tests #8458

Merged

8 tasks

pkoutsovasilis mentioned this pull request Jun 27, 2025

feat: singleTest mage target for each integration test package #8691

Open

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[tests] split up serverless and resource leaks integration tests #8396

[tests] split up serverless and resource leaks integration tests #8396

Uh oh!

pkoutsovasilis commented Jun 9, 2025 •

edited

Loading

Uh oh!

pkoutsovasilis commented Jun 9, 2025

Uh oh!

elasticmachine commented Jun 9, 2025

Uh oh!

pchila left a comment

Uh oh!

v1v left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

v1v left a comment

Uh oh!

Uh oh!

Uh oh!

elastic-sonarqube bot commented Jun 11, 2025

Uh oh!

elasticmachine commented Jun 11, 2025

Uh oh!

v1v left a comment

Uh oh!

Uh oh!

github-actions bot commented Jun 11, 2025

Uh oh!

mergify bot commented Jun 11, 2025 •

edited

Loading

Uh oh!

Uh oh!

[tests] split up serverless and resource leaks integration tests #8396

[tests] split up serverless and resource leaks integration tests #8396

Uh oh!

Conversation

pkoutsovasilis commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Disruptive User Impact

How to test this PR locally

Related issues

Uh oh!

pkoutsovasilis commented Jun 9, 2025

Uh oh!

elasticmachine commented Jun 9, 2025

Uh oh!

pchila left a comment

Choose a reason for hiding this comment

Uh oh!

v1v left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

v1v left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

elastic-sonarqube bot commented Jun 11, 2025

Quality Gate passed

Uh oh!

elasticmachine commented Jun 11, 2025

💚 Build Succeeded

History

Uh oh!

v1v left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Jun 11, 2025

Uh oh!

mergify bot commented Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

Uh oh!

pkoutsovasilis commented Jun 9, 2025 •

edited

Loading

mergify bot commented Jun 11, 2025 •

edited

Loading