[9.0] (backport #8396) [tests] split up serverless and resource leaks integration tests

[mergify]

What does this PR do?

This PR refactors the Elastic Agent integration test structure by introducing dedicated sub-packages for specific test categories. The changes include:

• New serverless and leak test packages under testing/integration/, each with their own TestMain.
• Moved long-running resource leak tests to the leak package and removed the dependency on the TEST_LONG_RUNNING env var.
• Moved TestLogIngestionFleetManaged into a reusable function and invoked it from both integration and serverless packages.
• Normalized test entrypoints across Windows (PowerShell) and Unix (Bash) runners.
• Made PreinstalledPackages public and importable by other test packages.
• Renamed groups_test.go to groups.go to support shared usage.
• Enhanced mage integration:check to validate all test packages (integration/, integration/serverless, integration/leak).
• Updated Buildkite test scripts and pipeline definitions to accommodate the split.

Why is it important?

The testing/integration package had grown into a monolithic and hard-to-maintain suite containing various unrelated test types. This structural refactor:

• Improves clarity and maintainability by grouping tests by setup type (e.g., serverless, extended runtime).
• Reduces reliance on complex filtering logic using env vars or CLI args.
• Enables tailored TestMain implementations per package for better control of setup/teardown behavior.
• Builds a solid baseline for better alignment of Buildkite pipelines and runtime configurations.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

None expected for end-users. CI and internal contributors may need to adjust test invocation to use the correct package path or mage targets (e.g., mage integration:testServerless, integration:testForResourceLeaks).

How to test this PR locally

# Run standard integration tests
mage integration:test

# Run serverless smoke tests
mage integration:testServerless

# Run extended leak tests
mage integration:testForResourceLeaks

Related issues

• Relates Split out integration tests into different packages #8087
• Closes Split up serverless integration tests into dedicated package #8409
• Closes Split up runtime resource leak integration tests into dedicated package #8410

---

This is an automatic backport of pull request #8396 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 92e139c has failed:

On branch mergify/bp/9.0/pr-8396
Your branch is up to date with 'origin/9.0'.

You are currently cherry-picking commit 92e139ca2.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   .buildkite/bk.integration.pipeline.yml
	modified:   .buildkite/integration.pipeline.yml
	new file:   .buildkite/scripts/buildkite-integration-tests.ps1
	new file:   .buildkite/scripts/steps/integration_tests_tf.ps1
	modified:   .buildkite/scripts/steps/integration_tests_tf.sh
	modified:   docs/test-framework-dev-guide.md
	modified:   magefile.go
	modified:   pkg/testing/define/parser.go
	modified:   testing/integration/endpoint_test_tools.go
	modified:   testing/integration/fleetserver_test.go
	renamed:    testing/integration/groups_test.go -> testing/integration/groups.go
	modified:   testing/integration/init.go
	renamed:    testing/integration/agent_long_running_leak_test.go -> testing/integration/leak/long_running_test.go
	new file:   testing/integration/leak/main_test.go
	renamed:    testing/integration/agent_long_test_apache.json -> testing/integration/leak/testdata/agent_long_test_apache.json
	renamed:    testing/integration/agent_long_test_base_system_integ.json -> testing/integration/leak/testdata/agent_long_test_base_system_integ.json
	new file:   testing/integration/logs_ingestion.go
	modified:   testing/integration/logs_ingestion_test.go
	modified:   testing/integration/metrics_monitoring_test.go
	modified:   testing/integration/monitoring_probe_preserve_text_cfg_test.go
	modified:   testing/integration/monitoring_probe_reload_test.go
	new file:   testing/integration/serverless/logs_ingestion_test.go
	new file:   testing/integration/serverless/main_test.go

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	both modified:   .buildkite/scripts/buildkite-integration-tests.sh
	deleted by them: .buildkite/scripts/integration-tests.ps1
	both modified:   testing/integration/auditd_monitoring_test.go
	both modified:   testing/integration/network_traffic_monitoring_test.go
	both modified:   testing/integration/osquery_monitoring_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[pkoutsovasilis]

let's wait for the CI run on main to finish (92e139c), I want to be extra careful before merging this backport 🙂

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: ebf055d

History

• 💔 Build #22230 failed 46c0e75
• 💔 Build #22225 failed 84dbcd4

cc @pkoutsovasilis