revamp: use a classic test for schema validation by cbeauchesne · Pull Request #6437 · DataDog/system-tests

cbeauchesne · 2026-03-04T23:40:49Z

Motivation

Previously, schema validation was performed at scenario level, since we wanted to have those assertions for all end-to-end scenarios. As it was not possible to declare more than one scenario on test classes, I used this pattern.

But not only it's now possible to declare several scenario on one class, but we can also declare a group.

Having schema deep inside internals is a friction for users who want to improve it.

Changes

Move everything related to schema validation in tests/schemas

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Test logic is modified? -> Get a review from RFC owner.
- Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
A docker base image is modified?
- the relevant build-XXX-image label is present
A scenario is added, removed or renamed?
- Get a review from R&P team

github-actions · 2026-03-04T23:41:31Z

CODEOWNERS have been resolved as:

tests/schemas/__init__.py                                               @DataDog/system-tests-core
tests/schemas/test_schemas.py                                           @DataDog/system-tests-core
tests/schemas/utils/__init__.py                                         @DataDog/system-tests-core
.shellcheck                                                             @DataDog/system-tests-core
conftest.py                                                             @DataDog/system-tests-core
pyproject.toml                                                          @DataDog/system-tests-core
tests/parametric/test_process_discovery.py                              @DataDog/system-tests-core @DataDog/apm-sdk-capabilities
tests/test_the_test/scenarios.json                                      @DataDog/system-tests-core
tests/test_the_test/test_compute_libraries_and_scenarios.py             @DataDog/system-tests-core
tests/test_the_test/test_conventions.py                                 @DataDog/system-tests-core
utils/__init__.py                                                       @DataDog/system-tests-core
utils/_context/_scenarios/endtoend.py                                   @DataDog/system-tests-core
utils/_context/_scenarios/otel_collector.py                             @DataDog/system-tests-core
utils/_decorators.py                                                    @DataDog/system-tests-core
utils/interfaces/_core.py                                               @DataDog/system-tests-core
utils/scripts/compute_libraries_and_scenarios.py                        @DataDog/system-tests-core
utils/scripts/libraries_and_scenarios_rules.yml                         @DataDog/system-tests-core
tests/schemas/utils/agent/-request.json                                 @DataDog/system-tests-core
tests/schemas/utils/agent/api/v0.1/configurations-request.json          @DataDog/system-tests-core
tests/schemas/utils/agent/api/v0.1/org-request.json                     @DataDog/system-tests-core
tests/schemas/utils/agent/api/v0.1/pipeline_stats-request.json          @DataDog/system-tests-core
tests/schemas/utils/agent/api/v0.1/status-request.json                  @DataDog/system-tests-core
tests/schemas/utils/agent/api/v0.2/stats-request.json                   @DataDog/system-tests-core
tests/schemas/utils/agent/api/v0.2/traces-request.json                  @DataDog/system-tests-core
tests/schemas/utils/agent/api/v1/check_run-request.json                 @DataDog/system-tests-core
tests/schemas/utils/agent/api/v1/discovery-request.json                 @DataDog/system-tests-core
tests/schemas/utils/agent/api/v1/metadata-request.json                  @DataDog/system-tests-core
tests/schemas/utils/agent/api/v1/series-request.json                    @DataDog/system-tests-core
tests/schemas/utils/agent/api/v1/validate-request.json                  @DataDog/system-tests-core
tests/schemas/utils/agent/api/v2/apmtelemetry-request.json              @DataDog/system-tests-core
tests/schemas/utils/agent/api/v2/appsecevts-request.json                @DataDog/system-tests-core
tests/schemas/utils/agent/api/v2/debugger-request.json                  @DataDog/system-tests-core
tests/schemas/utils/agent/api/v2/logs-request.json                      @DataDog/system-tests-core
tests/schemas/utils/agent/api/v2/profile-request.json                   @DataDog/system-tests-core
tests/schemas/utils/agent/api/v2/series-request.json                    @DataDog/system-tests-core
tests/schemas/utils/agent/intake/-request.json                          @DataDog/system-tests-core
tests/schemas/utils/library/-request.json                               @DataDog/system-tests-core
tests/schemas/utils/library/appsec/proxy/api/v2/appsecevts-request.json  @DataDog/system-tests-core
tests/schemas/utils/library/appsec/proxy/v1/input-request.json          @DataDog/system-tests-core
tests/schemas/utils/library/debugger/v1/diagnostics-request.json        @DataDog/system-tests-core
tests/schemas/utils/library/debugger/v1/input-request.json              @DataDog/system-tests-core
tests/schemas/utils/library/info-request.json                           @DataDog/system-tests-core
tests/schemas/utils/library/process-discovery.json                      @DataDog/system-tests-core
tests/schemas/utils/library/profiling/v1/input-request.json             @DataDog/system-tests-core
tests/schemas/utils/library/symdb/v1/input-request.json                 @DataDog/system-tests-core
tests/schemas/utils/library/telemetry-request.json                      @DataDog/system-tests-core
tests/schemas/utils/library/telemetry/proxy/api/v2/apmtelemetry-request.json  @DataDog/system-tests-core
tests/schemas/utils/library/v0.1/pipeline_stats-request.json            @DataDog/system-tests-core
tests/schemas/utils/library/v0.4/misc/trace-item.json                   @DataDog/system-tests-core
tests/schemas/utils/library/v0.4/traces-request.json                    @DataDog/system-tests-core
tests/schemas/utils/library/v0.5/traces-request.json                    @DataDog/system-tests-core
tests/schemas/utils/library/v0.6/stats-request.json                     @DataDog/system-tests-core
tests/schemas/utils/library/v0.7/config-request.json                    @DataDog/system-tests-core
tests/schemas/utils/library/v0.7/misc/config-state.json                 @DataDog/system-tests-core
tests/schemas/utils/miscs/README.md                                     @DataDog/system-tests-core
tests/schemas/utils/miscs/appsec/contexts/_definitions/all_context.json  @DataDog/system-tests-core
tests/schemas/utils/miscs/appsec/contexts/actor/0.1.0.json              @DataDog/system-tests-core
tests/schemas/utils/miscs/appsec/contexts/agent/0.1.0.json              @DataDog/system-tests-core
tests/schemas/utils/miscs/appsec/contexts/code_location/0.1.0.json      @DataDog/system-tests-core
tests/schemas/utils/miscs/appsec/contexts/host/0.1.0.json               @DataDog/system-tests-core
tests/schemas/utils/miscs/appsec/contexts/http/0.1.0.json               @DataDog/system-tests-core
tests/schemas/utils/miscs/appsec/contexts/http/1.0.0.json               @DataDog/system-tests-core
tests/schemas/utils/miscs/appsec/contexts/http/parameters.json          @DataDog/system-tests-core
tests/schemas/utils/miscs/appsec/contexts/library/0.1.0.json            @DataDog/system-tests-core
tests/schemas/utils/miscs/appsec/contexts/service/0.1.0.json            @DataDog/system-tests-core
tests/schemas/utils/miscs/appsec/contexts/service_entry_span/0.1.0.json  @DataDog/system-tests-core
tests/schemas/utils/miscs/appsec/contexts/span/0.1.0.json               @DataDog/system-tests-core
tests/schemas/utils/miscs/appsec/contexts/sqreen/0.1.0.json             @DataDog/system-tests-core
tests/schemas/utils/miscs/appsec/contexts/tags/0.1.0.json               @DataDog/system-tests-core
tests/schemas/utils/miscs/appsec/contexts/trace/0.1.0.json              @DataDog/system-tests-core
tests/schemas/utils/miscs/appsec/events/0.1.0.json                      @DataDog/system-tests-core
tests/schemas/utils/miscs/appsec/events/1.0.0.json                      @DataDog/system-tests-core
tests/schemas/utils/miscs/appsec/events/_definitions/all_events.json    @DataDog/system-tests-core
tests/schemas/utils/miscs/appsec/events/_definitions/rule/0.1.0.json    @DataDog/system-tests-core
tests/schemas/utils/miscs/appsec/events/_definitions/rule/1.0.0.json    @DataDog/system-tests-core
tests/schemas/utils/miscs/appsec/events/_definitions/rule_match/0.1.0.json  @DataDog/system-tests-core
tests/schemas/utils/miscs/appsec/events/_definitions/rule_match/1.0.0.json  @DataDog/system-tests-core
tests/schemas/utils/miscs/ip.json                                       @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/common/events/apm-onboarding.json   @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/common/events/dependencies-loaded.json  @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/common/events/distributions.json    @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/common/events/generate-metrics.json  @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/common/events/integrations-change.json  @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/common/objects/apm_onboarding_header.json  @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/common/objects/dependency.json      @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/common/objects/distribution.json    @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/common/objects/error.json           @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/common/objects/integration.json     @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/common/objects/logs.json            @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/common/objects/metric.json          @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/common/objects/metric_with_namespace.json  @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/common/objects/product.json         @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/common/objects/products.json        @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/common/objects/remote_config.json   @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/common/objects/tags.json            @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/v1/events/app-started.json          @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/v1/objects/application.json         @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/v1/objects/configuration.json       @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/v1/objects/header.json              @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/v1/objects/host.json                @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/v1/telemetry_request.json           @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/v2/events/app-started.json          @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/v2/events/client-configuration-change.json  @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/v2/events/extended-heartbeat.json   @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/v2/events/logs.json                 @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/v2/events/message-batch.json        @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/v2/events/product-change.json       @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/v2/objects/application.json         @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/v2/objects/configuration.json       @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/v2/objects/header.json              @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/v2/objects/host.json                @DataDog/system-tests-core
tests/schemas/utils/miscs/telemetry/v2/telemetry_request.json           @DataDog/system-tests-core
tests/schemas/utils/otel_collector/api/v2/series-request.json           @DataDog/system-tests-core
tests/schemas/utils/otel_collector/intake-request.json                  @DataDog/system-tests-core
tests/schemas/utils/otel_collector/latest/api/token-request.json        @DataDog/system-tests-core
tests/schemas/utils/otel_collector/latest/dynamic/instance-identity/document-request.json  @DataDog/system-tests-core
tests/schemas/utils/otel_collector/latest/meta-data/hostname-request.json  @DataDog/system-tests-core
tests/schemas/utils/otel_collector/latest/meta-data/instance-id-request.json  @DataDog/system-tests-core
tests/schemas/utils/schemas_validators.py                               @DataDog/system-tests-core
utils/interfaces/schemas/Dockerfile                                     @DataDog/system-tests-core
utils/interfaces/schemas/README.md                                      @DataDog/system-tests-core
utils/interfaces/schemas/index.html                                     @DataDog/system-tests-core
utils/interfaces/schemas/serve.sh                                       @DataDog/system-tests-core
utils/interfaces/schemas/serve_doc.py                                   @DataDog/system-tests-core

datadog-official · 2026-03-05T08:50:54Z

⚠️ Tests

✨ Fix all issues with BitsAI or with Cursor

⚠️ Warnings

❄️ 97 New flaky tests detected

tests.schemas.test_schemas.Test_DdtraceSchemas.test_agent[akka-http] from system_tests_suite (Datadog) (Fix with Cursor)

AssertionError: agent has schema error, please check logs

self = <tests.schemas.test_schemas.Test_DdtraceSchemas object at 0x7f154a5c7a70>

    def test_agent(self):
        known_bugs = [
            SchemaBug(
                endpoint="/api/v2/debugger",
                data_path="$",
                condition=context.library > "nodejs@5.36.0",
...

tests.schemas.test_schemas.Test_DdtraceSchemas.test_agent[apache-mod-7.0-zts] from system_tests_suite (Datadog) (Fix with Cursor)

AssertionError: agent has schema error, please check logs

self = <tests.schemas.test_schemas.Test_DdtraceSchemas object at 0x7f6edc672ea0>

    def test_agent(self):
        known_bugs = [
            SchemaBug(
                endpoint="/api/v2/debugger",
                data_path="$",
                condition=context.library > "nodejs@5.36.0",
...

tests.schemas.test_schemas.Test_DdtraceSchemas.test_agent[apache-mod-7.0] from system_tests_suite (Datadog) (Fix with Cursor)

AssertionError: agent has schema error, please check logs

self = <tests.schemas.test_schemas.Test_DdtraceSchemas object at 0x7f5584e57e00>

    def test_agent(self):
        known_bugs = [
            SchemaBug(
                endpoint="/api/v2/debugger",
                data_path="$",
                condition=context.library > "nodejs@5.36.0",
...

View all

ℹ️ Info

🧪 All tests passed

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 76cf158 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!}

revamp: use a classic test for schema validation

892488c

Fix test_the_test, and adds a test use case

fa3ce1c

cbeauchesne added 2 commits March 5, 2026 10:23

Add APMSP-2590, fix schema assertion logic

73f10f0

Adds auxiliary_test

76cf158

cbeauchesne marked this pull request as ready for review March 5, 2026 17:07

cbeauchesne requested review from a team as code owners March 5, 2026 17:07

cbeauchesne requested review from mtoffl01 and removed request for a team March 5, 2026 17:07

cbeauchesne merged commit 01d133a into main Mar 5, 2026
2708 of 2712 checks passed

cbeauchesne deleted the cbeauchense/schema-tests branch March 5, 2026 17:08

cbeauchesne added the changelog label Mar 5, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

revamp: use a classic test for schema validation#6437

revamp: use a classic test for schema validation#6437
cbeauchesne merged 4 commits intomainfrom
cbeauchense/schema-tests

cbeauchesne commented Mar 4, 2026

Uh oh!

github-actions Bot commented Mar 4, 2026 •

edited

Loading

Uh oh!

datadog-official Bot commented Mar 5, 2026 •

edited by datadog-datadog-prod-us1-2 Bot

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

cbeauchesne commented Mar 4, 2026

Motivation

Changes

Workflow

Reviewer checklist

Uh oh!

github-actions Bot commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-official Bot commented Mar 5, 2026 • edited by datadog-datadog-prod-us1-2 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions Bot commented Mar 4, 2026 •

edited

Loading

datadog-official Bot commented Mar 5, 2026 •

edited by datadog-datadog-prod-us1-2 Bot

Loading