Enable PHP FFE evaluation metric system tests

[leoromanovsky]

Motivation

PHP FFE evaluation metrics need system-test coverage before dd-trace-php PR 3911 can be treated as review-ready. The PHP runtime now records feature_flag.evaluations through native/libdatadog-sidecar code, so we need the shared FFE metric tests enabled against a locally built PHP tracer artifact rather than relying only on PHPT coverage.

Design doc: https://docs.google.com/document/d/1NvMfTpZWLBlFmEFNjdnlMyeVpy5l7KD8qujGFco6w2w/edit?tab=t.0

Related PRs:

• feat(ffe): add runtime-backed PHP feature flag evaluation dd-trace-php#3906: M1 runtime evaluation base
• Add FFE evaluation metrics dd-trace-php#3911: PHP native evaluation metrics
• Enable PHP FFE evaluation system tests #7003: PHP FFE evaluation scaffold base

Changes

This PR enables tests/ffe/test_flag_eval_metrics.py for PHP in the manifest only. It intentionally does not enable exposure tests; exposure validation is tracked separately in #7031 so each PHP milestone can be reviewed independently.

Decisions

The PR is stacked on leo.romanovsky/pr-g-php-ffe-scaffold because PR 3906 has not landed yet and the PHP FFE system-test hooks are introduced there. The local validation used a PHP 8.0 debug artifact from the PR 3911 dd-trace-php branch because the default FEATURE_FLAGGING_AND_EXPERIMENTATION weblog uses apache-mod-8.0.

Validation evidence:

cd /Users/leo.romanovsky/go/src/github.com/DataDog/dd-trace-php-ffe-metrics-restack
./tooling/bin/build-debug-artifact gnu-aarch64-8.0-nts \
  /Users/leo.romanovsky/go/src/github.com/DataDog/system-tests-pr-g-php-ffe-scaffold/binaries

cd /Users/leo.romanovsky/go/src/github.com/DataDog/system-tests-pr-g-php-ffe-scaffold
DOCKER_HOST=unix:///Users/leo.romanovsky/.colima/default/docker.sock \
DOCKER_CONFIG=/tmp/system-tests-empty-docker-config \
./build.sh php -i weblog -w apache-mod-8.0

DOCKER_HOST=unix:///Users/leo.romanovsky/.colima/default/docker.sock \
DOCKER_CONFIG=/tmp/system-tests-empty-docker-config \
TEST_LIBRARY=php \
./run.sh FEATURE_FLAGGING_AND_EXPERIMENTATION tests/ffe/test_flag_eval_metrics.py -vv

Result: 17 passed in 81.21s.

[github-actions]

CODEOWNERS have been resolved as:

manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php

[datadog-prod-us1-5]

 

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 53 Pipeline jobs failed

Testing the test | System Tests (php, dev) / End-to-end #1 / apache-mod-7.1 1     

🔧 Fix in code (Fix with Cursor).

21 failed tests due to assertion errors: expected 'expected' but got 'default' or 'default_fallback'.

🧪 1 Test failed

tests.ffe.test_dynamic_evaluation.Test_FFE_Flag_Parse_Error_Isolation.test_valid_flag_unaffected[apache-mod-7.1] from system_tests_suite   (Fix with Cursor)

AssertionError: Valid flag corrupted by malformed neighbour: expected 'expected', got 'default' (parse error on one flag must not abort processing of the whole config)
assert 'default' == 'expected'
  - expected
  + default

self = <tests.ffe.test_dynamic_evaluation.Test_FFE_Flag_Parse_Error_Isolation object at 0x7f39e6e910a0>

    def test_valid_flag_unaffected(self):
        assert self.r_valid.status_code == 200, f"Valid flag evaluation failed: {self.r_valid.text}"
        result = json.loads(self.r_valid.text)
...

Testing the test | System Tests (php, dev) / End-to-end #1 / apache-mod-7.1-zts 1     

🔧 Fix in code (Fix with Cursor).

21 failed tests due to assertion failures. Error in tests/ffe/test_dynamic_evaluation.py: Result value is 'default', expected 'expected'.

🧪 1 Test failed

tests.ffe.test_dynamic_evaluation.Test_FFE_Flag_Parse_Error_Isolation.test_valid_flag_unaffected[apache-mod-7.1-zts] from system_tests_suite   (Fix with Cursor)

AssertionError: Valid flag corrupted by malformed neighbour: expected 'expected', got 'default' (parse error on one flag must not abort processing of the whole config)
assert 'default' == 'expected'
  - expected
  + default

self = <tests.ffe.test_dynamic_evaluation.Test_FFE_Flag_Parse_Error_Isolation object at 0x7f070c4f34d0>

    def test_valid_flag_unaffected(self):
        assert self.r_valid.status_code == 200, f"Valid flag evaluation failed: {self.r_valid.text}"
        result = json.loads(self.r_valid.text)
...

Testing the test | System Tests (php, dev) / End-to-end #1 / apache-mod-7.2-zts 1     

🔧 Fix in code (Fix with Cursor).

21 failed tests. Error: Multiple assertion failures regarding unexpected 'default' values instead of expected outputs across various tests in feature flag evaluation.

🧪 1 Test failed

tests.ffe.test_dynamic_evaluation.Test_FFE_Flag_Parse_Error_Isolation.test_valid_flag_unaffected[apache-mod-7.2-zts] from system_tests_suite   (Fix with Cursor)

AssertionError: Valid flag corrupted by malformed neighbour: expected 'expected', got 'default' (parse error on one flag must not abort processing of the whole config)
assert 'default' == 'expected'
  - expected
  + default

self = <tests.ffe.test_dynamic_evaluation.Test_FFE_Flag_Parse_Error_Isolation object at 0x7f86d0105130>

    def test_valid_flag_unaffected(self):
        assert self.r_valid.status_code == 200, f"Valid flag evaluation failed: {self.r_valid.text}"
        result = json.loads(self.r_valid.text)
...

View all 53 failed jobs.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: a412f4f | Docs | Datadog PR Page | Give us feedback!}