[BUG]: Playwright projects and config filters ignored when DD_TRACE_ENABLED=true

[hatufacci]

Tracer Version(s)

5.71.0

Node.js Version(s)

22.18.0

Bug Report

@playwright/test version: 1.55.0

When running Playwright tests with Datadog CI Visibility enabled (DD_TRACE_ENABLED=true), the test runner ignores projects, testMatch, and grep filters defined in the Playwright config file. Instead, all tests in the directory are executed, even when the config should restrict the selection.

Steps to Reproduce:

Use Playwright with a config file that defines multiple projects and uses testMatch/grep for filtering.
Set DD_TRACE_ENABLED=true and run tests via npx playwright test --config=playwright.config.ts.
Observe that all tests are run, not just those matching the config filters.
Expected Behavior:
Only tests matching the projects, testMatch, and grep filters in the config should be executed.

Actual Behavior:
All tests are executed, ignoring the config’s filtering logic.

Disabling DD_TRACE_ENABLED restores correct test selection.

Reproduction Code

example of playwright config:

import { defineConfig } from "@playwright/test";
import dotenv from "dotenv";

dotenv.config({ quiet: true });

export default defineConfig({
timeout: 1200000,
testDir: "./tests/serverside/data",
globalSetup: require.resolve("./tests/global.setup.ts"),
globalTeardown: require.resolve("./tests/global.teardown.ts"),
/* Run tests in files in parallel /
fullyParallel: true,
/ Fail the build on CI if you accidentally left test.only in the source code. /
forbidOnly: !!process.env.CI,
/ Retry on CI only /
retries: process.env.CI ? 1 : 0,
/ Opt out of parallel tests on CI. /
workers: process.env.CI ? 10 : 6,
reporter: process.env.CI
? [["html"], ["json", { outputFile: "test-results.json" }], ["junit", { outputFile: "junit-report.xml" }]]
: "html",
use: {
browserName: "chromium",
headless: true,
actionTimeout: 15000,
baseURL: process.env.FUNCTIONS_APP_URL,
contextOptions: {
viewport: { width: 1440, height: 900 },
},
/ Collect trace when retrying the failed test. See https://playwright.dev/docs/trace-viewer */
trace: "retain-on-failure",
video: "retain-on-failure",
screenshot: "only-on-failure",
},
projects: [
{
name: "setupPipeline",
testMatch: ["/da.setup.spec.ts"],
},
{
name: "audienceDB",
dependencies: ["setupPipeline"],
grep: RegExp("(C3115370)"),
testMatch: ["/pipelineDB.spec.ts"],
},
{
name: "settings",
dependencies: ["setupPipeline"],
testMatch: ["/dataSettings.spec.ts"],
},
{
name: "filters",
dependencies: ["setupPipeline"],
testMatch: ["/dataFilters.spec.ts"],
},
],
});

Error Logs

Running 2122 tests using 10 workers

but with DD_TRACE_ENABLED=false (correct number of tests)
Running 32 tests using 10 workers

Tracer Config

DD_TRACE_ENABLED=true

Operating System

Ubuntu 22.04.5 LTS (Jammy Jellyfish)

Bundling

Unsure

[juan-fernandez]

hey @hatufacci thanks for the report.

Could you help me with these questions:

1. How do you initialize test optimization? Do you pass NODE_OPTIONS='-r dd-trace/ci/init' to npx playwright test --config=playwright.config.ts ? Or do you do something else?

2. Could you run another test but setting these env vars:

• DD_CIVISIBILITY_EARLY_FLAKE_DETECTION_ENABLED=false
• DD_CIVISIBILITY_FLAKY_RETRY_ENABLED=false
• DD_TEST_MANAGEMENT_ENABLED=false
• DD_CIVISIBILITY_IMPACTED_TESTS_DETECTION_ENABLED=false
  This helps me rule out issues with our advanced features

[juan-fernandez]

@hatufacci did you have time to look at the above comment?

[hatufacci]

@juan-fernandez Hey, we use export NODE_OPTIONS="-r dd-trace/ci/init" in run_tests.sh and import "dd-trace/ci/init"; in global.setup.ts file and this file is used in playwright config:
"```
export default defineConfig({
globalSetup: require.resolve("./tests/global.setup.ts"),"

there is no issue if use script with grep f.e. "npx playwright test --config=playwright.config.smoke.ts --grep=@smoke" but issue when use f.e. "npx playwright test --config=playwright.config.smokeDa.ts"

[juan-fernandez]

@juan-fernandez Hey, we use export NODE_OPTIONS="-r dd-trace/ci/init" in run_tests.sh and import "dd-trace/ci/init"; in global.setup.ts file and this file is used in playwright config: "``` export default defineConfig({ globalSetup: require.resolve("./tests/global.setup.ts"),"

there is no issue if use script with grep f.e. "npx playwright test --config=playwright.config.smoke.ts --grep=@smoke" but issue when use f.e. "npx playwright test --config=playwright.config.smokeDa.ts"

thanks! Can you try removing import "dd-trace/ci/init" from global.setup.ts? That's not the correct way to initialize the library. The NODE_OPTIONS should be good enough. I don't think that the source of the problem though.

Did you try passing the environment variables I mentioned in my earlier comment?

DD_CIVISIBILITY_EARLY_FLAKE_DETECTION_ENABLED=false
DD_CIVISIBILITY_FLAKY_RETRY_ENABLED=false
DD_TEST_MANAGEMENT_ENABLED=false
DD_CIVISIBILITY_IMPACTED_TESTS_DETECTION_ENABLED=false

[jbinto]

We've been slowly rolling out Test Optimization to our repos and had to quickly roll back ones that target Playwright, I suspect due to this issue. I believe we are using DD_TRACE_ENABLED=true, but this is a Next.js repo so it's not straightforward to figure out how the various .env files map thru docker, build time vs runtime, client vs server, etc.

The behaviour we were seeing was even stranger: in a suite with ~4000 tests (which gets split further by our CI parallelization), the runner was attempting to execute a large multiple of that: 166151, 86942, etc. I haven't figured the what the multiplier means, but my first guess is 20x plays in here somehow (Retries any tests you’re attempting to fix 20 times.).

I do think this issue is related, because each container was running the same number of tests, which suggests that the command line file prompt is being ignored.

I will attempt to reproduce and provide more information this afternoon.

[juan-fernandez]

We've been slowly rolling out Test Optimization to our repos and had to quickly roll back ones that target Playwright, I suspect due to this issue. I believe we are using DD_TRACE_ENABLED=true, but this is a Next.js repo so it's not straightforward to figure out how the various .env files map thru docker, build time vs runtime, client vs server, etc.

The behaviour we were seeing was even stranger: in a suite with ~4000 tests (which gets split further by our CI parallelization), the runner was attempting to execute a large multiple of that: 166151, 86942, etc. I haven't figured the what the multiplier means, but my first guess is 20x plays in here somehow (Retries any tests you’re attempting to fix 20 times.).

I do think this issue is related, because each container was running the same number of tests, which suggests that the command line file prompt is being ignored.

I will attempt to reproduce and provide more information this afternoon.

yeah the library is causing the issue for sure 😄. We don't modify the command line though, but there could be other ways in which this could be happening. I want to identify which feature is potentially causing this, hence the env vars to disable them 😄

[hatufacci]

@juan-fernandez Hey, we use export NODE_OPTIONS="-r dd-trace/ci/init" in run_tests.sh and import "dd-trace/ci/init"; in global.setup.ts file and this file is used in playwright config: "``` export default defineConfig({ globalSetup: require.resolve("./tests/global.setup.ts"),"

there is no issue if use script with grep f.e. "npx playwright test --config=playwright.config.smoke.ts --grep=@smoke" but issue when use f.e. "npx playwright test --config=playwright.config.smokeDa.ts"

thanks! Can you try removing import "dd-trace/ci/init" from global.setup.ts? That's not the correct way to initialize the library. The NODE_OPTIONS should be good enough. I don't think that the source of the problem though.

Did you try passing the environment variables I mentioned in my earlier comment?

DD_CIVISIBILITY_EARLY_FLAKE_DETECTION_ENABLED=false
DD_CIVISIBILITY_FLAKY_RETRY_ENABLED=false
DD_TEST_MANAGEMENT_ENABLED=false
DD_CIVISIBILITY_IMPACTED_TESTS_DETECTION_ENABLED=false

@juan-fernandez i could confirm that with these false settings - it seems all works well

[juan-fernandez]

@juan-fernandez Hey, we use export NODE_OPTIONS="-r dd-trace/ci/init" in run_tests.sh and import "dd-trace/ci/init"; in global.setup.ts file and this file is used in playwright config: "``` export default defineConfig({ globalSetup: require.resolve("./tests/global.setup.ts"),"

there is no issue if use script with grep f.e. "npx playwright test --config=playwright.config.smoke.ts --grep=@smoke" but issue when use f.e. "npx playwright test --config=playwright.config.smokeDa.ts"

thanks! Can you try removing import "dd-trace/ci/init" from global.setup.ts? That's not the correct way to initialize the library. The NODE_OPTIONS should be good enough. I don't think that the source of the problem though.
Did you try passing the environment variables I mentioned in my earlier comment?

DD_CIVISIBILITY_EARLY_FLAKE_DETECTION_ENABLED=false
DD_CIVISIBILITY_FLAKY_RETRY_ENABLED=false
DD_TEST_MANAGEMENT_ENABLED=false
DD_CIVISIBILITY_IMPACTED_TESTS_DETECTION_ENABLED=false

@juan-fernandez i could confirm that with these false settings - it seems all works well

thanks! That's useful. I've narrowed down that could be happening. My suspicion is some incompatibility with fullyParallel. I'll investigate and let you know

[juan-fernandez]

hey! I'm fixing this issue in #6736. The problem was not with fullyParallel but with early flake detection's logic in playwright: https://docs.datadoghq.com/tests/flaky_tests/early_flake_detection/?tab=javascripttypescript. This is why passing DD_CIVISIBILITY_EARLY_FLAKE_DETECTION_ENABLED=false fixes the issue.

I'll let you know once the fix is released so you can go back to enabling early flake detection if you want.

[juan-fernandez]

https://github.com/DataDog/dd-trace-js/releases/tag/v5.74.0 should include the fix for this issue. Feel free to reopen if you think otherwise