Skip to content

test(debugger): budget limit for capture expressions#7111

Merged
tylfin merged 3 commits into
mainfrom
tyler.finethy/DEBUG-5730
Jun 11, 2026
Merged

test(debugger): budget limit for capture expressions#7111
tylfin merged 3 commits into
mainfrom
tyler.finethy/DEBUG-5730

Conversation

@tylfin

@tylfin tylfin commented Jun 8, 2026

Copy link
Copy Markdown
Member

Motivation

Extends the Live Debugger budget tests to ensure that probes configured with capture expressions respect the 1 snapshot per second limit, the same budget already enforced for full-snapshot log probes.

Changes

  • New probe fixture probe_capture_expressions_budgets.json: a line probe on the Budgets method with captureSnapshot: false and a captureExpressions entry on the loops parameter.
  • New test Test_Debugger_Probe_Budgets::test_log_line_capture_expression_budgets: fires 150 requests and asserts that only 1-20 snapshots carrying capture expressions are emitted (matching the existing test_log_line_budgets tolerance for the ~1/s budget).
  • Marked the test as a known bug for Java (DEBUG-5730) in manifests/java.yml: capture-expression probes currently emit on every invocation (150/150) and are not yet rate limited.

Testing

Ran DEBUGGER_PROBES_SNAPSHOT locally for Java:

  • Without the manifest entry, the test correctly fails (got 150 ... should be 1-20), confirming the budget is bypassed.
  • With the DEBUG-5730 bug declaration, the test is skipped as expected.

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

CODEOWNERS have been resolved as:

tests/debugger/utils/probes/probe_capture_expressions_budgets.json      @DataDog/debugger @DataDog/system-tests-core
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
tests/debugger/test_debugger_probe_budgets.py                           @DataDog/debugger @DataDog/system-tests-core

@datadog-official

datadog-official Bot commented Jun 8, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: bfc6cda | Docs | Datadog PR Page | Give us feedback!

@tylfin tylfin requested review from P403n1x87 and jpbempel June 8, 2026 18:15
@tylfin tylfin marked this pull request as ready for review June 8, 2026 19:51
@tylfin tylfin requested review from a team as code owners June 8, 2026 19:51
@tylfin tylfin requested review from daniel-romano-DD, manuel-alvarez-alvarez and wconti27 and removed request for a team June 8, 2026 19:51
Add a Live Debugger budget test verifying that probes configured with
capture expressions respect the 1 snapshot per second limit. Java and Python are
marked as a known bug (DEBUG-5730) since capture-expression probes are
not yet rate limited there.
@tylfin tylfin force-pushed the tyler.finethy/DEBUG-5730 branch from e4490c0 to f5f7c5b Compare June 8, 2026 19:52

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e4490c0cc6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

lines=self.method_and_language_to_line_number("Budgets", self.get_tracer()["language"]),
)

def test_log_line_capture_expression_budgets(self):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Add the new test to the scenario map

This new test is not added to tests/test_the_test/scenarios.json, which is the map loaded by utils/scripts/compute_libraries_and_scenarios.py to translate manifest nodeids and changed test files into scenarios. Because the new manifest entries use this exact nodeid, future PRs that only adjust this test’s manifest status will not select DEBUGGER_PROBES_SNAPSHOT, so the CI impact calculation can silently miss the scenario. Please regenerate/update the scenario map entry for this test.

Useful? React with 👍 / 👎.


# Probe budgets aren't exact and can take time to be applied, so we allow a range of 1-20 snapshots with
# capture expressions for 150 requests, matching the 1/s snapshot limit.
assert 1 <= snapshots_with_captures <= 20, (

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm slightly concerned about the stability of this test. From experience, every time we;ve introduced time-based assertions, we've eventually been hit by the fact that we don;t control the underlying machine the test runs on, which introduces a lot of non-determinism.

In the end, we often had to expand the asserted window significantly to reduce flakiness, which defeats the purpose of the assertion.

Let's give this a try, but if it turns out that we get too many failures that cannot be correlated with a real issue, we'll need to rethink this approach.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAIK the lack of stability for the debugger tests was always a missing waiting signal, we should be fairly deterministic as long as we gate on the proper signals

@cbeauchesne cbeauchesne left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved, with one comment :)

@tylfin tylfin merged commit d4ca060 into main Jun 11, 2026
827 of 829 checks passed
@tylfin tylfin deleted the tyler.finethy/DEBUG-5730 branch June 11, 2026 18:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants