feat(dynamic-sampling): add EAP based query for retrieving transaction volumes#115047
feat(dynamic-sampling): add EAP based query for retrieving transaction volumes#115047constantinius wants to merge 73 commits into
Conversation
…r-org-get-eap-transaction-volumes
| @dataclass | ||
| class _EAPProjectTransactionVolumesAccumulator: | ||
| transaction_counts: list[tuple[str, float]] = field(default_factory=list) | ||
| total_num_transactions: float = 0 | ||
| indexed: int = 0 |
There was a problem hiding this comment.
I'm not sure why we need this - we don't seem to do anything with the results AFAICT?
There was a problem hiding this comment.
They are mapped to the equivalence of ProjectTransactions: total_num_transactions and
total_num_classes
…transaction volumes with max results limit
| def run_batched_spans_table_query( | ||
| query: dict[str, Any], | ||
| chunk_size: int, | ||
| max_results: int | None = None, | ||
| ) -> Iterator[list[dict[str, Any]]]: | ||
| if max_results is not None and max_results <= 0: | ||
| return | ||
|
|
||
| offset = 0 | ||
|
|
||
| while True: | ||
| limit = chunk_size | ||
| if max_results is not None: | ||
| limit = min(limit, max_results - offset) | ||
|
|
||
| result = Spans.run_table_query(**query, offset=offset, limit=limit) | ||
| data = result.get("data", []) | ||
|
|
||
| if not data: | ||
| return | ||
|
|
||
| yield data | ||
|
|
||
| offset += len(data) | ||
|
|
||
| if len(data) < limit: | ||
| return | ||
|
|
||
| if max_results is not None and offset >= max_results: | ||
| return |
There was a problem hiding this comment.
I think if you rebase, you'll get this from master
closes https://linear.app/getsentry/issue/DE-970/migrate-custom-integrations-form-from-legacy-form-system --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: TkDodo <dominik.dorfmeister@sentry.io>
closes [DE-998 Migrate projectKeys/list/loaderScript.tsx from legacy form system](https://linear.app/getsentry/issue/DE-998/migrate-projectkeyslistloaderscripttsx-from-legacy-form-system) --------- Co-authored-by: Claude <noreply@anthropic.com>
Same as #115049 but just the frontend-end.
…nd update (#115052) using an empty object to prevent cache collision does no longer work because empty objects get stripped from the QueryKey. This didn't surface earlier because apiOptions and useApiQuery were using a different cache structure, but now they are unified so useReplayData and usePollReplayRecord re-use the same cache. Using the same cache _should_ be fine but here it's not because this will lead to new ReplayPlayer instances being created, as they depend on the query result. The real issue is that we re-create ReplayPlayer instances in useMemo, which needs a proper follow-up fix: https://github.com/getsentry/sentry/blob/7e47f8d9537e68e5a3f53307184ad9791549c0f6/static/app/utils/replays/hooks/useLoadReplayReader.tsx#L68-L88
…nges link (#114824) Update billing-related copy in two places in the AM checkout/cancel flows. - Renames the label displayed in plan features from 'metric alerts' to 'Metric Monitors' to match updated product terminology. - Displays '1,000' instead of 'Unlimited' for the metric detector limit on plans with an unlimited reserved value. - Removes the link to the Zendesk article about upcoming Developer plan changes from the cancel subscription banner, as it is no longer relevant. --------- Co-authored-by: Claude Sonnet 4 <noreply@example.com>
- We will need a way to get all of the results in all of the longer queries for this new pipeline - Add a function that creates an iterator over a batch of results
…115002) For large events, figuring out which attribute is the problem can be hard. The assertion helper should make this simpler.
… for release to GA (#115008) Coming out of discussion with @bcoe and @manessaraj: we'll have the default be 500. Otherwise we're ready to ship 🚀 100k is supported nicely for the current API. Adding an "include all columns" checkbox will be next, once some backend stress testing is done.
Prevent compact time-series charts from rendering crowded y-axis labels by letting ECharts hide overlapping y-axis labels. This keeps tick selection, data ranges, and label sizing in ECharts instead of adding caller-specific axis controls. Co-authored-by: Codex <noreply@openai.com>
Fetch the latest DetectorGroup ids first, then load only those groups instead of joining every candidate group before DISTINCT ON. This avoids building and sorting a wide joined Group/Project rowset that mostly gets discarded. A before/after EXPLAIN on the slow detector list page showed execution time drop from 32,400ms to 1,075ms. The wide Group/Project join went from ~99k rows to 20 and the disk sort spill went away.
Bumps [pillow](https://github.com/python-pillow/Pillow) from 12.1.1 to 12.2.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/python-pillow/Pillow/releases">pillow's releases</a>.</em></p> <blockquote> <h2>12.2.0</h2> <p><a href="https://pillow.readthedocs.io/en/stable/releasenotes/12.2.0.html">https://pillow.readthedocs.io/en/stable/releasenotes/12.2.0.html</a></p> <h2>Documentation</h2> <ul> <li>Update 12.2.0 release notes <a href="https://redirect.github.com/python-pillow/Pillow/issues/9522">#9522</a> [<a href="https://github.com/hugovk"><code>@hugovk</code></a>]</li> <li>Add loader plugins: AMOS abk, Atari Degas, 40+ more obscure formats via Netpbm <a href="https://redirect.github.com/python-pillow/Pillow/issues/9482">#9482</a> [<a href="https://github.com/bitplane"><code>@bitplane</code></a>]</li> <li>Update Python versions <a href="https://redirect.github.com/python-pillow/Pillow/issues/9515">#9515</a> [<a href="https://github.com/radarhere"><code>@radarhere</code></a>]</li> <li>Jeffrey A. Clark -> Jeffrey 'Alex' Clark <a href="https://redirect.github.com/python-pillow/Pillow/issues/9513">#9513</a> [<a href="https://github.com/aclark4life"><code>@aclark4life</code></a>]</li> <li>Add release notes for <a href="https://redirect.github.com/python-pillow/Pillow/issues/9394">#9394</a>, <a href="https://redirect.github.com/python-pillow/Pillow/issues/9419">#9419</a> and <a href="https://redirect.github.com/python-pillow/Pillow/issues/9456">#9456</a> <a href="https://redirect.github.com/python-pillow/Pillow/issues/9467">#9467</a> [<a href="https://github.com/radarhere"><code>@radarhere</code></a>]</li> <li>Add Amiga Workbench .info loader to 3rd party plugins list <a href="https://redirect.github.com/python-pillow/Pillow/issues/9459">#9459</a> [<a href="https://github.com/bitplane"><code>@bitplane</code></a>]</li> <li>Merge PFM documentation into PPM <a href="https://redirect.github.com/python-pillow/Pillow/issues/9434">#9434</a> [<a href="https://github.com/radarhere"><code>@radarhere</code></a>]</li> <li>Update macOS tested Pillow versions <a href="https://redirect.github.com/python-pillow/Pillow/issues/9431">#9431</a> [<a href="https://github.com/radarhere"><code>@radarhere</code></a>]</li> <li>Fix CVE number <a href="https://redirect.github.com/python-pillow/Pillow/issues/9430">#9430</a> [<a href="https://github.com/hugovk"><code>@hugovk</code></a>]</li> </ul> <h2>Dependencies</h2> <ul> <li>Update xz to 5.8.3 <a href="https://redirect.github.com/python-pillow/Pillow/issues/9523">#9523</a> [<a href="https://github.com/radarhere"><code>@radarhere</code></a>]</li> <li>Update libjpeg-turbo to 3.1.4.1 <a href="https://redirect.github.com/python-pillow/Pillow/issues/9507">#9507</a> [<a href="https://github.com/radarhere"><code>@radarhere</code></a>]</li> <li>Update libpng to 1.6.56 <a href="https://redirect.github.com/python-pillow/Pillow/issues/9499">#9499</a> [<a href="https://github.com/radarhere"><code>@radarhere</code></a>]</li> <li>Update freetype to 2.14.3 <a href="https://redirect.github.com/python-pillow/Pillow/issues/9485">#9485</a> [<a href="https://github.com/radarhere"><code>@radarhere</code></a>]</li> <li>Updated libavif to 1.4.1 <a href="https://redirect.github.com/python-pillow/Pillow/issues/9479">#9479</a> [<a href="https://github.com/radarhere"><code>@radarhere</code></a>]</li> <li>Updated harfbuzz to 13.2.1 <a href="https://redirect.github.com/python-pillow/Pillow/issues/9461">#9461</a> [<a href="https://github.com/radarhere"><code>@radarhere</code></a>]</li> <li>Update Ghostscript to 10.7.0 <a href="https://redirect.github.com/python-pillow/Pillow/issues/9469">#9469</a> [<a href="https://github.com/radarhere"><code>@radarhere</code></a>]</li> <li>Update harfbuzz to 13.0.1 <a href="https://redirect.github.com/python-pillow/Pillow/issues/9453">#9453</a> [<a href="https://github.com/radarhere"><code>@radarhere</code></a>]</li> <li>Update libavif to 1.4.0 <a href="https://redirect.github.com/python-pillow/Pillow/issues/9460">#9460</a> [<a href="https://github.com/radarhere"><code>@radarhere</code></a>]</li> <li>Update freetype to 2.14.2 <a href="https://redirect.github.com/python-pillow/Pillow/issues/9449">#9449</a> [<a href="https://github.com/radarhere"><code>@radarhere</code></a>]</li> <li>Update actions/download-artifact action to v8 <a href="https://redirect.github.com/python-pillow/Pillow/issues/9451">#9451</a> [@<a href="https://github.com/apps/renovate">renovate[bot]</a>]</li> <li>Updated libpng to 1.6.55 <a href="https://redirect.github.com/python-pillow/Pillow/issues/9425">#9425</a> [<a href="https://github.com/radarhere"><code>@radarhere</code></a>]</li> </ul> <h2>Testing</h2> <ul> <li>Cleanup .spider extension in the same test where it is added <a href="https://redirect.github.com/python-pillow/Pillow/issues/9517">#9517</a> [<a href="https://github.com/radarhere"><code>@radarhere</code></a>]</li> <li>Run tests in parallel via tox for 3.5x speedup <a href="https://redirect.github.com/python-pillow/Pillow/issues/9516">#9516</a> [<a href="https://github.com/hugovk"><code>@hugovk</code></a>]</li> <li>Enable colour in CI logs <a href="https://redirect.github.com/python-pillow/Pillow/issues/9486">#9486</a> [<a href="https://github.com/hugovk"><code>@hugovk</code></a>]</li> <li>Update Ghostscript to 10.7.0 <a href="https://redirect.github.com/python-pillow/Pillow/issues/9469">#9469</a> [<a href="https://github.com/radarhere"><code>@radarhere</code></a>]</li> <li>Simplify TGA test code <a href="https://redirect.github.com/python-pillow/Pillow/issues/9477">#9477</a> [<a href="https://github.com/radarhere"><code>@radarhere</code></a>]</li> <li>Update tests to check for ValueError when encoding an empty image <a href="https://redirect.github.com/python-pillow/Pillow/issues/9464">#9464</a> [<a href="https://github.com/radarhere"><code>@radarhere</code></a>]</li> <li>Upgrade CI from <code>macos-15-intel</code> to <code>macos-26-intel</code> <a href="https://redirect.github.com/python-pillow/Pillow/issues/9454">#9454</a> [<a href="https://github.com/hugovk"><code>@hugovk</code></a>]</li> <li>Add check-case-conflict hook <a href="https://redirect.github.com/python-pillow/Pillow/issues/9446">#9446</a> [<a href="https://github.com/radarhere"><code>@radarhere</code></a>]</li> <li>Specify platform when pulling docker image <a href="https://redirect.github.com/python-pillow/Pillow/issues/9440">#9440</a> [<a href="https://github.com/radarhere"><code>@radarhere</code></a>]</li> <li>GHA: Cache libavif and webp builds for Ubuntu <a href="https://redirect.github.com/python-pillow/Pillow/issues/9437">#9437</a> [<a href="https://github.com/hugovk"><code>@hugovk</code></a>]</li> <li>Update macOS tested Pillow versions <a href="https://redirect.github.com/python-pillow/Pillow/issues/9431">#9431</a> [<a href="https://github.com/radarhere"><code>@radarhere</code></a>]</li> </ul> <h2>Other changes</h2> <ul> <li>Check calloc return value <a href="https://redirect.github.com/python-pillow/Pillow/issues/9527">#9527</a> [<a href="https://github.com/radarhere"><code>@radarhere</code></a>]</li> <li>Check all allocs in the Arrow tree <a href="https://redirect.github.com/python-pillow/Pillow/issues/9488">#9488</a> [<a href="https://github.com/wiredfool"><code>@wiredfool</code></a>]</li> <li>Reject non-numeric elements inside list coords <a href="https://redirect.github.com/python-pillow/Pillow/issues/9526">#9526</a> [<a href="https://github.com/hugovk"><code>@hugovk</code></a>]</li> <li>Move variable declaration inside define <a href="https://redirect.github.com/python-pillow/Pillow/issues/9525">#9525</a> [<a href="https://github.com/radarhere"><code>@radarhere</code></a>]</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/python-pillow/Pillow/commit/3c41c095064200a02672d89cc5ff629eaf4b0d4f"><code>3c41c09</code></a> 12.2.0 version bump</li> <li><a href="https://github.com/python-pillow/Pillow/commit/cdaa29eb520291c4f1fb50fb71ae46502d41e460"><code>cdaa29e</code></a> Check calloc return value (<a href="https://redirect.github.com/python-pillow/Pillow/issues/9527">#9527</a>)</li> <li><a href="https://github.com/python-pillow/Pillow/commit/585b2f5a780722c8a5bfffb3a40f7f42e8a205be"><code>585b2f5</code></a> Check calloc return value</li> <li><a href="https://github.com/python-pillow/Pillow/commit/ecf011ea15991d4cebacd946e58270cc30b0f2c1"><code>ecf011e</code></a> Check all allocs in the Arrow tree (<a href="https://redirect.github.com/python-pillow/Pillow/issues/9488">#9488</a>)</li> <li><a href="https://github.com/python-pillow/Pillow/commit/cf6de8ca9b23e714aa5310e1c791eda66fc0b670"><code>cf6de8c</code></a> Reject non-numeric elements inside list coords (<a href="https://redirect.github.com/python-pillow/Pillow/issues/9526">#9526</a>)</li> <li><a href="https://github.com/python-pillow/Pillow/commit/ffdcede6516b28d9667c92929854023d17048b64"><code>ffdcede</code></a> Update 12.2.0 release notes (<a href="https://redirect.github.com/python-pillow/Pillow/issues/9522">#9522</a>)</li> <li><a href="https://github.com/python-pillow/Pillow/commit/7929d7760fe5a307ba5ae6eabdf70ae4486b147c"><code>7929d77</code></a> Added security release notes (<a href="https://redirect.github.com/python-pillow/Pillow/issues/149">#149</a>)</li> <li><a href="https://github.com/python-pillow/Pillow/commit/c4f7aa5dfb4dbd1242978ac235e01b9934ec6d3c"><code>c4f7aa5</code></a> Added security release notes</li> <li><a href="https://github.com/python-pillow/Pillow/commit/22cdb5f2e4b15250c06563b1124ac1667342712f"><code>22cdb5f</code></a> Move variable declaration inside define (<a href="https://redirect.github.com/python-pillow/Pillow/issues/9525">#9525</a>)</li> <li><a href="https://github.com/python-pillow/Pillow/commit/fc15b3b01899408ec989d7804c5283e13802d057"><code>fc15b3b</code></a> Resize tall images vertically first (<a href="https://redirect.github.com/python-pillow/Pillow/issues/9524">#9524</a>)</li> <li>Additional commits viewable in <a href="https://github.com/python-pillow/Pillow/compare/12.1.1...12.2.0">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/getsentry/sentry/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
|
🚨 Warning: This pull request contains Frontend and Backend changes! It's discouraged to make changes to Sentry's Frontend and Backend in a single pull request. The Frontend and Backend are not atomically deployed. If the changes are interdependent of each other, they must be separated into two pull requests and be made forward or backwards compatible, such that the Backend or Frontend can be safely deployed independently. Have questions? Please ask in the |
| if use_log_scale: | ||
| lower_bound = bucket_ranges.min_value + y_log_scale ** ( | ||
| current_bucket * bucket_size | ||
| ) | ||
| upper_bound = bucket_ranges.min_value + y_log_scale ** ( | ||
| (current_bucket + 1) * bucket_size | ||
| ) |
There was a problem hiding this comment.
Bug: The logarithmic scale bucketing logic incorrectly calculates the first bucket's lower bound when min_value is not zero, causing data points near the minimum to be excluded.
Severity: MEDIUM
Suggested Fix
Revise the formula for calculating bucket boundaries in a logarithmic scale. Instead of adding an offset to the min_value, the calculation should correctly map the logarithmic range to the data's actual range, ensuring the first bucket starts at min_value.
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.
Location: src/sentry/api/endpoints/organization_events_heatmap.py#L177-L183
Potential issue: The formula for calculating logarithmic bucket boundaries, `lower_bound
= bucket_ranges.min_value + y_log_scale ** (current_bucket * bucket_size)`, is flawed.
When `current_bucket` is 0, the lower bound becomes `min_value + 1`. This incorrectly
excludes any data points that fall within the range `[min_value, min_value + 1)`. This
issue only manifests when the queried data has a minimum value greater than zero and a
logarithmic scale is used, as existing tests only cover cases with a minimum of zero.
Did we get this right? 👍 / 👎 to inform future reviews.
| else: | ||
| continue | ||
| break | ||
|
|
||
| return [ | ||
| { | ||
| "org_id": config.organization.id, | ||
| "project_id": project_id, | ||
| "transaction_counts": project_volumes.transaction_counts, | ||
| "total_num_transactions": project_volumes.total_num_transactions, | ||
| "total_num_classes": len(project_volumes.transaction_counts), |
There was a problem hiding this comment.
Bug: When max_transactions is hit, get_eap_transaction_volumes returns a partial sum for total_num_transactions, violating the expected contract and causing incorrect sampling rate calculations.
Severity: MEDIUM
Suggested Fix
When the max_transactions limit is reached and the loop terminates early, the function should set total_num_transactions to None instead of returning a partial sum. This correctly signals to downstream consumers that the total is unknown, preserving the established data contract.
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.
Location: src/sentry/dynamic_sampling/per_org/tasks/queries.py#L157-L186
Potential issue: The `get_eap_transaction_volumes` function can return a partial sum for
`total_num_transactions` when the `max_transactions` limit is reached. The downstream
`TransactionsRebalancingModel` expects `total_num_transactions` to be either the true
total for the project or `None` if the total is unknown. Providing a partial sum as if
it were the true total will cause the model to perform budget calculations with an
incorrect, underestimated total, leading to inaccurate dynamic sampling rates for the
project.
Did we get this right? 👍 / 👎 to inform future reviews.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 165e4e6. Configure here.
| if y_log_scale == 1: | ||
| raise ParseError("logScale cannot be 1") | ||
| else: | ||
| y_log_scale = False |
There was a problem hiding this comment.
Log scale base 0 silently accepted without validation
Low Severity
The yLogScale parameter validation rejects base 1 (ParseError) but accepts base 0, which is also mathematically undefined. When yLogScale=0 is passed, y_log_scale is set to 0, use_log_scale evaluates to False (since bool(0) is False), and the endpoint silently falls back to linear scale. A user explicitly requesting log scale with base 0 gets no error, unlike base 1 which is properly rejected. The validation at y_log_scale == 1 needs a companion check for y_log_scale <= 0 (or < 2 depending on desired minimum base).
Reviewed by Cursor Bugbot for commit 165e4e6. Configure here.


Closes https://linear.app/getsentry/issue/TET-2306/create-transaction-volume-query-for-eap