Skip to content

ref(aci): Remove a try/except block for more detailed errors#111439

Merged
saponifi3d merged 3 commits into
masterfrom
jcallender/aci/proxy-seer-errors
Mar 27, 2026
Merged

ref(aci): Remove a try/except block for more detailed errors#111439
saponifi3d merged 3 commits into
masterfrom
jcallender/aci/proxy-seer-errors

Conversation

@saponifi3d
Copy link
Copy Markdown
Contributor

@saponifi3d saponifi3d commented Mar 24, 2026

Description

When investigating an issue, it was difficult to determine the source of the issue because we hit the exception block and lost which of the ValidationError was bubbling up.

This will just allow the more specific validation error to bubble up and reformat them to all be ValidationErrors that can be handled in the API.

The code still catches the exception and retriggers it to ensure the control flow is consistent and the validation error types are correct.

@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Mar 24, 2026
@github-actions
Copy link
Copy Markdown
Contributor

Backend Test Failures

Failures on ebb9c06 in this run:

tests/sentry/incidents/endpoints/validators/test_validators.py::TestMetricAlertsUpdateDetectorValidator::test_anomaly_detection__send_historical_data_snuba_update_failslog
src/sentry/seer/anomaly_detection/store_data_workflow_engine.py:242: in send_historical_data_to_seer
    response = make_store_data_request(body, viewer_context=viewer_context)
/opt/hostedtoolcache/Python/3.13.1/x64/lib/python3.13/unittest/mock.py:1167: in __call__
    return self._mock_call(*args, **kwargs)
/opt/hostedtoolcache/Python/3.13.1/x64/lib/python3.13/unittest/mock.py:1171: in _mock_call
    return self._execute_mock_call(*args, **kwargs)
/opt/hostedtoolcache/Python/3.13.1/x64/lib/python3.13/unittest/mock.py:1226: in _execute_mock_call
    raise effect
E   urllib3.exceptions.TimeoutError

During handling of the above exception, another exception occurred:
src/sentry/seer/anomaly_detection/store_data_workflow_engine.py:155: in handle_send_historical_data_to_seer
    send_historical_data_to_seer(
src/sentry/seer/anomaly_detection/store_data_workflow_engine.py:252: in send_historical_data_to_seer
    raise TimeoutError
E   urllib3.exceptions.TimeoutError

During handling of the above exception, another exception occurred:
src/sentry/seer/anomaly_detection/store_data_workflow_engine.py:109: in update_detector_data
    handle_send_historical_data_to_seer(
src/sentry/seer/anomaly_detection/store_data_workflow_engine.py:164: in handle_send_historical_data_to_seer
    raise TimeoutError(f"Failed to send data to Seer - cannot {method} detector.")
E   urllib3.exceptions.TimeoutError: Failed to send data to Seer - cannot update detector.

During handling of the above exception, another exception occurred:
tests/sentry/incidents/endpoints/validators/test_validators.py:1362: in test_anomaly_detection__send_historical_data_snuba_update_fails
    update_validator.save()
.venv/lib/python3.13/site-packages/rest_framework/serializers.py:205: in save
    self.instance = self.update(self.instance, validated_data)
src/sentry/incidents/metric_issue_detector.py:397: in update
    self.update_data_source(instance, data_source, seer_updated)
src/sentry/incidents/metric_issue_detector.py:326: in update_data_source
    update_detector_data(instance, validated_data_source)
src/sentry/seer/anomaly_detection/store_data_workflow_engine.py:119: in update_detector_data
    raise ValidationError("Timed out sending data to Seer, unable to update detector")
E   django.core.exceptions.ValidationError: ['Timed out sending data to Seer, unable to update detector']
tests/sentry/incidents/endpoints/validators/test_validators.py::TestMetricAlertsUpdateDetectorValidator::test_anomaly_detection__send_historical_data_update_failslog
src/sentry/seer/anomaly_detection/store_data_workflow_engine.py:242: in send_historical_data_to_seer
    response = make_store_data_request(body, viewer_context=viewer_context)
/opt/hostedtoolcache/Python/3.13.1/x64/lib/python3.13/unittest/mock.py:1167: in __call__
    return self._mock_call(*args, **kwargs)
/opt/hostedtoolcache/Python/3.13.1/x64/lib/python3.13/unittest/mock.py:1171: in _mock_call
    return self._execute_mock_call(*args, **kwargs)
/opt/hostedtoolcache/Python/3.13.1/x64/lib/python3.13/unittest/mock.py:1226: in _execute_mock_call
    raise effect
E   urllib3.exceptions.TimeoutError

During handling of the above exception, another exception occurred:
src/sentry/seer/anomaly_detection/store_data_workflow_engine.py:155: in handle_send_historical_data_to_seer
    send_historical_data_to_seer(
src/sentry/seer/anomaly_detection/store_data_workflow_engine.py:252: in send_historical_data_to_seer
    raise TimeoutError
E   urllib3.exceptions.TimeoutError

During handling of the above exception, another exception occurred:
src/sentry/seer/anomaly_detection/store_data_workflow_engine.py:109: in update_detector_data
    handle_send_historical_data_to_seer(
src/sentry/seer/anomaly_detection/store_data_workflow_engine.py:164: in handle_send_historical_data_to_seer
    raise TimeoutError(f"Failed to send data to Seer - cannot {method} detector.")
E   urllib3.exceptions.TimeoutError: Failed to send data to Seer - cannot update detector.

During handling of the above exception, another exception occurred:
tests/sentry/incidents/endpoints/validators/test_validators.py:1309: in test_anomaly_detection__send_historical_data_update_fails
    update_validator.save()
.venv/lib/python3.13/site-packages/rest_framework/serializers.py:205: in save
    self.instance = self.update(self.instance, validated_data)
src/sentry/incidents/metric_issue_detector.py:375: in update
    seer_updated = self.update_anomaly_detection(instance, validated_data)
src/sentry/incidents/metric_issue_detector.py:361: in update_anomaly_detection
    update_detector_data(instance, validated_data)
src/sentry/seer/anomaly_detection/store_data_workflow_engine.py:119: in update_detector_data
    raise ValidationError("Timed out sending data to Seer, unable to update detector")
E   django.core.exceptions.ValidationError: ['Timed out sending data to Seer, unable to update detector']

…the ValiditonError type and the re-raises were used as flow control.
@getsentry getsentry deleted a comment from sentry-warden Bot Mar 24, 2026
@saponifi3d saponifi3d force-pushed the jcallender/aci/proxy-seer-errors branch from 0e179b7 to b72343e Compare March 24, 2026 19:44
@saponifi3d saponifi3d marked this pull request as ready for review March 24, 2026 19:46
@saponifi3d saponifi3d requested a review from a team as a code owner March 24, 2026 19:46
Comment thread src/sentry/incidents/metric_issue_detector.py Outdated
Comment thread src/sentry/incidents/metric_issue_detector.py Outdated
Comment thread src/sentry/incidents/metric_issue_detector.py Dismissed
Comment thread src/sentry/incidents/metric_issue_detector.py Dismissed
Copy link
Copy Markdown
Member

@ceorourke ceorourke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm assuming we're ok with the information exposure risk

@saponifi3d
Copy link
Copy Markdown
Contributor Author

@ceorourke if you have any context on things that might be getting exposed outside of those expected cases, that'd be helpful! It seemed limited to me: https://github.com/getsentry/sentry/blob/master/src/sentry/seer/anomaly_detection/store_data_workflow_engine.py#L118-L125 -- but not sure if i'm missing anything else?

I guess it could potentially throw an exception on any line, and we probably wouldn't want to expose it? (I had also tried to let them bubble up from the original, but they're different validation errors, so the API won't create them as proper validation errors. 😭)

Comment on lines +331 to +333
except (TimeoutError, MaxRetryError, ParseError, ValidationError) as e:
# don't update the snuba query if we failed to send data to Seer
raise serializers.ValidationError(
"Failed to send data to Seer, cannot update detector"
)
raise serializers.ValidationError(str(e))
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: The narrowed exception handling in update_data_source and update_anomaly_detection no longer catches DetectorException, which will cause an unhandled exception and a 500 error.
Severity: CRITICAL

Suggested Fix

Add DetectorException to the list of caught exceptions in the try...except blocks within update_data_source() and update_anomaly_detection() to ensure it is handled correctly and converted to a serializers.ValidationError.

Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.

Location: src/sentry/incidents/metric_issue_detector.py#L331-L333

Potential issue: The exception handling in `update_data_source()` and
`update_anomaly_detection()` was narrowed to catch a specific list of exceptions, but
this list omits `DetectorException`. This exception is raised by the
`_fetch_related_models()` function when related data for a detector is missing or
inconsistent. Because `DetectorException` is no longer caught, it will propagate up as
an unhandled exception, causing the API endpoint to return a 500 error instead of a
graceful `serializers.ValidationError` as it did previously. This can occur during
normal API operations if detector data is in an inconsistent state.

@saponifi3d saponifi3d merged commit 89ef2bd into master Mar 27, 2026
63 checks passed
@saponifi3d saponifi3d deleted the jcallender/aci/proxy-seer-errors branch March 27, 2026 23:52
@github-actions github-actions Bot locked and limited conversation to collaborators Apr 12, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants