Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add more context cancellation check at evaluation time #13437

Merged
merged 1 commit into from Jan 22, 2024

Conversation

yeya24
Copy link
Contributor

@yeya24 yeya24 commented Jan 21, 2024

We found out that even if the query already timed out and context canceled, the query is still running at evaluation phase.

This pr adds more context cancellation check when evaluating parser.Call and parser.VectorSelector. We already have such kind of per series ctx cancellation check here.

Example trace below. The query itself timed out after 1 min, but promqlInnerEval eval parser.Call ran for 3.4 min.

image

Signed-off-by: Ben Ye <benye@amazon.com>
Copy link
Member

@bboreham bboreham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems reasonable

Copy link
Contributor

@aknuds1 aknuds1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@bboreham bboreham merged commit c095ba2 into prometheus:main Jan 22, 2024
24 checks passed
@yeya24 yeya24 deleted the check-ctx-eval-series branch January 22, 2024 18:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants