query_tombstone_page_limit applies to unpaged queries #17241
Comments
denesb
added a commit
to denesb/scylla
that referenced
this issue
Feb 9, 2024
The reason we introduced the tombstone-limit (query_tombstone_page_limit), was to allow paged queries to return incomplete/empty pages in the face of large tombstone spans. This works by cutting the page after the tombstone-limit amount of tombstones were processed. If the read is unpaged, it is killed instead. This was a mistake. First, it doesn't really make sense, the reason we introduced the tombstone limit, was to allow paged queries to process large tombstone-spans without timing out. It does not help unpaged queries. Furthermore, the tombstone-limit can kill internal queries done on behalf of user queries, because all our internal queries are unpaged. This can cause denial of service. So in this patch we disable the tombstone-limit for unpaged queries altogether, they are allowed to continue even after having processed the configured limit of tombstones. Fixes: scylladb#17241
denesb
added
backport/5.2
mykaul
added
Field-Tier1
denesb
added a commit
that referenced
this issue
Feb 15, 2024
The reason we introduced the tombstone-limit (query_tombstone_page_limit), was to allow paged queries to return incomplete/empty pages in the face of large tombstone spans. This works by cutting the page after the tombstone-limit amount of tombstones were processed. If the read is unpaged, it is killed instead. This was a mistake. First, it doesn't really make sense, the reason we introduced the tombstone limit, was to allow paged queries to process large tombstone-spans without timing out. It does not help unpaged queries. Furthermore, the tombstone-limit can kill internal queries done on behalf of user queries, because all our internal queries are unpaged. This can cause denial of service. So in this patch we disable the tombstone-limit for unpaged queries altogether, they are allowed to continue even after having processed the configured limit of tombstones. Fixes: #17241 Closes #17242 (cherry picked from commit f068d1a)
denesb
added a commit
that referenced
this issue
Feb 15, 2024
The reason we introduced the tombstone-limit (query_tombstone_page_limit), was to allow paged queries to return incomplete/empty pages in the face of large tombstone spans. This works by cutting the page after the tombstone-limit amount of tombstones were processed. If the read is unpaged, it is killed instead. This was a mistake. First, it doesn't really make sense, the reason we introduced the tombstone limit, was to allow paged queries to process large tombstone-spans without timing out. It does not help unpaged queries. Furthermore, the tombstone-limit can kill internal queries done on behalf of user queries, because all our internal queries are unpaged. This can cause denial of service. So in this patch we disable the tombstone-limit for unpaged queries altogether, they are allowed to continue even after having processed the configured limit of tombstones. Fixes: #17241 Closes #17242 (cherry picked from commit f068d1a)
|
Backported to 5.4 and 5.2. |
denesb
removed
Backport candidate
backport/5.2
dgarcia360
pushed a commit
to dgarcia360/scylla
that referenced
this issue
Apr 30, 2024
The reason we introduced the tombstone-limit (query_tombstone_page_limit), was to allow paged queries to return incomplete/empty pages in the face of large tombstone spans. This works by cutting the page after the tombstone-limit amount of tombstones were processed. If the read is unpaged, it is killed instead. This was a mistake. First, it doesn't really make sense, the reason we introduced the tombstone limit, was to allow paged queries to process large tombstone-spans without timing out. It does not help unpaged queries. Furthermore, the tombstone-limit can kill internal queries done on behalf of user queries, because all our internal queries are unpaged. This can cause denial of service. So in this patch we disable the tombstone-limit for unpaged queries altogether, they are allowed to continue even after having processed the configured limit of tombstones. Fixes: scylladb#17241 Closes scylladb#17242
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The appropriate query-tombstone-page-limit (tombstone-limit) is selected based on the classification of the query, just like we do with the semaphores and the page limit. System queries have an infinite tombstone limit. Yet internal queries executed on behalf of user queries use the user limit. It was observed that in some cases this lead to internal queries into
system_schematables failing due to the tombstone limit affecting unpaged queries (internal queries are all unpaged).The best solution to prevent internal queries from being affected is to make the tombstone-limit not affect unpaged queries at all. Having the tombstone limit affect unpaged queries doesn't make sense in the first place. The reason this limit was introduced was to prevent paged queries from timing out. It doesn't help unpaged queries, if anything this is counter productive, because unpaged queries which worked before, will fail with the tombstone limit and raising the limit can break paged queries.
The text was updated successfully, but these errors were encountered: