-
Notifications
You must be signed in to change notification settings - Fork 24.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RestGetRollupCapsAction and RestGetRollupIndexCapsAction invoke expensive GetRollupCapsAction on transport threads #92179
Comments
Pinging @elastic/es-analytics-geo (Team:Analytics) |
With the 8.7 release of Elasticsearch, we have made a new downsampling capability associated with the new time series datastreams functionality generally available (GA). This capability was in tech preview in ILM since 8.5. Downsampling provides a method to reduce the footprint of your time series data by storing it at reduced granularity. The downsampling process rolls up documents within a fixed time interval into a single summary document. Each summary document includes statistical representations of the original data: the min, max, sum, value_count, and average for each metric. Data stream time series dimensions are stored unchanged. Downsampling is superior to rollup because:
Because of the introduction of this new capability, we are deprecating the rollups functionality, which never left the Tech Preview/Experimental status, in favor of downsampling and thus we are closing this issue. We encourage you to migrate your solution to downsampling and take advantage of the new TSDB functionality. |
I think we need to keep this open, it's a pretty bad bug (it can have a severe performance impact on unrelated operations and in extreme cases will destabilise the cluster). Deprecating the feature isn't really enough to stop folks from hitting this bug, and we will live with this deprecated feature for a very long time yet. Also the fix should be pretty straightforward, just fork the expensive computation to a better threadpool manually. |
Isn't this just a matter of replacing this
with something like this?
(for both) |
Yeah something like that, plus some calls to |
FWIW we've worked around #97916 in various other places in the transport layer rather than the REST layer as follows: Lines 84 to 85 in 5ed31c3
... Lines 101 to 107 in 5ed31c3
Doing in the REST layer would also be ok, but this is the established workaround pattern. Either way, please include |
These APIs are potentially harmful to large clusters prior to 8.10, see elastic/elasticsearch#92179.
These APIs are potentially harmful to large clusters prior to 8.10, see elastic/elasticsearch#92179.
This is a known issue around rest actions. They execute transport actions in the current thread instead of the thread that the action is registered for with the transport service in code like:
This is normally not an issue since the expensive actions that also have a REST endpoint are
TransportMasterNodeAction
s. In the case ofRestGetRollupCapsAction
andRestGetRollupIndexCapsAction
which aren't, this breaks the fix in #89803 if the action is used via the REST layer.-> I think until we figure out if we want to change the way threading works with the client in the above code, we should manually fork to the management pool in these actions to fix the slow action blocking transport threads.
The text was updated successfully, but these errors were encountered: