-
Notifications
You must be signed in to change notification settings - Fork 8.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[APM] Service map request fails when searching outside of the boost window #179229
Comments
Pinging @elastic/apm-ui (Team:APM) |
Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services) |
does this always fail outside the boost window? for example, does it consistently fail one day after the boost window or does it fail 12 days after the boost window? does the amount of data/complexity of the service map matter? |
@chrisdistasio having the data that I specified in a precondition, it was always failing when searching outside the boost window. The amount of data matter, especially the number of dependencies. |
@smith can we have someone assigned to this to do some additional investigation to determine whether this is directly related to some issue searching outside boost window? is it possible to quantify the number of services and dependencies that appear to be a threshold for causing the issue? trying to get a better understanding of the severity as it relates to boost window. for example, i have trouble getting a response in qa for 15 mins - https://keepserverless-qa-oblt-ba6694.kb.eu-west-1.aws.qa.elastic.cloud/app/apm/service-map?comparisonEnabled=false&environment=ENVIRONMENT_ALL&kuery=&offset=1d&rangeFrom=now-15m&rangeTo=now&serviceGroup= |
@chrisdistasio this is prioritized in our refining queue on our internal backlog, so the first steps are determining under which conditions this happens. |
@ablnk Is it possible to access the environment where the error is happening? I'd like to have a closer look to the logs. |
I'm getting the 502 in QA deployment with ~1800 services and ~170 dependencies, within 15m range, which I believe is still within the 7d boost window. The map is somewhat complex (5m range) I managed to crash the server when I used a 1d range. Looks like it's related to the amount of data affecting it on serverless. Searches outside the boost window could potentialize the likelihood of this error happening. |
The solution for this might be related to https://github.com/elastic/observability-serverless/issues/18 and #181790 |
@chrisdistasio @paulb-elastic Are we aiming for the short term or the long term solution here? The short term being we keep the scripted metric aggregation, but make it faster. The tradeoff here is the service map's accuracy or completeness as we could lose some connections and discovered services in the timerange, though we already do some sampling of the trace ids we pass in so this exists to an extent already, I believe. The long term is removing the scripted aggregation and moving this processing somewhere else, not sure where. |
The steer I've been given is to focus on the shorter term solution for now (although happy to hear @chrisdistasio's opinion too). |
Thanks @paulb-elastic. I've been having some conversation in another issue, and will get started on optimizing the scripted metric agg and running some tests. Then figure out what this limit number should be which is the more challenging bit as Dario mentioned. |
I had a chat with Dario the week before last and we aligned on a strategy to avoid OOMs. optimization strategy
tradeoffs
issues to resolve / dependencies
|
Thanks for the update @neptunian
Are you ok proceeding with this, and not blocked? |
@paulb-elastic Yes, not blocked. I'll be focusing on it this week. |
I had a chat with @nik9000 about the strategy above. We came to the conclusion that limiting in the reduce phase likely won't resolve the error. Once a large number of documents have arrived at the reduce phase there are already memory issues whether or not we limit adding them to the Instead, I think we should tweak the limits with the levers we already have in place such as decreasing terminate_after from 100k per shard, and less max traces (currently 1,000); One thing I've found out after talking to @crespocarlos about his PR #183802, is the synthtrace data that exists in serverless where we are testing is not realistic data. One reason why things are so slow and we might be getting OOM errors is that we are actually running the synthtrace scenario for OOMs: Here is a response i get on the edge cluster, some of the docs are from OOM scenario: ![]() This scenario creates very long traces (not common or realistic) in order to trigger an OOM. Sometimes when I can get it to load I have a neverending long trace. This is the service map below. You cannot see the services because I've zoomed out to show how long this trace can be. ![]() Given the fact there is a recursive function in the reduce_phase we can end up with an OOM in there because of that or a stack overflow due to the depth of the trace. I think we should verify the data we are using to test with is realistic and not broken before pulling any levers. It's also very difficult to test otherwise. |
@neptunian fwiw, you can count the number of documents by executing a request that only counts the documents, but does not execute the aggregation. For the second request, the filter cache will kick in and it will run the aggregation on the collected documents. Unfortunately that is a hard stop, and you cannot partially process traces with that strategy. You could chunk up the documents differently until you get a number of documents that make sense, but it's probably more sensible to nudge the user to tweak these settings if possible (?). Re:
this is the whole point no? It's not about realistic data, it's about making sure that no data at all can trigger an OOM. |
@dgieselaar Do you mean the settings for
I guess I'm assuming that if we were trying to make sure no data could trigger an OOM we wouldn't have recursive functions in the service map and waterfall that don't have any safeguards to account for a trace being that long. Unless that is just an oversight. When I run the service_map_oom scenario I typically get a run time error in kibana logs caused by |
yes (or any other settings we add). I think the fewer settings we have the better, but not sure what is practical.
When you say "service map" and "waterfall" are you talking about browser code? If so, it is something we have not optimized for. This data (spans or descendants that refer to each other as parents) is corrupted, and we can't guard against every imaginable form of data corruption. However, an OOM for an Elasticsearch node has much more devastating consequences (like paging Nik at night).
If I understand what you are saying correctly, is that we run the risk of making things slower for the common case? I think that is an acceptable trade-off. Ideally we gain some ground elsewhere but that is out of scope. This is just us making sure we don't take down an Elasticsearch node. |
Perhaps the fact that synthtrace is currently generating unrealistic data (with spans having their parent spans as children) makes the effort of optimizing the services map harder than it should be. The fact that Sandy is getting a stack_overflow_error could be one symptom of that. The non-unique transaction ids could also lead to data not being deterministic because, at any given time, a transaction from a service created by a scenario could become either a parent or child span from a service created by another scenario. For instance, service-8 can be created by both the other_bucket_group and services_map_oom scenarios. However, I believe none of them is configured to create a service map like the one below: ![]() In the worst-case scenario, this could lead to a cyclic dependency. |
Let's separate a few things:
|
100% agree. Just thinking here that if my PR gets merged and oblt cluters data is erased, we might lose the circumstances that are causing the OOMs I'd also suggest refactoring the recursive function in the |
We shouldn't - if you run this script in isolation (so no chance of collision because of different synthtrace processes) it should successfully generate an OOM. But let's verify.
Ah, I don't think it's recursive. It's basically mapping over the services in reverse order (e.g. the deepest service first), and then creating a downstream span that has the upstream span as a child. I think... :) |
Also we should exclude the Service Map OOM scenario from the oblt clusters 😅 |
Thanks @dgieselaar . Unfortunately @crespocarlos changes didn't resolve it, but if I just remove the recursive call ( For reference, this is the kibana stack overflow error: error
|
@dgieselaar Similar to this idea, since the query returns the shards the query ran on, I thought I could try getting the number of shards in one query ( I was able to disable parallelization on elasticsearch using an undocumented internal setting and verified that the Instead of trying to depend on |
Great finds! Very interesting stuff. Re: terminate_after, I think it's fine that for a small number it's inaccurate, but if We need to be careful with dividing no of shards by doc count, because data is not guaranteed to be unevenly distributed, e.g. one shard might contain 99% of the data. As data streams are partitioned by service this is not an unlikely scenario. |
I see no variation for scripted metric agg count with size: 0, terminate_after: 100
Response:
scripted metric agg count with size: 1, terminate_after: 100
Response:
So it seems
Yea, or the user could have partitioned their data streams via various namespaces. I think this is a tradeoff. I'm thinking in a real world scenario there probably won't be a huge amount of matching docs from 50 traces(?) so it would be unlikely to go over the max per shard limit unless it was a very large number of shards. It does seem like a safe way to avoid the possibility of too many docs being sent to the coordinating node, though. sidenote: I said previously "I think disabling parallelization should help here because we could get an accurate shard count and divide that by our max total doc count." I don't think this is true since whether it's on or off it returns the same docs, so I'm not sure it would matter if it was off. I spoke with @nik9000 about this and he'd rather have it off to be safe rather than some performance loss we might have, so that PR was merged. |
TIL! I think setting
Oh, I'm not sure what you mean then. Can you give some examples with numbers? |
I opened this PR a couple days ago #186417. I added some testing directions to test locally with smaller amounts of data. This strategy involves dividing by shards to limit the amount of total documents that can be processed. Depending on how evenly the user's data is distributed this could mean we are not able to match as many docs if some users have mostly empty shards or unevenly distributed data which means a potentially less accurate service map. This is a tradeoff. I wanted to go over some issues still to be resolved / noted:
|
@neptunian what is the point of setting a max trace depth? |
@dgieselaar I don't think it would be as much an issue now since we are limiting the docs. But I was getting stack overflow errors and crashing kibana using synthtrace data if I didn't add some limitation. I don't know if this is the right way to go about that handling that scenario though. I've asked @crespocarlos to look into that and he said there might be some better optimizations. |
…6417) ## Summary #179229 This PR addresses the need to limit the amount of data that the scripted metric aggregation in the service map processes in one request which can lead to timeouts and OOMs, often resulting in the user seeing [parent circuit breaker](https://www.elastic.co/guide/en/elasticsearch/reference/current/circuit-breaker.html#parent-circuit-breaker) errors and no service map visualization. This query can fire up to 20 times max depending on how many trace ids are fetched in subsequent query, contributing more to exceeding the total allowable memory. These changes will not remove the possibility of OOMs or circuit breaker errors. It doesn't control for multiple users or other processes happening in kibana, rather we are removing the current state of querying for an unknown number of documents by providing a hard limit and a way to easily tweak that limit. ## Changes - Make get_service_paths_from_trace_ids "shard aware" by adding an initial query, `get_trace_ids_shard_data` without the aggregations and only the trace id filter and other filters in order to see how many shards were searched - Use a baseline of 2_576_980_377 bytes max from new config `serverlessServiceMapMaxAvailableBytes`, for all get_service_paths_from_trace_ids queries when hitting the `/internal/apm/service-map` - Calculate how many docs we should retrieve per shard and set that to `terminateAfter` and also as part of the map phase to ensure we never send more than this number to reduce - Calculation is: ((serverlessServiceMapMaxAvailableBytes / average document size) / totalRequests) / numberOfShards Eg: 2_576_980_377 / 495 avg doc size = 5,206,020 total docs 5,206,020 total docs / 10 requests = 520,602 docs per query 520,602 docs per query / 3 shards = **173,534 docs per shard** Since 173,534 is greater than the default setting `serviceMapTerminateAfter`, docs per shard is 100k - Ensure that `map_script` phase won't process duplicate events - Refactor the `processAndReturnEvent` function to replace recursion with a loop to mitigate risks of stack overflow and excessive memory consumption when processing deep trees ## Testing ### Testing that the scripted metric agg query does not exceed the request circuit breaker - start elasticsearch with default settings - on `main`, without these changes, update the request circuit breaker limit to be 2mb: ``` PUT /_cluster/settings { "persistent": { "indices.breaker.request.limit": "2mb" } } ``` - run synthtrace `node scripts/synthtrace.js service_map_oom --from=now-15m --to=now --clean` - Go to the service map, and you should see this error: <img width="305" alt="Screenshot 2024-06-20 at 2 41 18 PM" src="https://github.com/elastic/kibana/assets/1676003/517709e5-f5c0-46bf-a06f-5817458fe292"> - checkout this PR - set the apm kibana setting to 2mb(binary): `xpack.apm.serverlessServiceMapMaxAvailableBytes: 2097152`. this represents the available space for the [request circuit breaker](https://www.elastic.co/guide/en/elasticsearch/reference/current/circuit-breaker.html#request-circuit-breaker), since we aren't grabbing that dynamically. - navigate to the service map and you should not get this error and the service map should appear --------- Co-authored-by: Carlos Crespo <carloshenrique.leonelcrespo@elastic.co> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Version:
Serverless project v 8.14.0
Description:
GET internal/apm/service-map
request fails with status code 502 and returnsbackend closed connection
when filtering data outside of the boost window.Preconditions:
I reproduced the issue having ~154 dependencies and 1011 services.
Steps to reproduce:
Expected behavior:
Service map data returned for set time interval.
Logs:
Log messages look somewhat fragmentary:
The text was updated successfully, but these errors were encountered: