[APM][Poc] Optimize memory usage in the reduce script

[crespocarlos]

Summary

POC to improve the memory consumption of the scripted metrics agg.

Current version

Running the script with the oom.ts scenario caused an Out of Memory (OOM) error. The heap usage spiked to around 96% before crashing.

The problem is caused by the paths object, which is a matrix containing the entire path for all events:

Rows: 8517
Columns (avg): 252 (could be many more)
Fields in each item: 3 (service.name, service.environment, agent.name)
Size of a field in bytes (avg): 55

paths object consumption (ROUGH ESTIMATION): 8517 rows * 252 columns * (3 fields * 55 bytes/field) = ~355 MB (all parallel requests combined)

This amount of data can also lead to a "content length bigger than the maximum allowed string" exception.

Refactored version

With the refactored version, the heap usage appears linear and didn't exceed 90% in any of the 3 calls made.

The query was streamlined to respond with what is necessary for the frontend to render the map in the connections object. This creates a much smaller object that the query has to hold in memory compared to what paths require.

eg:

"service-446~>service-447": {
  "destination": {
      "span.subtype": "unknown",
      "span.destination.service.resource": "service-447",
      "span.type": "app"
    },
    "source": {
      "service.environment": "Synthtrace: service_map",
      "service.name": "service-446",
      "agent.name": "nodejs"
    }
  },

With the oom scenario,connections holds approximately 550 unique connections. Therefore:

Keys: ~550 * 2 (source and destination) = 1100
Fields in each key: 3 (service.name, service.environment, agent.name fields)
Size of a field in bytes (avg): 55

connections object consumption (ROUGH ESTIMATION): 1100 keys * (3 fields * 55 bytes/field) = ~177KB (all parallel requests combined)

Worst-case scenarios, such as a long cyclic map, might still cause the space complexity to be O(N^2) and could potentially lead to an OOM.

[obltmachine]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[crespocarlos]

/ci

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: bf0bb62

Failed CI Steps

• Jest Tests #16
• Check Types

Test Failures

• [job] [logs] Jest Tests #16 / AllCasesListGeneric Actions Assignees should show the assignees column on platinum license

Metrics [docs]

✅ unchanged

History