Possible memory leak when AutoCollectPreAggregatedMetrics enabled (2.x)

[umamimolecule]

We have a Node web API that connects to Azure table storage and logs to Application Insights. It was upgraded to use version 2.9.1 of the App Insights package and some time after that, we noticed the application was running slowly and with high memory usage. After a restart, the memory starts low and gradually ramps up over time.

We are using the default configuration for the Application Insights client, ie enableAutoCollectDependencies is true and enableAutoCollectPreAggregatedMetrics ` is also true.

After adding some log statements in the Application Insights package, I discovered that the AutoCollectPreAggregatedMetrics._dependencyCountersCollection collection keeps growing forever. It turns out in our case the vast moajority of the items in this array are for HTTP calls to Azure Table Storage, here's an example showing two counters:

[
  {
    "dimensions": {
      "cloudRoleInstance": "MyMachineName.local",
      "cloudRoleName": "Web",
      "dependencySuccess": true,
      "dependencyType": "HTTP",
      "dependencyTarget": "http://127.0.0.1:10002/devstoreaccount1/MyTable(PartitionKey='b0264796-46d3-42bc-a34b-ca449e7ce19b',RowKey='3d203f45-6500-4f0d-8c8e-73281bab1315')",
      "dependencyResultCode": "404"
    },
    "totalCount": 1,
    "lastTotalCount": 0,
    "intervalExecutionTime": 5,
    "lastTime": 1717745473248,
    "lastIntervalExecutionTime": 0
  },
  {
    "dimensions": {
      "cloudRoleInstance": "MyMachineName.local",
      "cloudRoleName": "Web",
      "dependencySuccess": true,
      "dependencyType": "HTTP",
      "dependencyTarget": "http://127.0.0.1:10002/devstoreaccount1/MyTable(PartitionKey='3c9ac79e-0c23-4388-a8d0-6897e8eafd4d',RowKey='3c37aee4-5654-48b8-b8ea-8f086b74abfb')",
      "dependencyResultCode": "404"
    },
    "totalCount": 1,
    "lastTotalCount": 0,
    "intervalExecutionTime": 2,
    "lastTime": 1717745474255,
    "lastIntervalExecutionTime": 0
  }
]

In this case, two HTTP calls were made to table storage to fetch an entity. Because the URLs in the dependencyTarget are different, the code in AutoCollection/PreAggregatedMetrics.ts treats them as different counters and appends to the collection. I have a minimal repro repository here

Our application handles a lot of traffic to and from table storage, with most of the calls having a unique value in the row key value, hence most of the calls to table storage are for unique URLs.

Can you advise if this is expected behaviour, or if there is any way to clear out the aggregated counters periodically? For now we have disabled auto-collection of pre-aggregated metrics to work around the issue.

[hectorhdzg]

@umamimolecule thanks for reproducible environment, this is a high cardinality issue caused by Span created by Azure SDK instrumentation being used internally.

[hectorhdzg]

The fix would be available in next release