Memory leak with high `tracesSampleRate`

[raymondhechen]

Is there an existing issue for this?

• I have checked for existing issues https://github.com/getsentry/sentry-javascript/issues
• I have reviewed the documentation https://docs.sentry.io/
• I am using the latest SDK release https://github.com/getsentry/sentry-javascript/releases

How do you use Sentry?

Sentry Saas (sentry.io)

Which SDK are you using?

@sentry/node - express

SDK Version

10.27.0

Framework Version

Express 5.1.0

Link to Sentry event

No response

Reproduction Example/SDK Setup

No response

Steps to Reproduce

Set tracesSampleRate to 1 and observe memory usage, likely with a lot of Sentry.startSpan() usage.

Expected Result

Memory usage to not keep growing.

Actual Result

Up until 10pm in the screenshot, we were using tracesSampleRate of 1. Then we switched it to 0.2. The dotted lines indicate new deployments (thus resetting memory). We've been having memory leak issues for several versions of @sentry/node now, but we didn't realize until yesterday that the biggest lever that changed this was the tracesSampleRate. The chart unfortunately doesn't show additional memory leaks, but it's consistently been there at high growth rates until now. If there any ideas on why this is happening and how Sentry is affecting this, would be greatly appreciated, as we can see memory usage still growing but just far slower now.

Additional Context

We wrap Sentry methods in our helper functions in case this affects anything:

import * as Sentry from '@sentry/node'

interface ServerSpanArgs {
  name: string // human-readable name
  op?: string // semantic operation
  attributes?: Record<string, string | number | boolean | undefined>
}

export const withServerTrace = async <T>(
  { name, op = 'service', attributes }: ServerSpanArgs,
  fn: (span: Sentry.Span) => Promise<T> | T
): Promise<T> => {
  return Sentry.startSpan({ name, op, attributes }, async (span) => {
    try {
      return await fn(span)
    } catch (err) {
      span.setStatus({ code: 2, message: 'internal_error' })
      Sentry.captureException(err)
      throw err
    }
  })
}

export const captureServerException = (
  error: unknown,
  extra?: Record<string, unknown>
) => {
  Sentry.captureException(error, { extra })
}

Priority

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it.}

[linear]

JS-1224

[Lms24]

Hey @raymondhechen thanks for writing in! Can you share a bit more around your usage of Sentry?

• How do you init the SDK? Can you show us some code and how/where you call it?
• How are you using withServerTrace?
• To me withServerTrace looks like you want the span within to be the local root span (not sure though of course) of the trace. Are you disabling auto instrumentation?

What you've shown so far already gives us some hints. An increase in tracesSampleRate could explain a memory increase very well, given that the SDK might record a lot of spans. But it's too early to say if it is actually about spans or just data getting applied more often.

[raymondhechen]

Thanks for the response @Lms24! Here's how we init the SDK:

import * as Sentry from '@sentry/node'
import 'dotenv/config'

const DISABLED_DEFAULT_INTEGRATIONS = [
  'Amqplib',
  'Kafka',
  'OpenAI',
  'Tedious',
  'Prisma',
  'Mysql2',
  'Mysql',
  'Mongoose',
  'Mongo',
  'Fastify',
  'Hapi',
  'Koa',
  'Firebase',
  'Anthropic_AI',
]

Sentry.init({
  // debug: true,
  environment: process.env.NODE_ENV,
  dsn: process.env.SENTRY_SERVER_DSN,
  // filter out default integrations that are not needed
  integrations: (integrations) => {
    return integrations.filter(function (integration) {
      return !DISABLED_DEFAULT_INTEGRATIONS.includes(integration.name)
    })
  },
  ignoreErrors: ['AbortError'],
  tracesSampleRate: 0.2,
  sendDefaultPii: true,
})

We use withServerTrace() by just wrapping any function we want to measure the performance of. For example:

const result = await withServerTrace(
    { name: 'foo', op: 'bar' },
    async () => {
      return Promise.resolve({})
    }
  )

We really just use withServerTrace() as a helper function to wrap Sentry.startSpan() to start a span within a parent span or an express route/handler trace. I assumed it may be a memory leak issue because memory usage just keeps growing, and reducing trace sampling rate without changing anything else resulted in at least slower growth. Let me know if specific examples of our withServerTrace() could be helpful, although we really just wrap all kinds of async functions with it.