Memory leaks and RequestContext

[roderik]

I’ve been hunting a memory leak in my nest application for a few days, and I think I’m fairly sure that I do not understand how the RequestContext is supposed to work.

Given the following cron job, I thought it should (since the decorator is there) create a new context, use the em from this clean context, and then this cron is done, garbage collect. (maybe with some delay)

What I’m seeing is my heap space jumping 3-400MB, but never dropping (maybe a fraction).

I see the same for my requests, every request i can check the RequestContext.currentRequestcontext.id and get a new id, but a call with a simple resolver->service->repository.findAndCount() (returns 6 complex items) loses 1-2MB.

  constructor(
    @Inject(forwardRef(() => ApplicationEntityService))
    protected applicationService: ApplicationEntityService,
    @InjectRepository(Usage)
    private readonly usageRepository: UsageRepository,
  ) {}

  @Cron(CronExpression.EVERY_HOUR)
  @UseRequestContext()
  async ensureStripeSubscriptionItemCron() {
    await this.applicationService.list() // does this.applicationRepository.findAll() in the service
    await this.usageRepository.findAll()
  }

I've removed about everything from my app that could interfere, and it still happens.

What am I missing here?

[roderik]

To give some idea, this is just the cronjob every hour

[roderik]

Trying some snapshots, i see 7 em's are retained with 61k of the usage entities and 61k of other wrapped entities, they are all retained by the mongo driver if i read this correct.

[jgburet]

You need to not use the first entity manager the orm give you once connected to your database, but instead fork it.

For example, each time you receive a new http request, in your Express/Koa/whatever middlewares chain, make sure to pass to the code downstream a child of the root entity manager. You can get one by doing em.fork().
Of course, this works the same each time you have process workers getting a task from your job queue, or each time you run a CLI command.

[roderik]

I can confirm i get new RequestContext id's every time. In the screenshots above you will also see the domain mentioned in the retainers.

Not an expert in memory traces, but it seems the mongo driver is keeping a reference.

[roderik]

I was hoping mongo would GC itself at some point, but no (drops are restarts due to k8s memory limits)

[roderik]

No improvement from downgrading a few versions of mikro, nor switching to AsyncLocalStorage

[roderik]

AsyncLocalStorage is not for the @UseRequestContext annotation, so duh....

[roderik]

In the end:

• not loading 62k items every 5 seconds helps a lot
• concurrently, so not really reproducible, but removing Sentry Tracing (uses a request context and mongo instrumentation) also seem to have helped a lot