Possible memory leak? After cold start, lambda keeps consuming memory

[nicoan]

Hello,

Working on some lambdas I noticed that the memory consumption keeps incrementing and never goes down. I made several tests all of them threw more or less the same results. Here's the code I used for testing:

use lambda_http::{service_fn, Error, Request};

pub async fn function_handler(_: Request) -> Result<&'static str, Error> {
    Ok("")
}

#[tokio::main]
async fn main() -> Result<(), Error> {
    lambda_http::run(service_fn(function_handler)).await
}

Cargo.toml

[package]
name = "test_leak"
version = "0.1.0"
edition = "2021"

[dependencies]
lambda_http = "0.14.0"
tokio = { version = "1", features = ["macros"] }

To run the tests, I used an script that calls an API Gateway endpoint that calls the lambda:

#!/bin/bash
for ((i=1; i<=5000; i++)); do
    curl -i -X POST <api_g_url> -H 'Content-Type: application/json' --header 'Content-Length: 6' -d '"test"'
done

Here are the graphs of the memory consumption from the tests:

[jlizen]

A few questions:

1. Is the memory growing without bounds to the point where the lambda is crashing? IE, are we sure this isn't just the allocator holding onto memory and not releasing it to the system, even if isn't actively used? Since, these metrics would reflect all memory held by the process, not just memory that is in active use.

2. Does the same behavior appear locally when run with cargo lambda? Presumably it should, since the runtime is unchanged, just the outside orchestrator is different? Heap profiling would be much simpler in your local than cargo lambda.

3. What happens when you switch the allocator from system allocator to jemalloc?

[nicoan]

Hello @jlizen

1. I never got to the point where the lambda crashed. I made two kind of tests:

• Hitting it continously for 5, 10 and 15 min without pause: here the memeory continously grew
• Do a little pause and hit it again continously: this was to prevent a cold start, the memory started grewing again from the last point, the lambda did not free the memory.

Unfortunately I no longer have access to an AWS account to continue this kind of tests.

Regarding 2 I did not try that. I'll try to profile it with jemalloc to try to figure out what is happening

[jlizen]

the memory continuously grew

Ah, I see, in the charts it looked like it was plateauing after an initial increase. If it continues growing further with subsequent executions, that is indeed suspicious.

[nicoan]

the memory continuously grew

Ah, I see, in the charts it looked like it was plateauing after an initial increase. If it continues growing further with subsequent executions, that is indeed suspicious.

Yes! It plateaus, then spike, then plateau, then spike, but it always goes up. The plateau intervals are not regular, that is also strange

[PartiallyUntyped]

Hi folks,

We've investigated this, and done tests both locally and on production.
So the reason for max-memory-used is fragmentation, with the culprit being serde.

We run thousands of invokes both overriding the global alloc with tracking allocator, and did a flamegraph, and we see that the active memory stays fixed after a few invokes, in both cases there's some continuous allocation due to serde parsing payloads and then deallocating.

Many thanks for prompting this investigation!

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.

[jlizen]

@PartiallyUntyped is there a recommendation for users for reducing fragmentation? Conventional wisdom is usually to use jemalloc, though these days I reach for mimalloc first. Did you try those in profiling / did they behave differently?

Also, do we have a sense of scale wrt extent of continuous allocation pressure from serde usage and how it affects invocation latency? Needing to request more memory from the os with some frequency seems like it could impact tail latencies. Ie, is this something we care to keep an open issue for around optimization?

[nicoan]

Awesome news that you found the cause! Sorry I couldn’t do the profiling myself, I’ve been short on time.

[PartiallyUntyped]

So our doc is pending internal review and we will share our findings here. We have been pretty rigorous on this and @mpindaru has done a great job!

[indiv0]

@PartiallyUntyped hello! Seems like I might be running into the same issue. Are there any updates on the doc? Would like to know the recommended strategy to address this problem.

As a more specific sub-question: it seems like ApiGatewayProxyRequestContext<T> defaults to T = serde_json::Value. Is there any way to replace that with ijson::IValue (or some other, more memory-efficient type) via the lambda_runtime/lambda_http public API?