Memory leak - SecretBuffer and SecretBytes

[jamshale]

I've been investigating an apparent memory leak in the native memory space when running a credo mediator agent under moderate load. The memory accumulates during crypto processing and then stops when encrypted payloads stops. See openwallet-foundation/credo-ts#2383.

After doing a deep profile with debug symbols using jeprof I was able to determine a majority of accumulating allocations are coming from askar libaries and crypto buffers. I believe there may be a problem in the ffi. This isn't occurring when askar is used via python.

Here is a couple interesting observations using chatGPT to analyze.

Big allocations are vector growth in crypto buffers

13.6 MB  alloc::raw_vec::RawVecInner::try_allocate_in
7.9 MB   alloc::vec::Vec::with_capacity
5.6 MB   alloc::slice::to_vec_in
3.8 MB   askar_crypto::buffer::secret::from (secret.rs:222)

These are standard Rust vector expansions, but they’re happening inside crypto buffer helpers (SecretBuffer, SecretBytes). 
That means something is creating Vec<u8>s for sensitive material and keeping them around.

FFI boundary allocations

3.8 MB  askar_crypto::buffer::secret::from (secret.rs:222)
1.0 MB  askar_storage::protect::profile_key::decrypt_entry_tags (profile_key.rs:231)
4.3 MB  aries_askar::ffi::key::askar_key_aead_decrypt::{{closure}}
0.5 MB  aries_askar::ffi::handle::ArcHandle::create (handle.rs:19)


This is a smoking gun 👆 — memory growth is happening where values cross the Rust ⇄ Node.js FFI boundary.
In particular, ArcHandle::create suggests secrets/keys are wrapped in an Arc and registered in the FFI handle table. 
If Node.js doesn’t call free (or the drop callback never fires), the handles accumulate.

I'm hoping someone with more experience in this project may have some ideas on how to fix it.

I think this problem basically prevents a credo mediator from operating correctly under load and I imagine issuer/verifier and mobile agents using asker will also have a memory leak.

[andrewwhitehead]

Looking at the JS wrapper code, it doesn't seem like the destructor is called for these types when they are received over the FFI. In one example (there could be others) I can see the SecretBuffer being copied, but askar_buffer_free is never called:

askar-wrapper-javascript/packages/askar-nodejs/src/ffi/conversion.ts

Line 26 in 21e2388

const buffer = Uint8Array.from(secretBufferToBuffer(secretBuffer))

[jamshale]

Yes. I've been looking at this a lot last week and I'm fairly certain that's the issue. It's not a huge leak but enough over time to cause problems under load.

I'm going to finish testing the fix out and open a PR. Hopefully soon.

I'm pretty certain the EncryptedBuffer is also not freed and missing the free function in the rust layer.

[andrewwhitehead]

EncryptedBuffer can be handled by just freeing the contained SecretBuffer.