Possible Memory Leak

[kramttocs]

All Windows based

Server and node running as a service.
I am transcoding a lot of small files so it churns through them pretty quickly individually.
I am seeing the Process Mem X/X at the top (and validated with Task Manager) steadily climb. It got to 3900MB today so I paused the Node and restarted Server. It's been running for a couple hours and is at 440/507MB currently.

It does drop at times but in the long run, rises.

[Brendonwbrown]

This may indeed be the issue that has been grinding my server to a halt every two days.

[HaveAGitGat]

Thanks for opening an issue, have not encountered this with Docker container or Windows myself. Does task manager show a specific sub-process that has not closed down correctly? Such as an unclosed FFmpeg process or something like that?

[kramttocs]

I know this one may be tough or impossible to track down. I didn't see anything that stood out as far as ffmpeg processes went. I may have another batch of small files I'll be running through but the Process Memory has remained reasonable for awhile now.
Sorry, wish I could give better details. Will add more if I notice anything.

[HaveAGitGat]

Ok thanks, yeah memory issues are always a pain but let me know if you re-encounter the issue and can have another look at the process and subprocess details,

[Brendonwbrown]

Seeing this issue reappearing over the past week. PC Memory fills up: when browsing resources, the top process for Working Memory is Memory Compression, and second process is Tdarr_Server_Runtime

Restarted server 2 hours ago, ffmpeg process is currently converting.

Is it normal to have multiple Tdarr_Server_Runtime and Tdarr_Node_Runtime processes running?

[kramttocs]

Right now my server process shown via Tdarr UI is at 2100/2500mb. It fluctuates but never goes down significantly, just climbs slowly. If there is anything we can provide you to help with this, sounds like @Brendonwbrown and myself may be able to do so.

[kramttocs]

I am doing all my music files so like before with my small security videos, I am processing a lot of small files. For me that's been a common factor.

[HaveAGitGat]

@Brendonwbrown yes it's normal to have multiple runtime processes. Each time a subprocess is created, a new runtime process appears. A node with 1 worker running will have at least 3 runtime processes. Likewise on the server there's a lot of multi-threaded stuff (such as file scanner) each creating their own subprocess, but should close down once done.

[HaveAGitGat]

@kramttocs are you adding more files to Tdarr? This will cause more ram usage over time.

[kramttocs]

@kramttocs are you adding more files to Tdarr? This will cause more ram usage over time.

Kind of but I don't think that's the root of what's happening here. I understand that the more it has to watch will increase the ram usage but I would expect that to be either 1) consistent or 2) temporary without the need for a restart.

1. by this I mean if file count was X which translates into memory Y then I would expect a restart of the Server service (after startup settles down) to have a memory of Y (within reason)

2. while scanning and doing all of the stuff for files it found I would expect the memory to ramp up but then over time drop back down

The problem with me thinking it's either of these is that if I restart the memory usage stays pretty low and reasonable so I don't think it's 1. With 2 it never goes down when it gets into this situation.

I know I keep mentioning a lot of small files but that's really when I've seen it. The two situations are:

1. My security recordings. mp4 to mkv. A lot of small mp4 files (like 500kb and such) It has gone through these and is now in the 700mb+ range so I haven't seen the issue lately.

2. My music library. I've stopped doing this for now (unrelated to this issue).

[Kapujino]

Hello,

I also suspect an issue with the memory management at some point.

How I use Tdarr (great software btw):
Tdarr_Server running in kubernetes and the Tdarr_Node is running on a notebook, doing transcoding whenever it's powered on.
Due to missing features (e.g. regex rename) I don't want to configure Tdarr to run against my whole library yet.

That's why I add separate series as "library" in Tdarr -> let them transcode -> delete the library in Tdarr.

Here is a memory graph of the Tdarr_Server over 30 days with my type of usage:

If I had to guess, I would say that when I delete a library in Tdarr it's not properly cleaned up from memory.

I hope that helps to identify the issue.

[HaveAGitGat]

I've made some changes to this in 2.20.01 (just released), so hopefully that fixes/improves things for you.

[miniserver1]

I'm unsure if this is related but I'm having similar issues.

I've had a look through existing and historical issues.

I've been having issues for the past month or so whereby running the latest version of Tdarr in a Linux Container (LXC) on a Proxmox node with nothing else running in the container causes the memory usage to increase over time and ultimately hit the allocation and becoming unresponsive.

I've increased the memory allocation to compensate but it reaches 2GB of memory usage within the space of 12 hours. You will note this in the graph below over the past month. The memory usage increases exponentially until it just caps out and then CPU usage hits 100% and it then refuses all requests.

If I restart the container I don't get long to interact with the container or Tdarr until it locks up.

I've attached the logs to see if anyone can give me any pointers as I'm enjoying using Tdarr!

tdarr-server.txt

I keep the container shut down to stop it exhausting all of the system resources. I'm running version 2.22.01 and have the Node running in the same container as the server.

[Kapujino]

@HaveAGitGat I think your changes are noticable. Atleast on my end it looks like a huge improvement in memory usage.
Although this might partly be caused by more restarts (due to new versions).

The graph shows the last 6 months with every new color showing a restart.
The way I use Tdarr did not change but the memory consumption decreased.
Many thanks for the change!

[HaveAGitGat]

@Kapujino that's interesting thanks for sharing. I've recently been working on migrating Tdarr to sqlite3 and redoing a whole lot of stuff in order to reduce memory usage (94 files changes so far, will be quite a big update under the hood).

Tdarr Server core process memory usage with 1 million files queued and at idle:
v2.23.01: ~10GB
v2.24.01: ~150MB

Tdarr Server core process memory usage with 1 million files queued and with 10 workers running
v2.23.01: ~10GB
v2.24.01: ~250MB

Will post here when I have a decent container to try, should be soon.

[HaveAGitGat]

These 2 images have passed the tests and seem to be working well:

docker.io/haveagitgat/tdarr_acc:dev_2.23.01_2024_08_02T16_37_27z
docker.io/haveagitgat/tdarr_node_acc:dev_2.23.01_2024_08_02T16_37_27z

It should reduce RAM usage by over 95% for large libraries. On startup a database backup will be created, check Tools->Backups for how to restore if you need to go back to the previous version. DB migration status is shown in the logs and the UI.

[HaveAGitGat]

This update is now in pre-release for version 2.24.01:

You can try it using these steps across all platforms:
https://docs.tdarr.io/docs/releases/changing-version#pre-releases

- This extensive 'under-the-hood' update significantly improves large library performance and RAM usage
- Tdarr servers with more than 1 million files can expect over a 97% reduction in RAM usage in the main server process and much lower times for workers to request new tasks
- A database backup will be created on startup before a migration occurs
- If you need to revert to a previous version, follow the steps on the Tools->Backups to restore previous data
- Added auth loading animation
- On Stats tab, added pop up when viewing pie files
- On Stats tab, moved stream stats from System to per-library tabs
- Fixed classic plugin issue causing dependencies to be reinstalled on every run if version number specified

[HaveAGitGat]

2.24.01 released. Feel free to add info here if need be.