Replies: 4 comments 7 replies
|
Impressive. I barely have enough time or skills to maintain a python a simple script, let alone adding in Go. I don't think it's a good for fit for this specific repo. I'd be thrilled if you have any suggestions for the 3 leaks you found. I've been chasing them for a while and have slowly fixed the couple as they pop up. Do they only manifest in CPU? What kind of hardware are you running? And out of curiosity, what's your use case for so many consistent transcriptions? I probably should add a list of forks at the bottom of the readme... |
|
Totally makes sense, and kinda what I expected, who wants to take on an additional 24k lines of code for shits and giggles 😅 I'll go ahead and cut PRs for each of the memory leaks I've found as well as some tests that validate they are fixed/detect regression. I'll break things up into separate PRs to make it easier to review. One thing that I can suggest is something I've been slowly adding to my repos are AI workflows that allow me to request changes via issues and comments. Here is an example of me doing work on my kubernetes cluster using issues: lenaxia/talos-ops-prod#591, another example lenaxia/talos-ops-prod#926 Here are the two primary github workflows that I am using, they are calling my own LiteLLM instance, backed by a unlimited z.ai GLM-4.7 subscription, using OpenCode to do agentic deep dives of the code:
It really helps with getting small changes done quickly, in a way that you can review. In terms of my workload, its just a normal plex/sonarr/radarr workflow, probably like 2-3 episodes a day. But I also have a lot of episodes that need subtitles added. The long running nature of the k8s pod I think adds to the memory leak problems. Been running on CPU, I havent spun it up on my GPUs yet, though probably should. I mean the ultimate issue here is LLMs, "Lenaxia, why would you rebuild this to be highly scalable and microservice oriented?" "Because I could, and LLMs made it easy" 🤦♂️ |
|
I've created 4 PRs which should be merged in order. They contain first a set of test infrastructure that validates that the memory leaks actually exist, then a separate PR for each of the memory leaks I identified. I'll probaably cut another PR with additional test coverage too. |
|
I’ve been having the same issue keeping subgen’s container alive, but I couldn’t use your fork as a drop-in replacement in my setup. The orchestrator expects a Plex or Jellyfin server and throws errors if neither is configured. In my case, subgen only acts as a provider for Bazarr – everything else is automated through Bazarr – so I don’t connect subgen to Plex at all. Still, this is a really nice idea. Hopefully the project author takes a look at your pull requests soon 🤞. |
|
Once the transcription finishes, the API becomes reachable again and Bazarr is able to communicate with it. Is this expected behavior? |
It's not, and for better or worse, Bazarr 'respected' it. I've rewritten some of the synchronous code that should fix it. SHould be up in about 5 minutes. 
|
|
Updated now. Let me know. A sigkill 137 is usually the host killing a process for OOM, may be unrelated... |
|
Glorious. Working perfectly now! I'll keep checking the container every so often to see if it stays up.
Yeah, I'll report back if it comes up again. I can transcribe content back to back without a restart in between now, so it's far more reliable. |
|
Thanks for the quick turnaround! |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
I've been running subgen in Kubernetes for media server automation, but encountered persistent memory leak issues. Specifically, there were three memory leaks causing pods to OOM and restart: task_results dictionary accumulation, timer thread leaks, and unclosed BytesIO objects. After days of runtime, these would cause pods to OOM or consume excessive memory.
My initial goal was to just fix the memory leaks. What happened instead was a complete architectural rewrite. I split the system into a Go orchestrator and Python worker microservices with gRPC communication, enabling horizontal scaling. The result achieved 100% feature parity with the original subgen.py, with 71 tests passing and zero memory leaks verified.
The new architecture has an orchestrator in Go handling webhooks, queues, skip logic, and metrics, communicating via gRPC with Python workers that manage Whisper models with proper lazy loading and cleanup. This completely eliminated my OOM crashes while preserving all original features.
It can be deployed with a single docker compose still, but does have two containers running.
Moar Code
The original subgen.py is 2,144 lines. The new system has approximately 8,000 lines of functional Go code, 2,700 lines of functional Python code, plus 16,000 lines of test code across both. The increase accounts for production error handling, observability infrastructure (189 prometheus metrics), gRPC communication layer, and comprehensive test coverage.
Documentation and Process
This was built with heavy LLM assistance using a systematic approach, see README-LLM.md. The docs/BACKLOG/ directory has 55 design files breaking down 9 EPICs with story-driven development. The docs/WORKLOGS/ directory has 88 files documenting development progress, tests, bug fixes, and deployment guides. See docs/WORKLOGS/0068_2026-02-17_complete_docker_testing_all_passing.md for the complete test results showing feature parity.
What Next?
I'm happy to maintain this as a separate fork for users who need Kubernetes and horizontal scaling. But wanted to open a discussion in case there's interest upstream. Potential paths: maintain it as a fork, offer as optional deployment mode alongside the monolith, or cherry-pick specific memory fixes as incremental PRs to the Python version.
Also happy to just open a PR with the memory leak fixes for the original subgen.py.
Trade-offs
Advantages: no memory leaks, horizontal scaling, production observability, comprehensive testing.
Disadvantages: more code to maintain, requires Go + Python, multi-container deployment, higher complexity.
I understand if this doesn't fit the project's direction. The original's simplicity has real value. My rewrite optimizes for scale and reliability at the cost of complexity. Just wanted to share what I built and see if any might be useful upstream.
Repository: https://github.com/lenaxia/subgen
Test Results: https://github.com/lenaxia/subgen/blob/main/docs/WORKLOGS/0068_2026-02-17_complete_docker_testing_all_passing.md
Docker Images:
All reactions