Repository navigation
SummaryCronicle appears to leak file descriptors for job log files under sustained schedule load. Open FDs for Environment
SymptomsCronicle's FD count grows steadily over time and can exceed 42,000. When this happens, Cronicle eventually starts failing with:
Restarting Cronicle drops the FD count back down immediately. What I observedI checked the running Cronicle process and found that most open FDs are regular files under Example: sudo bash -lc 'pid=$(pgrep -f "Cronicle Server" | head -1); echo "PID=$pid"; echo -n "open_fds="; ls "/proc/$pid/fd" | wc -l; echo -n "jobs_fds="; lsof -p "$pid" 2>/dev/null | awk "NR>1 && \$9 ~ /^\/opt\/cronicle\/logs\/jobs\// {print \$9}" | wc -l; lsof -p "$pid" 2>/dev/null | awk "NR>1 && \$9 ~ /^\/opt\/cronicle\/logs\/jobs\// {print \$9}" | sort | uniq -c | sort -nr | head -30'Observed output at one point:
Later, even after the jobs had already completed hours earlier, the same Plugin title in all cases was usually Important detailThis does not look like a normal concurrency issue. I already verified that the relevant events are configured with:
The category has only The files that remain open are not active jobs anymore. In at least one case, the log file had belonged to a job that finished about 3 hours earlier, yet the FD was still held open by the Cronicle server process. Expected behaviorOnce a job completes, Cronicle should close the corresponding job log file descriptor. Actual behaviorCronicle keeps many completed job log files open indefinitely, causing FD usage to grow until the process hits the OS limit. WorkaroundRestarting Cronicle immediately resets the FD count and clears the problem. Now I have set cron to run Additional notesI initially suspected this might be caused by high job concurrency, but the issue persisted even after disabling catch-up and queued jobs, and limiting event concurrency. The remaining evidence points to a file descriptor leak or delayed close path in the server process.This problem occurs only on the Master node. Please help me with this problem. |
Replies: 3 comments 10 replies
|
Thank you for this detailed write-up. Can you please confirm that you are seeing this behavior on the "official" Cronicle, and the latest version (v0.9.118)? i.e. you're not on a 3rd party fork like cronicle-edge? |
|
I probably don't even know what cronicle-edge is. How can I check? |
|
I'll accept a PR if your AI can find a provable root cause and fix. I heard Claude Fable 5 is basically god level. But I can't afford it. |

Okay, is highly likely part of the problem right there. I don't understand why you would do this. The
/opt/cronicle/jobsdirectory is internal, and not "user-serviceable". You shouldn't read or write to it at all. And you should definitely not try to sync it between servers, nor any directories under/opt/cronicle. I cannot even fathom what that would do.For a multi-server cluster, you …