Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
0013774: since 2018.02.1, the regular maintenance cronjobs dont die and accumulate until there is no more free memory left #6812
Reported by bammes on 22 Feb 2018 20:16
Version: 2018.02.1 Community Edition
The regular maintenance Cronjobs get stuck with the following queries hanging until the database-connection times out:
log is full of these entries:
f0388 cronuser - 2018-02-22T20:05:00+00:00 WARN (4): Tinebase_Backend_Scheduler::cleanZombieTasks::121 cleaning up zombie scheduler task: Array
Comment posted by bammes on 27 Feb 2018 15:23
Looks like this is some deadlock when multiple cronjobs run at the same time.
Please note, that from time to time, it takes an excessive amount of time for the cronjob to finish (up to 10 hours). This looks like something is broken.
Comment posted by pmehrer on 13 Mar 2018 20:26
there are two problems here: one with the scheduler (the call to Tinebase.triggerAsyncEvents) and you worked around it.
the second problem is that Calendar_Controller_Event::updateConstraintsExdates seems to get stuck. Lets work on that first: Can you provide logs from one of theose 10 hours runs? like
Comment posted by bammes on 14 Mar 2018 11:28
Short analysis of what I have in my general syslog right now shows only one occurence today with 629 minutes.
root@www-1:~ # bzgrep tine20async /var/log/messages | grep real
Comment posted by bammes on 21 Mar 2018 17:23
Mar 21 00:50:00 www-1 tine20async: real 0m0.578s
Beginning of this issue already rotated by logrotate, but I got at least something of the ongoing task in the attached log.
Comment posted by pmehrer on 29 Mar 2018 15:36
the long running task:
in the log file you send I can't see anything related to that. Interessting would be the time from 00:00:00 to ... wherever the job gets stuck (I would assume we should see something within the first 15 minutes or so) Do you still have the logs from Mar 21?
thanks for providing information on this issue
Comment posted by bammes on 5 Apr 2018 18:45
I pasted everything I got, since I worked around this problem, my monitoring system does not complain about php eating all my memory. So I have to look in the logs from time to time and very often, the relevant logs have been rotated out.
But if you have a look on the list of jobs I posted before my last post, you can clearly see, that no all of these long-running jobs happen at midnight:
Mar 14 00:55:00 www-1 tine20async: real 0m0.728s
I will keep my eyes open and hope to catch another occurence of this bug with full log...
Comment posted by pmehrer on 19 Apr 2018 18:31
they do happen at midnight:
Mar 14 11:29:22 www-1 tine20async: real 629m22.249s
so the interessting log is not Mar 14 11:29, the interessting log is Mar 14 01:00 to Mar 14 01:15 if I can get that, I can help you.