Scheduler frequently stops scheduling tasks/dags

[mtustin-handy]

I'm using one of the versions after 1.6 (specifically commit e1f9a28), and this problem is happening more than once a day. It has happened in all versions I have used. Right now I have --num_runs=600 set on the scheduler, but that hasn't helped; the problem appears to occur before the scheduler can hit that count.

I also use the CeleryExecutor, backed by redis for celery message passing and results, and mysql as the database.

For some reason the scheduler frequently gets into a state where it doesn't schedule new dagruns or tasks. I haven't yet investigated whether it's failing to schedule tasks for existing dag runs also (I suspect that it is).

[mistercrunch]

What do the scheduler logs look like? The scheduler logs are very verbose, and you should clearly see where it's stuck.

We restart it every 5 runs using --num_runs mostly just to make sure we get a 100% refreshed dagbag. Keep in mind that this strategy doesn't work with LocalExecutor.

[mtustin-handy]

The logs aren't really giving me the detail I need. I'll lower my num_runs from 6000 to 10 and see how that does. Also, I've a branch with more debug logging in the works. I need to test it before I open the PR.

[bolkedebruin]

@mtustin-handy does this issue still occur? Please create Jira if so.