Keep the main script alive, threads with I/O

[tappoz]

I've defined this example script main_foo.py which dies just after I execute it with python main_foo.py. If I enable that infinite while loop currently commented out at the bottom of the following example, then I see the script staying alive and printing logging statements as expected.

I have 2 questions:

• Is this infinite while loop the expected way to keep alive the script? I was expecting the library was taking care itself of keeping the script alive with some sort of blocking infinite loop and I was surprised I had to take care of it myself, so I am wondering if I am using it the right way or is there a recommended way to do this?
• Let's say I have first_func() in the following example which could potentially take more time than the recurrence at which it is scheduled. If the code inside first_func() is mainly network I/O (REST APIs) and numpy / pandas calculations (done in native C), then I should be good to go with crython invoking it multiple times even if some old executions still have to complete. I am assuming crython is running first_func() in a separate thread at each match of the crontab string (e.g. '*/5 * * * * * *'), so whenever this first_func() is finished the thread will die and crython will go on with no hiccups. Given that there is no CPU intensive code running in python (because the CPU intensive part is done in C), then I shouldn't be blocking the main thread of the main process, right? With this setup I should not mess with the GIL, I think.

import crython
import time

import sys
import logging
logging.basicConfig(
  format='%(asctime)s %(processName)-10s %(levelname)-6s %(message)s',
  level=logging.DEBUG,
  stream=sys.stdout
)

# every 5 seconds
@crython.job(expr='*/5 * * * * * *')
def first_func():
    logging.info("start 1st func")
    time.sleep(3) # but this could sometimes take more than 5 seconds
    logging.info("finish 1st func")

# every minute at seconds 1, 5 and 10
@crython.job(expr='1,5,10 * * * * * *')
def second_func():
    logging.info("2nd func")

if __name__ == "__main__":
    crython.start()
    # TODO is the following needed? 
    # while True:
    #    time.sleep(1)

[ahawker]

Unfortunately there isn't a method on the crython module to block on the global CronTab worker. I can imagine this being called block or join.

I'll look to add this as a feature request. However, let me give you a quick example that will unblock you in the near term.

from crython.tab import CronTab

tab = CronTab(name='my-app')

# every 5 seconds
@tab.job(expr='*/5 * * * * * *')
def first_func():
    logging.info("start 1st func")
    time.sleep(3) # but this could sometimes take more than 5 seconds
    logging.info("finish 1st func")


# every minute at seconds 1, 5 and 10
@tab.job(expr='1,5,10 * * * * * *')
def second_func():
    logging.info("2nd func")


if __name__ == "__main__":
    tab.start()
    tab.join()

The crython module interface is just a simplified version around a global CronTab instance. In the example above, we're just creating our own. With access to the instance, we can call .join on it.

Hope that helps!

[tappoz]

Uhm @ahawker I've created a file called crontab_main.py containing a copy/paste of what you suggested:

from crython.tab import CronTab

tab = CronTab(name='my-app')

# every 5 seconds
@tab.job(expr='*/5 * * * * * *')
def first_func():
    logging.info("start 1st func")
    time.sleep(3) # but this could sometimes take more than 5 seconds
    logging.info("finish 1st func")


# every minute at seconds 1, 5 and 10
@tab.job(expr='1,5,10 * * * * * *')
def second_func():
    logging.info("2nd func")


if __name__ == "__main__":
    tab.start()
    tab.join()

then on the command line:

$ python crontab_main.py 
Traceback (most recent call last):
  File "crontab_main.py", line 6, in <module>
    @tab.job(expr='*/5 * * * * * *')
AttributeError: 'CronTab' object has no attribute 'job'

I am using the latest release available:

$ pip show crython
Name: crython
Version: 0.0.9
Summary: Lightweight task scheduler using cron expressions.
Home-page: https://github.com/ahawker/crython
Author: Andrew Hawker
Author-email: andrew.r.hawker@gmail.com
License: MIT
Location: /<PATH_TO_MY_CONDA_ENVIRONMENT>/Miniconda3/lib/python3.6/site-packages
Requires: 

I see you were working on this in a separate branch/PR feature/tab-join-support, is there any plan to release that functionality?

Thanks :)

[ahawker]

@tappoz This should be available in version 0.1.0. Feel free to re-open this issue if that release doesn't work out.