New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minutely Scheduled Tasks #304

Closed
caronc opened this Issue Nov 10, 2016 · 6 comments

Comments

Projects
None yet
2 participants
@caronc
Contributor

caronc commented Nov 10, 2016

Right now the Scheduler in NZBGet supports basic features such as:

  • *:30: Run every 30 minutes
  • 12:00,00:00: Run at 12pm and 12am

This is great! But for the script I recently created to scan for NZB-Files, it'd be cool if it was feasible to scan minutely or even just every 2 minutes using the built in NZBGet scheduler. Maybe support something as basic as this (i'm not sure the level of effort off hand):

  • *:*0,*:2,*:*4,*:*6,:*8: Run every 2 minutes (following similar architecture you already have in place).

Of course, i wouldn't complain if you got fancy and supported cron like commands such as:

  • *:*/2: Run every 2 minutes
  • *:*:00: Run every minute; but putting a different spin on the request by adding a seconds field into your time.
  • *:*:00,*:*:30: Run every thirty seconds (just continuing with the design you have in place and the addition of a seconds field).
  • *:*:*/30: Run every 30 seconds; just pushing the cron style approach instead of separating a request with so many comma's.

As it is right now, it's almost easier to just use the dirwatch in a cron (no help for Windows users) with:

# $> crontab -e
# Scan every 2 minutes:
*/2 * * * * /path/to/DirWatch.py -t /path/to/NZBGet/NzbDir ~/DropBox

Maybe my script is the only use case right now making this request unnecessary? Maybe there are more tools/scripts that would benefit from it?

Thoughts?

@hugbug

This comment has been minimized.

Show comment
Hide comment
@hugbug

hugbug Nov 11, 2016

Member

:0,:2,:4,:_6,:_8: Run every 2 minutes (following similar architecture you already have in place).

I guess I can do that. However the thing is the scheduler in nzbget isn't as sophisticated as cron. The support for * is implemented primitively by creating tasks for each time point. This is happen during reading of config file. The scheduler itself isn't aware of asterisk in time format.

So for example when you use *:00 the scheduler becomes 24 tasks created by config file reader. Not elegant, I know, it was a comfort function to help of defining of hourly tasks.

If we use that for minutes it will be easy to define minutely tasks resulting in 3600 task objects. All of them checked every second. Sounds not optimal.

It's surely possible to rewrite the scheduler to be more advanced to not require creating of separate tasks for each time point but that would be certainly more work.

Another concern is a possible performance impact of starting python scripts that often. A live example: my primary download machine is a Linux SAT receiver with CPU MIPS 400 MHz dual core. During download it eats the whole CPU (and still can't saturate my Internet link). When testing FakeDetector script I've found a problem: due to high load of CPU the startup time of the script was about 50 seconds (before the first line of the script was executed). Even on idle CPU the system requires several seconds to start python.

Yes, that's a slow machine. But even on much better machines it may become a problem when the CPU is fully utilised.

Alternative approach

Considering all of this I think a better approach could be to start the script once and keep it running. The script should then organise a run loop with sleep intervals.

When to start the script?

The user could configure the script to run (for example) every 15 minutes. On start the script should check the existence of another running instance. That can be achieved with lock-files.

The scheduler interval of 15 minutes means a delay between nzbget starts and the script starts. To avoid this delay we could extend the scheduler with ability to start scripts when the program starts. For example let's say the value START or a simple asterisk * in time field could mean "start that task on program start (only once)". For example TaskX.Time=*,*:00 would mean "on program start and then every hour". The additional "every hour" in the example is to reassure the script is restarted on crash.

Could be probably useful for other things too.

Member

hugbug commented Nov 11, 2016

:0,:2,:4,:_6,:_8: Run every 2 minutes (following similar architecture you already have in place).

I guess I can do that. However the thing is the scheduler in nzbget isn't as sophisticated as cron. The support for * is implemented primitively by creating tasks for each time point. This is happen during reading of config file. The scheduler itself isn't aware of asterisk in time format.

So for example when you use *:00 the scheduler becomes 24 tasks created by config file reader. Not elegant, I know, it was a comfort function to help of defining of hourly tasks.

If we use that for minutes it will be easy to define minutely tasks resulting in 3600 task objects. All of them checked every second. Sounds not optimal.

It's surely possible to rewrite the scheduler to be more advanced to not require creating of separate tasks for each time point but that would be certainly more work.

Another concern is a possible performance impact of starting python scripts that often. A live example: my primary download machine is a Linux SAT receiver with CPU MIPS 400 MHz dual core. During download it eats the whole CPU (and still can't saturate my Internet link). When testing FakeDetector script I've found a problem: due to high load of CPU the startup time of the script was about 50 seconds (before the first line of the script was executed). Even on idle CPU the system requires several seconds to start python.

Yes, that's a slow machine. But even on much better machines it may become a problem when the CPU is fully utilised.

Alternative approach

Considering all of this I think a better approach could be to start the script once and keep it running. The script should then organise a run loop with sleep intervals.

When to start the script?

The user could configure the script to run (for example) every 15 minutes. On start the script should check the existence of another running instance. That can be achieved with lock-files.

The scheduler interval of 15 minutes means a delay between nzbget starts and the script starts. To avoid this delay we could extend the scheduler with ability to start scripts when the program starts. For example let's say the value START or a simple asterisk * in time field could mean "start that task on program start (only once)". For example TaskX.Time=*,*:00 would mean "on program start and then every hour". The additional "every hour" in the example is to reassure the script is restarted on crash.

Could be probably useful for other things too.

@hugbug hugbug added the feature label Nov 11, 2016

@caronc

This comment has been minimized.

Show comment
Hide comment
@caronc

caronc Nov 11, 2016

Contributor

I like your alternative idea!

Do you think you could provide the PID of the NZBGet instance so that the python script can die quietly if NZBGet does? Just a way of still keeping an artificial Parent/Child relationship? This is especially the case if someone sends a reload signal to NZBGet, I'd want to know that too because maybe i shouldn't be running in memory anymore, or i have new paths to scan. A reload should effectively kill my forked process until the next Scheduled interval is reached (but i can do that myself knowing the PID or getting a SIGHUP of my own). Perhaps i should send a signal back to you of my PID for you to manage?

I presume the PID changes on a reload anyway so this would just work if you could pass that info along.

On a side note; don't knock yourself on the current architecture; as basic as it is; it works great. So all is good! I was just proposing options! :)

Contributor

caronc commented Nov 11, 2016

I like your alternative idea!

Do you think you could provide the PID of the NZBGet instance so that the python script can die quietly if NZBGet does? Just a way of still keeping an artificial Parent/Child relationship? This is especially the case if someone sends a reload signal to NZBGet, I'd want to know that too because maybe i shouldn't be running in memory anymore, or i have new paths to scan. A reload should effectively kill my forked process until the next Scheduled interval is reached (but i can do that myself knowing the PID or getting a SIGHUP of my own). Perhaps i should send a signal back to you of my PID for you to manage?

I presume the PID changes on a reload anyway so this would just work if you could pass that info along.

On a side note; don't knock yourself on the current architecture; as basic as it is; it works great. So all is good! I was just proposing options! :)

@hugbug

This comment has been minimized.

Show comment
Hide comment
@hugbug

hugbug Nov 11, 2016

Member

Currently NZBGet terminates all child processes when it shutdowns or reloads. Not sure if that is a good thing in our case as your script will not have a chance to exit gracefully.

Member

hugbug commented Nov 11, 2016

Currently NZBGet terminates all child processes when it shutdowns or reloads. Not sure if that is a good thing in our case as your script will not have a chance to exit gracefully.

hugbug added a commit that referenced this issue Nov 15, 2016

@hugbug hugbug added this to the v18 milestone Nov 15, 2016

hugbug added a commit that referenced this issue Nov 15, 2016

#304: graceful termination of scheduler scripts
If a script is running when the program must shutdown, the script
receives signal SIGINT (CTRL+BREAK on Windows) and has 10 seconds to
gracefully terminate until it is killed.
@hugbug

This comment has been minimized.

Show comment
Hide comment
@hugbug

hugbug Nov 15, 2016

Member

Script startup on program startup

For example TaskX.Time=*,*:00 would mean "on program start and then every hour".

Implemented * as time field for scheduler tasks.

Graceful script shutdown

If a script is running when the program must shutdown, the script receives signal SIGINT (SIGBREAK aka CTRL+BREAK on Windows) and has 10 seconds to gracefully terminate until it is killed.

Example script which handles shutdown signal:

#!/usr/bin/env python

### NZBGET SCHEDULER SCRIPT
# This script hangs in a loop but can be gracefully terminated on program shutdown.
#
# NOTE: This script requires Python to be installed on your system.

import os
import sys
import time
import signal

interrupted = False

def signal_handler(signum , address):
    print "Received terminate signal"
    global interrupted
    interrupted = True

if os.name == 'nt':
    signal.signal(signal.SIGBREAK, signal_handler)
else:
    signal.signal(signal.SIGINT, signal_handler)

a = 1
while not interrupted:
    try:
        a += 1
        print('[INFO] ' + str(a))
        sys.stdout.flush()
        time.sleep(1)
    except Exception as e:
        print('[ERROR] Exception: %s' % e)

print('[INFO] Exiting')
sys.stdout.flush()
Member

hugbug commented Nov 15, 2016

Script startup on program startup

For example TaskX.Time=*,*:00 would mean "on program start and then every hour".

Implemented * as time field for scheduler tasks.

Graceful script shutdown

If a script is running when the program must shutdown, the script receives signal SIGINT (SIGBREAK aka CTRL+BREAK on Windows) and has 10 seconds to gracefully terminate until it is killed.

Example script which handles shutdown signal:

#!/usr/bin/env python

### NZBGET SCHEDULER SCRIPT
# This script hangs in a loop but can be gracefully terminated on program shutdown.
#
# NOTE: This script requires Python to be installed on your system.

import os
import sys
import time
import signal

interrupted = False

def signal_handler(signum , address):
    print "Received terminate signal"
    global interrupted
    interrupted = True

if os.name == 'nt':
    signal.signal(signal.SIGBREAK, signal_handler)
else:
    signal.signal(signal.SIGINT, signal_handler)

a = 1
while not interrupted:
    try:
        a += 1
        print('[INFO] ' + str(a))
        sys.stdout.flush()
        time.sleep(1)
    except Exception as e:
        print('[ERROR] Exception: %s' % e)

print('[INFO] Exiting')
sys.stdout.flush()
@caronc

This comment has been minimized.

Show comment
Hide comment
@caronc

caronc Nov 15, 2016

Contributor

You're seriously working way to hard over there HugBug ;). What I might do is adopt this into my pynzbget wrapper so that anyone can just have the signal handling functionality who need it. I'll start with the dirwatch script (discussed earlier) as my first test.

Thanks for all your great work!

Contributor

caronc commented Nov 15, 2016

You're seriously working way to hard over there HugBug ;). What I might do is adopt this into my pynzbget wrapper so that anyone can just have the signal handling functionality who need it. I'll start with the dirwatch script (discussed earlier) as my first test.

Thanks for all your great work!

@hugbug

This comment has been minimized.

Show comment
Hide comment
@hugbug

hugbug Nov 16, 2016

Member

I may close this issue, right?
If I'm wrong - please reopen.

Member

hugbug commented Nov 16, 2016

I may close this issue, right?
If I'm wrong - please reopen.

@hugbug hugbug closed this Nov 16, 2016

hugbug added a commit that referenced this issue Dec 20, 2016

hugbug added a commit that referenced this issue Oct 9, 2017

hugbug added a commit that referenced this issue Oct 9, 2017

#304: graceful termination of scheduler scripts
If a script is running when the program must shutdown, the script
receives signal SIGINT (CTRL+BREAK on Windows) and has 10 seconds to
gracefully terminate until it is killed.

hugbug added a commit that referenced this issue Oct 9, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment