Enforce number of concurrent Jobs running at the same time

[dgarros]

Environment

• Python version: 3.7
• Nautobot version: 1.1.3

Proposed Functionality

As Austin the Network automation Engineer I want to be able to create a Nautobot Job that can be executed only once at a time.
If a second execution of the same Job is requested while the Job is already running, the new Job should be queued until the first one finishes.

there are few other options that could be interesting to explore but not sure what is possible within the existing Framework: terminate the existing job early and start the new one, allow N numbers of concurrent execution ...

Use Case

Some Jobs are accessing shared resources and if multiple Jobs are running at the same time it could result in some inconsistency.

Database Changes

Probably not

External Dependencies

No

[glennmatthews]

Not just Job-specific - we should probably have similar enforcement for Git repository sync/refresh as well.

[itdependsnetworks]

Not to feature creep here, but job isolation tied to inventory would be great as well.

[jathanism]

This might be possible with a simple Redis lock and a check on task start.

Also worth looking at for inspiration: https://github.com/steinitzu/celery-singleton

[jathanism]

Digging into this is tricky because of how we rely on the run_job Celery task to be the central way that Jobs are executed. We're not using a custom BaseTask subclass to dictate any behaviors. The celery-singleton package provides its own Singleton base task class, but that's also too much in the other direction. There isn't a way to dynamically select the base task class used on a decorated task function. Since this has to be decided at registration time.

Ideas:

• We might be able to take advantage of subtasks here?
• Subclass the Singleton task class to to make it only "dynamically" assert uniqueness when a certain flag is set?
• Call a different task other than run_job somehow?

[Kircheneer]

This is something that could be really useful for longer-running SSoT jobs. We have been facing issues where those have been accidentally run twice - best case one of them errors out, worst case is a different outcome then if only one was run.

[Kircheneer]

My implementation idea would be different to what Jathan had in mind when he first tackled this:

• stick a Meta class option on the job called is_singleton or similar
• stick a job_singleton_timeout variable into the config, default it to something like 4 hours
• at the beginning of run, if this is True, read or set a redis key parametrized by the job name/slug/natural key
  • if already set, error out the job
  • if not set, set it
• at the end of run / the job execution (have to look into how this is handled for 2.x), unset the redis key