Add ability to schedule Git sync

[jedelman8]

Environment

• Python version 3.9
• Nautobot version: 1.0.0b3

Proposed Functionality

This proposal is to migrate Git repository sync to be a Nautobot Job to be able to leverage the scheduling features therein.

Use Case

As a user, I want to schedule Git repository syncs so that my export templates, config contexts, and job are always up to date.

Database Changes

TBD

External Dependencies

TBD

Requirements (In-Scope for MVP)

• Schedule Git syncs to be performed on a regular basis
• Migrate Git sync tasks to be execute as a Job
• Establish a pattern for how system Jobs are provided
• System jobs will be enabled by default
• Migrate permissions from Git sync task to job
• Job will take repo as a parameter (therefor a schedule sync can sync a particular repo at a time)

Non-Requirements (Out of Scope for MVP)

• No "sync all repos" option
• Simplify (or eliminate) the Job Result related object pattern

TODO

1.4

Future

• Implement "Hello World" System Job (establishing System Job pattern) #1725
• Migrate Git Repo Sync to be a System Job #1726

[jedelman8]

Need to breakup into a few issues (in this priority)

• Scheduled Jobs
• Scheduled Git Sync
• Scheduler to be used by plugins
• Templated Jobs
• Webhooks for Jobs

[chadell]

just for clarification, when you say Scheduled Jobs and Scheduler to be used by plugins, what about Scheduled Jobs defined by plugins?

[itdependsnetworks]

While not explicit, it is implied. Since Plugins can create jobs, and if jobs can be scheduled, you would get both.

[jedelman8]

Once #374 gets in, we can revisit scheduling other types of operations.

[jedelman8]

While this was in near-term, I'd like to open it up for discussion to get feedback on priority of scheduling other operations. Anyone have feedback/input?

[chadell]

I think the Jobs scheduler is an interesting feature to ease the periodic trigger of Jobs (instead of relying on an external Cron job that triggers the API).
On another hand, not sure if it's already possible, but implementing part of this feature internally would be a good starting point, that later could be exposed to the user. I'm thinking on something like this issue, where some periodic evaluation of an internal logic.

[glennmatthews]

One possibility here is to create a "system" Job specifically for triggering the sync of a given GitRepository - this job could be built in or could be provided as part of a standard nautobot-jobs Git repo of its own if we wanted to. Either way, as a Job like any other, it could be scheduled just like any other job.

[jathanism]

In reference to this comment, I wanted to clarify a few things from my perspective.

Need to breakup into a few issues (in this priority)

• Scheduled Jobs

This is coming in v1.2.0, so it's already accounted for, except for the caveats detailed below.

• Scheduled Git Sync

This is a very tangible use-case for a good reference implementation of tying a model instance to a recurring schedule. While that is certainly a good start, we would need to consider how this plays out for extending this to other types of "regularly occuring change events".

• Scheduler to be used by plugins

Same as scheduled jobs, the scheduler (Celery beat) will be usable by any caller, however we need to assert the UX and proper documentation on it (assuming we haven't already). We need to be very cautious about talking about Nautobot Job (capital J) scheduling which is a layer built on top of Celery, and the underlying scheduling performed by "raw" Celery background tasks.

This controlled vocabulary is critically important to avoid confusion.

There are two rising patterns here in my mind:

1. As of v1.1.0, any plugin can define as many Celery tasks as it wants, and directly call them using Python e.g. foo.delay() to submit the task to the Celery task queue and have it be picked up by a worker. But these are not Jobs. They are Celery tasks. This is effectively same as executing background tasks using RQ in Nautobot <1.1.0.
2. As of v1.2.0, any plugin can do the same as item 1, but it must be done using database objects and because we have somewhat tightly-coupled Nautobot Job objects to the way we implemented the database scheduler and therefore schedules stored in the database, it's not going to be as straightforward as we think, unless we completely align around Job instances as the one "true" way to do background task executions. (Reference: https://django-celery-beat.readthedocs.io/en/latest/)

It's technically possible for an alternate scheduler to be used that performs "raw" Celery task execution, but it's outside of the scope of the design we're moving towards and because of our bespoke implementations of job schedules (NautobotScheduleEntry), ScheduledJob, and JobResult objects as well as a custom NautobotDatabaseScheduler, we have to consider all of these factors into the UX and functional requirements.

• Templated Jobs
• Webhooks for Jobs

These two are not going to be possible without making Job a first class managed model and not just an unmanaged stub model used merely for providing baseline add/change/delete permissions. Part of any more robust extensions to jobs means that we'll need to make this a priority.

Distributed computing is hard.

[jedelman8]

Updated title of issue re-scope it.
If future items are still needed, a new issue will be needed.

[bryanculver]

Side-effect of #3545 which is needed for #3228

[lampwins]

This is actually possible as for 2.0