Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add worker queue support to job class and model #2421

Merged
merged 33 commits into from
Sep 20, 2022

Conversation

gsnider2195
Copy link
Contributor

@gsnider2195 gsnider2195 commented Sep 14, 2022

Closes #2064, Closes #2065

What's Changed

Add new job model and class property named task_queues to present a list of supported queues for each job. This property defaults to an empty list which only allows the job to run on the default queue. The property may be defined on the class and can be overridden similar to other job properties like read_only and approval_required. The list of queues will be presented in the job run form to select the queue the job will be sent to. The queue can also optionally be specified in an api job run request, defaulting to the first queue in the list if the worker_queue parameter is not passed to the api call.

  • Add setting CELERY_TASK_DEFAULT_QUEUE with a default value of default to nautobot config
  • Add task_queues job class meta property
  • Add task_queues job model field
  • Add task_queues_override job model field
  • Add new model fields to job edit form
  • Add new model fields to job detail page
  • Add task queue field to job run form
  • Add task_queue field to job run API endpoint
  • Add task_queue to JobResult.job_kwargs to store the queue for populating the form when clicking re-run on a job result
  • Add example to example plugin jobs
  • Update docs

Job Run form

image

Job Edit form

image

Job Detail

image

TODO

  • Explanation of Change(s)
  • Attached Screenshots, Payload Example
  • Unit, Integration Tests
  • Documentation Updates (when adding/changing features)
  • Example Plugin Updates (when adding/changing features)
  • Outline Remaining Work, Constraints from Design

@gsnider2195 gsnider2195 linked an issue Sep 14, 2022 that may be closed by this pull request
@bryanculver bryanculver linked an issue Sep 14, 2022 that may be closed by this pull request
Copy link
Contributor

@jathanism jathanism left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:%s/worker.queue/task.queue/g
:%s/Worker queue/Task queue/g

examples/example_plugin/example_plugin/jobs.py Outdated Show resolved Hide resolved
nautobot/docs/additional-features/jobs.md Outdated Show resolved Hide resolved
nautobot/docs/additional-features/jobs.md Outdated Show resolved Hide resolved
nautobot/extras/api/views.py Outdated Show resolved Hide resolved
nautobot/extras/api/views.py Outdated Show resolved Hide resolved
nautobot/extras/forms/forms.py Outdated Show resolved Hide resolved
nautobot/extras/forms/forms.py Outdated Show resolved Hide resolved
@@ -649,6 +649,9 @@
# Instruct celery to report the started status of a job, instead of just `pending`, `finished`, or `failed`
CELERY_TASK_TRACK_STARTED = True

# Default celery queue name that will be used by workers and tasks if no queue is specified
CELERY_TASK_DEFAULT_QUEUE = os.getenv("NAUTOBOT_CELERY_TASK_DEFAULT_QUEUE", "default")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth calling out explicitly in the release-notes?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a section in the release notes overview on this feature and it includes an important admonition calling out this changed setting.


Environment Variable: `NAUTOBOT_CELERY_TASK_DEFAULT_QUEUE`

Default: `'default'`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a change from prior releases? It was celery before, right? Is there anything we need to call out in the release-notes about the upgrade changing this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a section to the release notes overview. See comment above

nautobot/extras/tests/test_views.py Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
5 participants