From b1ccbc61305247513305692c144230e21c03281a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Esm=C3=A9=20Cowles?= Date: Thu, 7 Feb 2019 14:07:53 -0500 Subject: [PATCH] Decision: Queue priorities and guidelines --- architecture-decisions/0002-queues.md | 31 +++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 architecture-decisions/0002-queues.md diff --git a/architecture-decisions/0002-queues.md b/architecture-decisions/0002-queues.md new file mode 100644 index 0000000000..f592e3b140 --- /dev/null +++ b/architecture-decisions/0002-queues.md @@ -0,0 +1,31 @@ +# 2. Sidekiq Queues + +Date: 2019-02-07 + +## Status + +Accepted + +## Context + +We have background jobs that are processed by Sidekiq workers on several dedicated background processing +machines. The background jobs include a variety of different tasks from different sources: +* processing user requests (e.g., ingesting files, (re)generating derivatives) +* cleaning up after user actions (e.g., propagating state and visibility, removing files before deleting + their parent resource) +* refreshing metadata from Voyager/PULFA +* bulk-ingesting content +* routine cleanup (cleaning out guest users) +* fixity checking + +## Decision + +We will have three queues for processing background jobs: +1. `default` for processing user-initiated requests needed for ingesting and displaying objects +2. `low` for bulk processing, validation, cleanup, etc. +3. `super_low` for fixity checking and other long-term preservation actions + +## Consequences + +* Fixity checking from user-initiated ingest will be lower priority than bulk ingest or processing, and may + be significantly delayed when there are large volumes of materials being ingested.