Use lower priority queue for large imports

[jacksonh]

When users import large libraries we should switch over to the low priority queue to keep performance predictable for all users.

To do this we can add a rate limit type check to the import path. In before adding to the task queue based on some heuristics we can decide to use either the normal or the low priority task queue. There is a task queue omnivore-prod-queue-low that dispatches to the cloud function at a lower rate (3 per second vs 500).

To do this we will need to create a low priority queue in the demo environment also.

As a first pass at our logic I think we can do:

5 articles added in the last minute: use low queue
default: use normal queue

You can see how we pass a priority param to the cloud task function here:

omnivore/packages/api/src/utils/createTask.ts

Line 18 in f7e3a16

const createHttpTaskWithToken = async ({

[jacksonh]

@sywhb not sure if there is a good way with elastic to do something like 'select count(*) from links where user_id ... and DATE(created_at) > 1hr ago

[sywhb]

@sywhb not sure if there is a good way with elastic to do something like 'select count(*) from links where user_id ... and DATE(created_at) > 1hr ago

@jacksonh Yeah. Elastic has this aggregation query to get count

[sywhb]

We have a low priority queue in the demo environment: omnivore-demo-queue-low