add UniqueKey option to extend Unique's behaviour

[benjajaja]

Instead of (queue, type, payload) use any custom key. This allows to
enforce unique tasks not limited to same payload and/or type.

Tested with --redis_cluster flag.

[hibiken]

@benjajaja Thanks for opening a pull request!

If you are able to generate a unique key, then the TaskID option may be sufficient for your use case :) Please take a look at this wiki. And let me know if this doesn't work for your use case 👍

[benjajaja]

@hibiken TaskID indeed works for uniqueness, unless you also use retention - then a completed task will block tasks of same TaskID until retention expires.

So that's my use case: I want some retention, so that I can query the task for a bit after completion, but I also want to use uniqueness options. Unique on its own isn't sufficient, because my payload has something like userID, someOtherUniquePerTaskThingId. This means my tasks are always "unique".

[hibiken]

I see. So I'm guessing you want to use this userID as the unique key to ensure that only one task for a given user is pending at any given time? If so, can we expect this task to happen every x hours? Can we use that knowledge to come up with a unique task ID that meets your requirement? For example, something like <user_id>:<yyyy-mm-dd-hh> to ensure that only one task exists for a user at any given hour. Would that work for you?

[benjajaja]

@hibiken right now I'm using the TaskID to ensure a single task of given criteria at a time. But then I can't add retention, because the task will block even though it's completed. I want the retention so that I can properly check if a task has been completed, beyond "not existing".

[benjajaja]

BTW I started by trying to add a "key" option to Unique, however it got awkward when it would be the only option wrapping two values.

[hibiken]

Presumably you want to generate a UniqueKey string with some TTL, correct?
My previous comment was suggesting that you could generate a unique task ID that captures these two values.

For example, if you want to use a unique_key of (user_id) and TTL of 1 hour then you could generate a unique task ID in a format <user_id>_<yyyy-mm-dd-hh:00:00> (e.g. "user123_2022-02-28-15:00:00").

This way, all tasks fall under the same time window will be deduplicated, but a task in the next time window won't be blocked by the previous task even if that previous task is still in the queue as a completed task (actually it doesn't matter which state the previous task is in).

[benjajaja]

Presumably you want to generate a UniqueKey string with some TTL, correct? My previous comment was suggesting that you could generate a unique task ID that captures these two values.

No, Unique already has its TTL, which I only care about to prevent locking some type of task forever.

I do want retention, so that I can call GetTaskInfo on a completed task (otherwise this returns "not found" after completion). Retention does not play well with uniqueness-by-task-id, as it blocks new incoming tasks.

For example, if you want to use a unique_key of (user_id) and TTL of 1 hour then you could generate a unique task ID in a format <user_id>_<yyyy-mm-dd-hh:00:00> (e.g. "user123_2022-02-28-15:00:00").

This way, all tasks fall under the same time window will be deduplicated, but a task in the next time window won't be blocked by the previous task even if that previous task is still in the queue as a completed task (actually it doesn't matter which state the previous task is in).

A time window is interesting, but I am not bound by any time windows. It would also allow for duplicate tasks, if the tasks run longer than the window. Thank you for coming up with this, but it won't really fit my case.

BTW, as I said I am using unique task-ids so far and it works, it's just that I can't retain them for more than some seconds, as that block newer tasks. It's not a big deal, but it would be nice to retain them longer, to do some more analysis.

[hibiken]

I see. Thank you for providing more details.

I've seen similar feedback/discussion around Unique option such as #369 and #376.

It seems that we need to allow user to specify:

• Whether the uniqueness lock should be released on task completion
• Unique key used for the uniqueness lock

in addition to the TTL value.
I'll take this feedback and come up with a solution that addresses these issues together.

[benjajaja]

Ok, thanks! I'll keep an eye on it.