Add the ability to set a priority

[queicherius]

Thanks to the ordering of the file system, by prefixing the filename with a number, we get a priority functionality for (essentially) free.

[alexpusch]

This is a cool idea but I have some issues with it:

• We cannot give real assurances that a message with some priority will be processed fully before an other message with a lower priority. This feature can be useful even without a hard assurance but this limitation should be documented.
• I would rather the message to remain a simple data object without metadata fields. Please pass the priority in options object of publish.

Thanks for the contribution!

[alexpusch]

Another issue is that you relay on that fs.readDir will return the files sorted alphanumerically. While this might be true for now I would rather to have an explicit sort in getNextMessageId

[queicherius]

We cannot give real assurances that a message with some priority will be processed fully before an other message with a lower priority. This feature can be useful even without a hard assurance but this limitation should be documented.

Isn't the "reading" part always be in the order of priority (or before this PR in order of timestamp)? I am not quite sure I understand the limitation. Is this about parallel processing, in which case both could start at the same time and the low priority one could finish first?

I would rather the message to remain a simple data object without metadata fields. Please pass the priority in options object of publish.

That's cool, I'll update this PR soon.

Another issue is that you relay on that fs.readDir will return the files sorted alphanumerically. While this might be true for now I would rather to have an explicit sort in getNextMessageId

Damn, I didn't realize that. That indeed is an issue already: 1 - 10 - 3 - 50 is the order on OSX. A possible solution would be limiting the priority to 1-9 and emitting the sorting - that should work with the native alphasort that is used for ordering by timestamp already.

Sorting the files manually seems like a performance killer for many messages to me. Speaking of that tho, it looks like readDir is not sorted on Windows. I also couldn't find a function to just "grab the first filename based on order" instead of the full directory either. 😒

---

Edit: Tested sorting for 10.000 messages, took about 75ms per sort on average on my machine.

[alexpusch]

10,000/75ms might sound fast, but for the current implementation we would need to sort for each consumed message separately. Lets say that more messages are coming it and the queue is steady at 10000. Consuming the first 10000 would take 750 seconds.

There might be some solutions for this -

1. Split the pending directory into sub-directories by priority, this way we won't have to worry about readDir ordering as we'll read each priority folder individually.
2. keep the entire list of files in each consumer process and have a file watcher keep track of new files. Inserting a new element into a sorted list is an O(logN) operation.

It seems that it's not 'free' as we previously thought. You are welcome to try to re-implement it, I'll keep this PR open.

[queicherius]

Alright, changes I made:

• Added "no assurances" into the README
• Changed the priority to be passed in as an option via publish(message, options) instead of in the message (and moved the default into queuelite instead of storage)
• Enforced priority to be between 1 and 9. This allows us to get away with not additionally sorting, since it works the same way the current timestamp sorting works (alphabetically sorted with same length for all file names)

This does not solve the issue of readDir not being sorted on Windows, but that's something else to tackle and unrelated to this feature.

[alexpusch]

Add an _ between to priority and the nanoSeconds. Just for human readability

[alexpusch]

You forgot up update the docs about the move of priority into an options object

[queicherius]

Fixed both issues. 🙂

[alexpusch]

Awesome, thanks!