Verk 2.0

[edgurgel]

Verk 2.0 is coming with some really cool features!

We will change the way that queues are represented. Instead of using Redis Lists we will use the newest data structure added to Redis: Streams!

Streams have very interesting characteristics that will solve a few issues that would be very complicated to solve without them!

It will:

• Simplify the failure detection of other Verk instances;
• Provide a fair distribution of jobs to the Verk instances consuming a queue;
• Verk Web will eventually be standalone. So basically it will not be necessary to run alongside a Verk instance.

If you want to know more I wrote a blog post explaining in detail the reason why Verk will have a new major version: https://blog.gurgel.me/verk-2-0-and-redis-streams

There is a branch with most of the changes needed: https://github.com/edgurgel/verk/tree/streams

It's still a work in progress but I'm expecting the first release candidate to come out in a few weeks.

It fixes edgurgel/verk_web#76 & edgurgel/verk_web#69 & edgurgel/verk_web#26

[thousandsofthem]

Overall - great work! ✋

1. How about nodes that don't run any jobs? (e.g. then connecting to a server via iex to interact with the system (including enqueueing new jobs as well) but not to run any jobs by itself)? Very typical scenario for me, there are hacks to make it work with current version but these don't work very well

2. "all the jobs being run in the cluster" as mentioned in references

3. Any thoughts on self-introspection for a job? i.e. within a job get it's current ID, queue, retry number. atm no such information is available but could be useful if custom logic is necessary

[edgurgel]

Great points! Thanks for taking the time to reply!

1. How about nodes that don't run any jobs? (e.g. then connecting to a server via iex to interact with the system (including enqueueing new jobs as well) but not to run any jobs by itself)? Very typical scenario for me, there are hacks to make it work with current version but these don't work very well

I've done this using the distillery remote_console command and it works really well. Another way that can be done is setting the queues configuration to be empty, but I would consider this a hacky way. Interested in other ideas 🤔

But yeah I think I need to think more about the client side of the Verk functions like enqueue so that no hacks are needed. Thanks for bringing this up 👍

2. "all the jobs being run in the cluster" as mentioned in references

I'm not sure if I understood this second point, but I'm really excited the the view of all running jobs will be accessible through Verk Web (or any tools that use Verk.Queue)

3. Any thoughts on self-introspection for a job? i.e. within a job get it's current ID, queue, retry number. atm no such information is available but could be useful if custom logic is necessary

This is already achievable by using Verk.Worker.current_job() from the process that is running the job. More here: https://github.com/edgurgel/verk/blob/master/lib/verk/worker.ex#L14

We should probably add to the README.md as it's clearly not easy to find this information? 🤔

[thousandsofthem]

Another way that can be done is setting the queues configuration to be empty

Yes, i'm using it this way. Somehow doesn't work every time, so yeah, "hacky way"

2. Just agreeing on points mentioned in initial post

3. Good to know, thanks! Yep, didn't find the info, adding generally useful commands into README would be very useful 👍 maybe adding stuff like Verk.Queue.count!("queue_foo") as well

[jnylen]

Is a way to unique a specific worker or queue possible in the new update?

[edgurgel]

Verk 1.0 and Verk 2.0 can support unique workers. We just never had traction enough to build it I think.

[adamzapasnik]

Not sure where to ask about it, so I'm gonna ask here.

Would it be possible to change it to warn/error level?
My reasoning is that debug is used to debug the code, but this is unknown error that we are silencing and it's not visible in a log file, neither can Sentry report it. I don't want to turn on debug level, because imo debug isn't meant for this. Wdyt?

verk/lib/verk/workers_manager.ex

Line 165 in acda6d8

Logger.debug("Worker got down, reason: #{inspect(reason)}, #{inspect([mref, worker])}")

[edgurgel]

@DiodonHystrix, it's a bit unrelated to Verk 2.0 but I will answer here!

Why would like to know this type of information? What kind of information do you expect from it? 🤔

This is internal to the workers manager. If you want to track retries on your workers just go to the existing tracking events that you can receive: https://github.com/edgurgel/verk/tree/v1.6.3#error-tracking

[adamzapasnik]

Like I said, I'd like to see the errors in my logs (without a need to switch to the debug level).

Currently, I'm using Sentry to report any incidents and it doesn't get notified about the error, thus I have no clue that there was an error in my queue. It could be fixed by setting it to a debug level, but I don't think that's a correct solution.

I didn't notice error-tracking section, it could probably solve my problem, but I think it's gonna be an over-engineered solution.

[tlvenn]

Hi @edgurgel,

Hope you are doing well, is there any update you could share regarding your plan for the 2.0 ?
Thanks !

[tlvenn]

Hi @edgurgel ,

Hope you are doing well, it would be great to have an idea of what happened with the plan (and the branch for that matter) for the 2.0 version ?

Just trying to gauge if this is something that will still happen at some point or not.
Thanks in advance.

[edgurgel]

Hi @tlvenn,

At this stage it's indefinitely on hold because I don't have the time to invest on getting Verk 2.0 out with all the changes required to Verk Web.

[tlvenn]

Hi @edgurgel ,

Thanks for getting back to me. Would it be possible to envision a v2 without verk Web but supporting telemetry so people can plug on to it to monitor verk ? Maybe that would be a pretty decent intermediate state without asking you for too much ? It seemed to me that the v2 branch you had was pretty close to being stable or was there much more work to do ?

In any event, thanks for all your contributions and I wish you a merry Christmas.