Backburner Specific Admin Panel

[nesquena]

After chatting with @kr today, have a few ideas to jot down for a first class backburner admin frontend panel. First I would want visibility into the jobs in the ready queue as well as a view showing failed jobs with backtraces and a way to 'kick' a job, all available via a sinatra web UI.

In particular, some ideas on how to do this.

Ready Jobs

Create a sinatra view that reserve(0) 100 jobs and then collects the information and immediately releases them. Aggregate the 100 jobs and display them in the sinatra view. By reserving and releasing immediately, we can create a view for both development and production by showing the next 100 jobs.

Buried Jobs

I would want a place for buried jobs where you can see the next 100 buried jobs across all queues perhaps. For this we could have a special tube called 'failed-jobs' that is special and used by the admin panel. In Backburner, everytime a job fails and is buried, we can then insert the job into the 'failed-jobs' queue before burying it. Then when we want to show last 100 failing jobs simply reserve(0) and release 100 jobs from that tube. In this way we can have a buried jobs list. We can even have a button to kick the job (which will then kick the job based on the real id). The way I was thinking about it, the failed-jobs tube could have jobs with this format:

{ "job-id" : 1234, "tube" : "foo", "backtrace" : "...", "tries" : 3 }

and then this can be used to display the buried jobs in a table.

Implementation Thoughts

Also, I love beanstalkd_view and it's such a great start, I wonder @denniskuczynski if you would have any interest being a core contributor to backburner and helping out with the admin panel. Ideally we could have a familiar feel to https://github.com/defunkt/resque#section_The_Front_End and obviously be unabashedly clear we are inspired by that project's interface as a point of reference (as well as beanstalkd_view itself).

[denniskuczynski]

@nesquena Sure, I'd love to help you guys with an admin panel.

I don't know if it's smart to try to try to model the Resque/Sidekiq admin panels too closely though.
Since those tools use Redis, it's a little easier to have query functionality, since the web client can just read from the same keys as the worker processes.

Beanstalkd is more of a pure work queue, you can only really query with the peek commands.

I'm a little scared about reserving jobs for display purposes, because it could affect the ordering or the processing speed (one of beanstalkd's big advantages). For example, if you reserve the first 100 jobs from a queue, the 101st job might be processed first by a worker, even if the 100 jobs are immediately released. This might be less of a problem for a buried job queues, since those already failed.

I was experimenting in beanstalkd_view of trying to peek jobs by id ranges (based on guessing a range from the id returned by peek-ready). But that's still kind of wonky.

I kind of like what this client is doing:
https://github.com/ptrofimov/beanstalk_console

I think the focus of the admin interface functionality should try to lean in the directions that beanstalkd does best.

Perhaps instead we could just display a big status table of all the backburner job queues with the queue statistics and in-depth details of the peek-ready job. (This could auto-refresh through javascript to give the users a sense of the internal processing.)

With a slick error interface -- having access to all the ready jobs might not be that necessary of a feature. I know I'm usually more interested in just the fact that the system is churning through jobs and only digging into the details of the jobs that fail or need to be retried.

Let me know what you think. Maybe @kr has some thoughts on good methods to query the state of beanstalkd jobs for administration?

[nesquena]

Yeah I can see what you mean. Seeing a bird's eye view of the backburner work queues and seeing how many jobs, other stats etc is definitely a great overview. And being able to explore buried jobs and inspect / kick those would definitely be more important then viewing ready jobs for sure.

I would love your help putting together the first admin interface for backburner and definitely appreciate your feedback on how to make an admin panel that best fits with beanstalkd's strengths.

[kr]

I think it's fine to reserve and release a bunch of jobs just
so you can display them. If you have more than one worker,
they're not guaranteed to run in order anyway.

[nesquena]

Good point, by design the order of the jobs processed really shouldn't have any impact.

[denniskuczynski]

Yeah, I guess that's true. It's more a theoretical issue than a practical one.

When I get a chance, I'll see about creating a fork/branch with a basic skeleton/prototype.

[nesquena]

Sounds great, thanks @denniskuczynski