Production usage reports

[saurabhnanda]

No description provided.

[saurabhnanda]

@mrputty @TomMD @jkachmar @garethstokes Would you be kind enough to to share your thoughts on the following questions:

• How are you using odd-jobs?
  • production / work-related project
  • personal side-project
  • not using yet
  • something else?
• If you are using odd-jobs, about how many jobs per hour (or day, or week) are you pushing through it?
• Did you find the documentation for odd-jobs to be adequate? Were you able to get up & running quickly?
• Are you using the web UI that comes bundled with odd-jobs? Is it good enough?
• Any bug-fixes or features that you'd like to see in odd-jobs

[mrputty]

We are evaluating odd-jobs for use in orchestrating data retrieval and processing for a data-oriented startup. So far things are looking promising, and our experience has been positive. We are currently doing some initial scale testing with it. Assuming we go forward with odd-jobs, this would be for production use in a commercial web site.

At full scale, we would be processing many thousands of jobs per day. These would be a mix of on-demand jobs triggered by individual customer actions, as well as offline jobs to maintain a large set of data gathered from multiple sources. This would include both shorter, higher-frequency jobs and long running low-frequency jobs.

The primary appeal of odd-jobs to us is that it makes it easy to integrate a PostgreSQL-based job queue into our existing infrastructure. Avoiding adding yet another component in the form of a separate queuing system is a huge advantage for us. The code is written in a direct and comprehensible style, which convinced us that we could take it over if it were to head off in a wildly different direction or to become abandoned.

There was certainly sufficient documentation to get us started. The odd-jobs library fit fairly well into the library and application structures we had already planned. The library is a bit opinionated about how to handle things like logging, but that wasn't a problem for us in practice.

We are using the web UI for testing. If we move forward with odd-jobs we will either need to augment this UI with additional features (probably things mostly specific to our use-case) or we'll just replace it entirely. I'm really not sure yet on the UI front. The provided UI is certainly handy for evaluation and testing though.

Here are some general UI features we would like to see added:

• Delete job(s) and for failed jobs at least, delete all.
• Save "named" jobs for future manual re-execution
• Job scheduling

Much more importantly, I would like to have the ability to define resource-limited jobs.

In the longer term, I think the library would benefit from a more abstract approach to database interaction. I also have a feeling that explicit transaction handling may become necessary at some point, and if that happens, I think it would be valuable to expose the odd-jobs database connection/transaction to callback functions for the client to allow the client to extend bookkeeping functionality safely.

I appreciate the work you have done in creating this, and am thankful that you have decided to share it with us.

Thanks!

[saurabhnanda]

We are currently doing some initial scale testing with it.

@mrputty if you are writing benchmarks, would it be possible to merge them upstream? If not the code, then at least the results.

The library is a bit opinionated about how to handle things like logging, but that wasn't a problem for us in practice.

This brings me to a related question - at scale, how do you plan to monitor the queue?

[mrputty]

I have no immediate plans to implement isolated benchmarks for just this library. So far we are using odd-jobs only for offline, non-performance critical activities, with mostly long-running jobs. In this context, we would not notice performance issues in the job queue unless it was very, very, very bad. Right now, monitoring is very crude, and is being done via AWS Cloud Watch and manual monitoring of logs.

In the relatively near future, we will start processing online jobs where performance will be much more important. However, this will still likely be monitored via system level tools, rather than via explicit benchmarks at the odd-jobs level.

As part of this, we will be putting in some work to more directly monitor latency and throughput in the queuing layer, and I'm happy to share some performance data when we have it. Also, as we go through the implementation of #38, I will certainly think some more about what benchmarks seem useful.

[saurabhnanda]

@kanagarajkm It would be great if you could share how you're using odd-jobs. I've listed down some questions at #44 (comment) , but would be happy to head any feedabck.

[kanagarajkm]

@saurabhnanda first of all, thanks for this awesome library, it works nicely.

How are you using odd-jobs?

We are in the testing phase, will be rolling out into production probably in a week time.

If you are using odd-jobs, about how many jobs per hour (or day, or week) are you pushing through it?

To start with, we are moving one of our long-running task into odd-jobs. For now, its going to be few jobs in a day. Once we move all the other ones, i could be more.

Did you find the documentation for odd-jobs to be adequate? Were you able to get up & running quickly?

I felt the docs are good. I could get it up and running in a couple of hours time.

Are you using the web UI that comes bundled with odd-jobs? Is it good enough?

We are not using the ui extensively. We already have progress monitoring for the task on our side.

Any bug-fixes or features that you'd like to see in odd-jobs

One of it is, setting max retry attempts per job.

[angedid]

Hi,
I am developping an appliction, and the application has a notification module. This module have to send e-mail to users.
We want to dettach the operation of sending the e-mail from the flow of the the process of sending the email by using a background job. We look on internet and discover the best Odd-Jobs library which has many features dealing with job on Haskell.
Supprisingly when i try to install, i get this error attached to this post

.
1- added to my package.yaml configuration file

• base >= 4.7 && < 5
• aeson
• text
• postgresql-simple
• direct-daemonize
• timing-convenience
• odd-jobs
  2- i added to my stack.yaml configuration file
  extra-deps:
• direct-daemonize-3.1
• timing-convenience-0.1
• odd-jobs-0.2.2

Please con you tell me how use odd-jobs in my stack project?
Thank in advance