Skip to content

Commit

Permalink
Updated documentation for 3.0
Browse files Browse the repository at this point in the history
  • Loading branch information
msavin committed Jun 24, 2018
1 parent 8937d0c commit 031fdf5
Show file tree
Hide file tree
Showing 3 changed files with 41 additions and 34 deletions.
20 changes: 8 additions & 12 deletions CHANGES.md
Expand Up @@ -6,11 +6,11 @@

Meteor evolved application development by going from past-time by default to real-time by default. The Steve Jobs package takes the next leap forward by letting your application run in future-time. 😁

The Steve Jobs package allows you to schedule tasks at a future date, in a way that is friendly to the Meteor framework. It provides a wide set of tools to help you get creative, while automating everything behind it.
The Steve Jobs package allows you to schedule tasks at a future date, in a way that is friendly to the Meteor framework and developer. It provides a wide set of tools to help you get creative, while automating everything behind it.

## Repeating Jobs (!!!)

The most requested feature was also the trickiest to implement, due to how the rest of the queue was built. I didn't want to hack in a completely new set of features on top of what already exists. Fortunately, it turns out there's a really simple way to achieve this effect while staying aligned with how the rest of the queue works.
The most requested feature was also the trickiest to implement, due to how the rest of the queue was built. I didn't want to hack in a completely new set of features on top of what already exists. Fortunately, there's a really simple way to achieve this effect while staying aligned with how everything works. The idea is to enable jobs to replicate themselves before they are resolved.

```javascript
Jobs.register({
Expand Down Expand Up @@ -53,9 +53,9 @@ The job above will try to get data through some means, and then:
- If the insert is not successful, the job will `reschedule` itself to run again in 30 seconds
- If it does not receive the data as expected, it will `reschedule` itself to run again in 5 minutes

`this.replicate` basically replicates the job and its arguments, while allowing you to set a new configuration for it. This enables you to repeat a job as many times as you wish, while dynamically setting the conditions for how it should happen.
`this.replicate` basically replicates the job and its arguments, while allowing you to set a new configuration for it. This enables you to repeat a job as many times as you wish, and to dynamically setting the conditions for how or when it should happen.

It's important to use `this.replicate` instead of `this.reschedule` to repeat a job because each job keeps track of its history. If you reschedule a job too many times, it's document would become humongous, which could have consequences. Additionally, using `this.replicate` makes it easier for your clear resolved job documents in the future.
It's important to use `this.replicate` instead of `this.reschedule` to repeat a job because each job keeps track of its history. If you reschedule a job too many times, it's document would become huge, and that might have consequences. Additionally, using `this.replicate` makes it easier for your clear resolved job documents in the future.

For jobs that that run very frequently, you can also use the new `this.remove` feature to remove the document from the database rather than just mark it as complete.

Expand Down Expand Up @@ -88,15 +88,13 @@ With `log`, you can configure how your application should log items. By default,

## Smarter MongoDB Behavior

Of all the pleasures that MongoDB offers, peace of mind is not one of them.
Of all the pleasures that MongoDB offers, peace of mind is not one of them. Or maybe that's it?

First, it can take a bit of time for the writes to be reflected in the reads, and that could make jobs run twice.

This was resolved by adding an extra condition to the MongoDB queries: the document must meet this criteria, _and_ its `_id` must not be that of the job that had just run.

Second, it turns out that MongoDB's upsert function may not be so reliable - if you run a few upserts at once, MongoDB might just insert all the documents. This is probably related to the first issue. This created a problem with the `dominator` function, as the queue might get confused as to which server is active.

This has been resolved by making the `serverId` field unique. It looks like MongoDB becomes more diligent when you set a unique field.
Second, it turns out that MongoDB's upsert function may not be so reliable - if you run a few upserts at once, MongoDB might just insert all the documents. This is probably related to the first issue. This created a problem with the `dominator` function, as the queue might get confused as to which server is active. This has been resolved by making the `serverId` field unique.

## What's Next?

Expand All @@ -108,8 +106,6 @@ I'm excited about transactions coming MongoDB 4.0. Along with Write Concerns, Re

MongoDB 4.0 is coming this summer, so I will evaluate then whether to keep evolving the project or to simply maintain what it does now.

The idea is, this could grow into a reliable queue that can run many jobs at once and scale horizontally. It would not be the fastest solution, but it may be so reliable, scalable and developer friendly, that speed would seem overrated.

If that were achieved, the next step would be to build an interface and REST API to let anyone run this as a standalone service.
The idea is, this could grow into a reliable queue that can run many jobs at once and scale horizontally. It would not be the fastest solution, but it may be so reliable, scalable and developer friendly, that speed would seem overrated. If that were achieved, the next step would be to build an interface and REST API to let anyone run this as a standalone service.

**With that said, your help can go a long way!** I'm looking for someone to design and implement the testing strategy, and for help in taking on the challenge of scaling the queue to run many jobs at once. If you can help with this, or if you have a different angle, do reach out!
**With that said, your help can go a long way!** I'm looking for help with the testing strategy, and for scaling the queue to run many jobs at once. If you can help with this, or if you have a different angle, do reach out!
53 changes: 32 additions & 21 deletions DOCUMENTATION.md
Expand Up @@ -103,23 +103,24 @@ Jobs.run("sendReminder", "jony@apple.com", "The future is here!", {
```

The configuration object supports the following inputs:
- **`in`** - Object
- The `in` parameter will schedule the job at a later time, using the current time and your inputs to calculate the due time.
- **`on`** - Object
- The `on` parameter override the current time with your inputs.
- **`in` and `on`** - Object
- The supported fields for in and on can be used in singular and/or plural versions:
- millisecond, second, minute, hour, day, month, and year
- milliseconds, seconds, minutes, hours, days, months, and years
- The date object will be updated in the order that is specified. This means that if it is year 2017, and you set `in` one year, but `on` 2019, the year 2019 will be the final result. However, if you set `on` 2019 and `in` one year, then the year 2020 will be the final result.
- **`priority`** - Number
- The default priority for each job is 0
- If you set it to a positive integer, it will run ahead of other jobs.
- If you set it to a negative integer, it will only run after all the zero or positive jobs have completed.
- **`date`** - Function
- Provide your own date. This stacks with the `in` and `on` operator, and will be applied before they run.
- **callback** - Function
- Run a callback function after scheduling the job

- **`in`** - Object
- The `in` parameter will schedule the job at a later time, using the current time and your inputs to calculate the due time.
- **`on`** - Object
- The `on` parameter override the current time with your inputs.
- **`in` and `on`** - Object
- The supported fields for in and on can be used in singular and/or plural versions:
- millisecond, second, minute, hour, day, month, and year
- milliseconds, seconds, minutes, hours, days, months, and years
- The date object will be updated in the order that is specified. This means that if it is year 2017, and you set `in` one year, but `on` 2019, the year 2019 will be the final result. However, if you set `on` 2019 and `in` one year, then the year 2020 will be the final result.
- **`priority`** - Number
- The default priority for each job is 0
- If you set it to a positive integer, it will run ahead of other jobs.
- If you set it to a negative integer, it will only run after all the zero or positive jobs have completed.
- **`date`** - Function
- Provide your own date. This stacks with the `in` and `on` operator, and will be applied before they run.
- **callback** - Function
- Run a callback function after scheduling the job

### Jobs.execute

Expand All @@ -137,7 +138,8 @@ Jobs.execute(docId)
Jobs.reschedule(jobId, {
in: {
minutes: 5
}
},
priority: 99999999
})
```

Expand All @@ -157,7 +159,7 @@ Jobs.replicate(jobId, {

### Jobs.start

`Jobs.start` allows you start all the queues. This runs automatically unless `autoStart` is reconfigured through `Jobs.configure`. If you call the function with no arguments, it will start all the queues. If you pass in a String, it will start a queue with that name. If you pass in an Array, it will loop over the items in it, and treat them like a string.
`Jobs.start` allows you start all the queues. This runs automatically unless `autoStart` is set to `false`. If you call the function with no arguments, it will start all the queues. If you pass in a String, it will start a queue with that name. If you pass in an Array, it will loop over the items in it, and treat them like a string.

```javascript
// Start all the queues
Expand Down Expand Up @@ -230,7 +232,16 @@ Jobs.cancel(jobId)

### Jobs.clear

`Jobs.clear` allows you to clear all or some of the jobs in your database.
`Jobs.clear` allows you to clear all or some of the jobs in your database. It supports `state` for selecting a job state, which can be `pending`, `success`, or `failure`, or `*"`to select all of them.

You can add the `name` arguments to specify a specific queue. You can also call an optional callback.

```javascript
var state = "pending";
var name = "sendEmail";
var cb = function (r) { console.log(r) }
Jobs.clear(state, name, cb)
```

### Jobs.remove

Expand All @@ -242,4 +253,4 @@ Jobs.remove(docId)

### Jobs.collection

`Jobs.collection` allows you to access the MongoDB collection where the jobs are stored. Ideally, you should not require interaction with the database directly. But hey, we're developers.
`Jobs.collection` allows you to access the MongoDB collection where the jobs are stored. Ideally, you should not require interaction with the database directly, but hey, we're developers. If you find a case where this is necessary, let me know.
2 changes: 1 addition & 1 deletion README.md
Expand Up @@ -12,7 +12,7 @@ Run scheduled tasks with Steve Jobs, the simple jobs queue made just for Meteor.
- Failed jobs are retried on server restart
- No third party dependencies

**The new 3.0 features repeating jobs and more improvements.** It can run hundreds of jobs in seconds with minimal CPU impact, making it a reasonable choice for many applications. To get started, check out the <a href="https://github.com/msavin/SteveJobs/wiki">**documentation**</a> and the <a href="#quick-start">**quick start**</a> below.
**The <a href="https://github.com/msavin/SteveJobs..meteor.jobs.scheduler.queue.background.tasks/blob/master/CHANGES.md">new 3.0</a> features repeating jobs and more improvements.** It can run hundreds of jobs in seconds with minimal CPU impact, making it a reasonable choice for many applications. To get started, check out the <a href="https://github.com/msavin/SteveJobs..meteor.jobs.scheduler.queue.background.tasks/blob/master/DOCUMENTATION.md">**documentation**</a> and the <a href="#quick-start">**quick start**</a> below.

## Developer Friendly GUI and API

Expand Down

0 comments on commit 031fdf5

Please sign in to comment.