Optimize find and modify

[lushc]

Rebased & included the fixes in #300 (comment).

As per @will123195's original comment this a breaking change (e.g. for Agendash) since an unlocked job now has its lockedAt field set to the Unix epoch rather than null, plus the field is always in the document so any $exists: false queries won't work. Likewise with the disabled field.

Overall, significant performance gains should be made by getting rid of the recursive lock warnings. With 2,000,000 jobs in the database, query time has gone from ~4000ms to ~400ms. Deviating from the original PR is a 2nd query introduced to fix the failing test highlighted in my comment, but it also appears to fix #410 where unfinished jobs weren't being resumed. For the execution plan to use the index effectively the $or has to remain top-level, so the query duplication is unavoidable, but thankfully it's low cost (~50ms).

[coveralls]

Coverage increased (+0.02%) to 91.618% when pulling 5fdbf66 on feature/optimize-find-and-modify into d8e1da3 on master.

[simison]

Assigned this first to 0.10.0 milestone like the original PR was, but I realised this needs to be in 1.0.0 since it has a breaking change.

[emhagman]

@lushc I am comfortable with Agendash as I've forked it for my own use already. I can fix the breaking change and make it backwards compatible if you'd like.

Sent from my Google Nexus 6P using FastHub

[simison]

@joeframbach I added you here in case you wanna have a look.

[loris]

Very cool @lushc, one question/idea though about your $or query:
The second query (the one rerunning expired jobs) seems to be something supposed to happen rarely, whereas the _findAndLockNextJob is ran pretty often (actually every 5 seconds by default, and in my case every second, because my use-cases needs very responsive queues). So I'm wondering if we should not have a second loop (setInterval) running less frequently (for example, every 10mn) dedicated to reset expired jobs to a state where they will be catch by the next iteration of _findAndLockNextJob.
What do you think?

[coveralls]

Coverage increased (+0.02%) to 91.618% when pulling 80afd02 on feature/optimize-find-and-modify into d8e1da3 on master.

[lushc]

@loris yeah it's quite a rare scenario so it's definitely less than ideal having the 2nd query to handle it being called so often, even if the performance impact is low. I'll have to experiment with the implementation of a separate interval, although one issue that comes to mind is respecting a definition's lock lifetime (and so passing the "runs a one-time job after its lock expires" test). We'll have to iterate through each definition to work out the job's lock deadline, but if the interval is infrequent then there's the chance the job gets picked up later than it should have. However, with it being an edge-case maybe that's OK? Especially if this "unstuck" interval is configurable (which it'll need to be to pass the test) and documented. In the meanwhile, if you could test the performance of this branch to see how it fits your use-case that'd be very much appreciated 👍

@OmgImAlexis saw your comment in the other PR, would be interested to hear if you're still getting those test timeouts? I've tried to replicate and couldn't on MongoDB 2.6.10.

[joeframbach]

Going forward in mongodb, ensureIndex is deprecated in favor of createIndex. Is this an appropriate time to also make this change?

[simison]

Might be a separate PR is better to keep this one more focused?

[loris]

Agree to keep it focused. IMO, the ensureIndex/createIndex should be removed in favor of a simple instruction in the setup in README.md

[simison]

Follow ups: #490

[coveralls]

Coverage increased (+0.05%) to 89.951% when pulling f0a8291 on feature/optimize-find-and-modify into 955fd0c on master.

[coveralls]

Coverage increased (+0.6%) to 80.56% when pulling 1fcc15d on feature/optimize-find-and-modify into 2411cb3 on master.

[oscar-b]

@lushc Any ETA on this PR and v1.1.0? Got a feeling this would help when used with Azure CosmosDB which currently consume quote a bit of RU (CosmosDB billable unit for how much capacity is being used) since the query is quite complex.

[lushc]

I've revisited this on version 2.0.2 with a MongoDB 3.2 instance and from my tests it looks like these changes would now incur a performance penalty. The JOB_PROCESS_WHERE_QUERY execution time is 383ms on master vs. 1056ms on the feature branch. The actual findOneAndUpdate performance gap is even larger, with Robomongo reporting 0.001 sec. query time vs. 0.651 sec. for the "optimised" query 😅

I've attached the find explain for each branch below. Since MongoDB 2.x support has been dropped there's no need to mitigate the query plan it generates, so I'm closing this PR.

/* master */
{
    "queryPlanner" : {
        "plannerVersion" : 1,
        "namespace" : "agenda-bench.agendaJobs",
        "indexFilterSet" : false,
        "parsedQuery" : {
            "$or" : [ 
                {
                    "$and" : [ 
                        {
                            "lockedAt" : {
                                "$eq" : null
                            }
                        }, 
                        {
                            "name" : {
                                "$eq" : "eight"
                            }
                        }, 
                        {
                            "nextRunAt" : {
                                "$lte" : ISODate("2018-11-29T21:24:47.576Z")
                            }
                        }, 
                        {
                            "$not" : {
                                "disabled" : {
                                    "$eq" : true
                                }
                            }
                        }
                    ]
                }, 
                {
                    "$and" : [ 
                        {
                            "name" : {
                                "$eq" : "eight"
                            }
                        }, 
                        {
                            "lockedAt" : {
                                "$lte" : ISODate("2018-11-29T21:14:42.576Z")
                            }
                        }, 
                        {
                            "$not" : {
                                "disabled" : {
                                    "$eq" : true
                                }
                            }
                        }
                    ]
                }, 
                {
                    "$and" : [ 
                        {
                            "name" : {
                                "$eq" : "eight"
                            }
                        }, 
                        {
                            "nextRunAt" : {
                                "$lte" : ISODate("2018-11-29T21:24:47.576Z")
                            }
                        }, 
                        {
                            "$not" : {
                                "disabled" : {
                                    "$eq" : true
                                }
                            }
                        }, 
                        {
                            "$not" : {
                                "lockedAt" : {
                                    "$exists" : true
                                }
                            }
                        }
                    ]
                }
            ]
        },
        "winningPlan" : {
            "stage" : "SUBPLAN",
            "inputStage" : {
                "stage" : "FETCH",
                "inputStage" : {
                    "stage" : "SORT_MERGE",
                    "sortPattern" : {
                        "nextRunAt" : 1.0,
                        "priority" : -1.0
                    },
                    "inputStages" : [ 
                        {
                            "stage" : "FETCH",
                            "filter" : {
                                "lockedAt" : {
                                    "$eq" : null
                                }
                            },
                            "inputStage" : {
                                "stage" : "IXSCAN",
                                "keyPattern" : {
                                    "name" : 1,
                                    "nextRunAt" : 1,
                                    "priority" : -1,
                                    "lockedAt" : 1,
                                    "disabled" : 1
                                },
                                "indexName" : "findAndLockNextJobIndex1",
                                "isMultiKey" : false,
                                "isUnique" : false,
                                "isSparse" : false,
                                "isPartial" : false,
                                "indexVersion" : 1,
                                "direction" : "forward",
                                "indexBounds" : {
                                    "name" : [ 
                                        "[\"eight\", \"eight\"]"
                                    ],
                                    "nextRunAt" : [ 
                                        "(true, new Date(1543526687576)]"
                                    ],
                                    "priority" : [ 
                                        "[MaxKey, MinKey]"
                                    ],
                                    "lockedAt" : [ 
                                        "[null, null]"
                                    ],
                                    "disabled" : [ 
                                        "[MinKey, true)", 
                                        "(true, MaxKey]"
                                    ]
                                }
                            }
                        }, 
                        {
                            "stage" : "IXSCAN",
                            "keyPattern" : {
                                "name" : 1,
                                "nextRunAt" : 1,
                                "priority" : -1,
                                "lockedAt" : 1,
                                "disabled" : 1
                            },
                            "indexName" : "findAndLockNextJobIndex1",
                            "isMultiKey" : false,
                            "isUnique" : false,
                            "isSparse" : false,
                            "isPartial" : false,
                            "indexVersion" : 1,
                            "direction" : "forward",
                            "indexBounds" : {
                                "name" : [ 
                                    "[\"eight\", \"eight\"]"
                                ],
                                "nextRunAt" : [ 
                                    "[MinKey, MaxKey]"
                                ],
                                "priority" : [ 
                                    "[MaxKey, MinKey]"
                                ],
                                "lockedAt" : [ 
                                    "(true, new Date(1543526082576)]"
                                ],
                                "disabled" : [ 
                                    "[MinKey, true)", 
                                    "(true, MaxKey]"
                                ]
                            }
                        }, 
                        {
                            "stage" : "FETCH",
                            "filter" : {
                                "$not" : {
                                    "lockedAt" : {
                                        "$exists" : true
                                    }
                                }
                            },
                            "inputStage" : {
                                "stage" : "IXSCAN",
                                "keyPattern" : {
                                    "name" : 1,
                                    "nextRunAt" : 1,
                                    "priority" : -1,
                                    "lockedAt" : 1,
                                    "disabled" : 1
                                },
                                "indexName" : "findAndLockNextJobIndex1",
                                "isMultiKey" : false,
                                "isUnique" : false,
                                "isSparse" : false,
                                "isPartial" : false,
                                "indexVersion" : 1,
                                "direction" : "forward",
                                "indexBounds" : {
                                    "name" : [ 
                                        "[\"eight\", \"eight\"]"
                                    ],
                                    "nextRunAt" : [ 
                                        "(true, new Date(1543526687576)]"
                                    ],
                                    "priority" : [ 
                                        "[MaxKey, MinKey]"
                                    ],
                                    "lockedAt" : [ 
                                        "[null, null]"
                                    ],
                                    "disabled" : [ 
                                        "[MinKey, true)", 
                                        "(true, MaxKey]"
                                    ]
                                }
                            }
                        }
                    ]
                }
            }
        },
        "rejectedPlans" : []
    },
    "executionStats" : {
        "executionSuccess" : true,
        "nReturned" : 179751,
        "executionTimeMillis" : 383,
        "totalKeysExamined" : 179754,
        "totalDocsExamined" : 179751,
        "executionStages" : {
            "stage" : "SUBPLAN",
            "nReturned" : 179751,
            "executionTimeMillisEstimate" : 330,
            "works" : 359506,
            "advanced" : 179751,
            "needTime" : 179754,
            "needYield" : 0,
            "saveState" : 2808,
            "restoreState" : 2808,
            "isEOF" : 1,
            "invalidates" : 0,
            "inputStage" : {
                "stage" : "FETCH",
                "nReturned" : 179751,
                "executionTimeMillisEstimate" : 330,
                "works" : 359505,
                "advanced" : 179751,
                "needTime" : 179754,
                "needYield" : 0,
                "saveState" : 2808,
                "restoreState" : 2808,
                "isEOF" : 1,
                "invalidates" : 0,
                "docsExamined" : 179751,
                "alreadyHasObj" : 0,
                "inputStage" : {
                    "stage" : "SORT_MERGE",
                    "nReturned" : 179751,
                    "executionTimeMillisEstimate" : 180,
                    "works" : 359505,
                    "advanced" : 179751,
                    "needTime" : 179754,
                    "needYield" : 0,
                    "saveState" : 2808,
                    "restoreState" : 2808,
                    "isEOF" : 1,
                    "invalidates" : 0,
                    "sortPattern" : {
                        "nextRunAt" : 1.0,
                        "priority" : -1.0
                    },
                    "dupsTested" : 179751,
                    "dupsDropped" : 0,
                    "inputStages" : [ 
                        {
                            "stage" : "FETCH",
                            "filter" : {
                                "lockedAt" : {
                                    "$eq" : null
                                }
                            },
                            "nReturned" : 0,
                            "executionTimeMillisEstimate" : 0,
                            "works" : 1,
                            "advanced" : 0,
                            "needTime" : 0,
                            "needYield" : 0,
                            "saveState" : 2808,
                            "restoreState" : 2808,
                            "isEOF" : 1,
                            "invalidates" : 0,
                            "docsExamined" : 0,
                            "alreadyHasObj" : 0,
                            "inputStage" : {
                                "stage" : "IXSCAN",
                                "nReturned" : 0,
                                "executionTimeMillisEstimate" : 0,
                                "works" : 1,
                                "advanced" : 0,
                                "needTime" : 0,
                                "needYield" : 0,
                                "saveState" : 2808,
                                "restoreState" : 2808,
                                "isEOF" : 1,
                                "invalidates" : 0,
                                "keyPattern" : {
                                    "name" : 1,
                                    "nextRunAt" : 1,
                                    "priority" : -1,
                                    "lockedAt" : 1,
                                    "disabled" : 1
                                },
                                "indexName" : "findAndLockNextJobIndex1",
                                "isMultiKey" : false,
                                "isUnique" : false,
                                "isSparse" : false,
                                "isPartial" : false,
                                "indexVersion" : 1,
                                "direction" : "forward",
                                "indexBounds" : {
                                    "name" : [ 
                                        "[\"eight\", \"eight\"]"
                                    ],
                                    "nextRunAt" : [ 
                                        "(true, new Date(1543526687576)]"
                                    ],
                                    "priority" : [ 
                                        "[MaxKey, MinKey]"
                                    ],
                                    "lockedAt" : [ 
                                        "[null, null]"
                                    ],
                                    "disabled" : [ 
                                        "[MinKey, true)", 
                                        "(true, MaxKey]"
                                    ]
                                },
                                "keysExamined" : 1,
                                "dupsTested" : 0,
                                "dupsDropped" : 0,
                                "seenInvalidated" : 0
                            }
                        }, 
                        {
                            "stage" : "IXSCAN",
                            "nReturned" : 179751,
                            "executionTimeMillisEstimate" : 150,
                            "works" : 179752,
                            "advanced" : 179751,
                            "needTime" : 0,
                            "needYield" : 0,
                            "saveState" : 2808,
                            "restoreState" : 2808,
                            "isEOF" : 1,
                            "invalidates" : 0,
                            "keyPattern" : {
                                "name" : 1,
                                "nextRunAt" : 1,
                                "priority" : -1,
                                "lockedAt" : 1,
                                "disabled" : 1
                            },
                            "indexName" : "findAndLockNextJobIndex1",
                            "isMultiKey" : false,
                            "isUnique" : false,
                            "isSparse" : false,
                            "isPartial" : false,
                            "indexVersion" : 1,
                            "direction" : "forward",
                            "indexBounds" : {
                                "name" : [ 
                                    "[\"eight\", \"eight\"]"
                                ],
                                "nextRunAt" : [ 
                                    "[MinKey, MaxKey]"
                                ],
                                "priority" : [ 
                                    "[MaxKey, MinKey]"
                                ],
                                "lockedAt" : [ 
                                    "(true, new Date(1543526082576)]"
                                ],
                                "disabled" : [ 
                                    "[MinKey, true)", 
                                    "(true, MaxKey]"
                                ]
                            },
                            "keysExamined" : 179752,
                            "dupsTested" : 0,
                            "dupsDropped" : 0,
                            "seenInvalidated" : 0
                        }, 
                        {
                            "stage" : "FETCH",
                            "filter" : {
                                "$not" : {
                                    "lockedAt" : {
                                        "$exists" : true
                                    }
                                }
                            },
                            "nReturned" : 0,
                            "executionTimeMillisEstimate" : 0,
                            "works" : 1,
                            "advanced" : 0,
                            "needTime" : 0,
                            "needYield" : 0,
                            "saveState" : 2808,
                            "restoreState" : 2808,
                            "isEOF" : 1,
                            "invalidates" : 0,
                            "docsExamined" : 0,
                            "alreadyHasObj" : 0,
                            "inputStage" : {
                                "stage" : "IXSCAN",
                                "nReturned" : 0,
                                "executionTimeMillisEstimate" : 0,
                                "works" : 1,
                                "advanced" : 0,
                                "needTime" : 0,
                                "needYield" : 0,
                                "saveState" : 2808,
                                "restoreState" : 2808,
                                "isEOF" : 1,
                                "invalidates" : 0,
                                "keyPattern" : {
                                    "name" : 1,
                                    "nextRunAt" : 1,
                                    "priority" : -1,
                                    "lockedAt" : 1,
                                    "disabled" : 1
                                },
                                "indexName" : "findAndLockNextJobIndex1",
                                "isMultiKey" : false,
                                "isUnique" : false,
                                "isSparse" : false,
                                "isPartial" : false,
                                "indexVersion" : 1,
                                "direction" : "forward",
                                "indexBounds" : {
                                    "name" : [ 
                                        "[\"eight\", \"eight\"]"
                                    ],
                                    "nextRunAt" : [ 
                                        "(true, new Date(1543526687576)]"
                                    ],
                                    "priority" : [ 
                                        "[MaxKey, MinKey]"
                                    ],
                                    "lockedAt" : [ 
                                        "[null, null]"
                                    ],
                                    "disabled" : [ 
                                        "[MinKey, true)", 
                                        "(true, MaxKey]"
                                    ]
                                },
                                "keysExamined" : 1,
                                "dupsTested" : 0,
                                "dupsDropped" : 0,
                                "seenInvalidated" : 0
                            }
                        }
                    ]
                }
            }
        }
    },
    "serverInfo" : {
        "host" : "65499cfe051e",
        "port" : 27017,
        "version" : "3.2.21",
        "gitVersion" : "1ab1010737145ba3761318508ff65ba74dfe8155"
    },
    "ok" : 1.0
}

/* feature/optimize-find-one-and-update */
{
    "queryPlanner" : {
        "plannerVersion" : 1,
        "namespace" : "agenda-bench.agendaJobs",
        "indexFilterSet" : false,
        "parsedQuery" : {
            "$or" : [ 
                {
                    "$and" : [ 
                        {
                            "disabled" : {
                                "$eq" : false
                            }
                        }, 
                        {
                            "lastFinishedAt" : {
                                "$eq" : null
                            }
                        }, 
                        {
                            "name" : {
                                "$eq" : "eight"
                            }
                        }, 
                        {
                            "nextRunAt" : {
                                "$eq" : null
                            }
                        }, 
                        {
                            "lockedAt" : {
                                "$lte" : ISODate("2018-11-30T11:36:29.650Z")
                            }
                        }
                    ]
                }, 
                {
                    "$and" : [ 
                        {
                            "disabled" : {
                                "$eq" : false
                            }
                        }, 
                        {
                            "name" : {
                                "$eq" : "eight"
                            }
                        }, 
                        {
                            "lockedAt" : {
                                "$lte" : ISODate("2018-11-30T11:36:29.650Z")
                            }
                        }, 
                        {
                            "nextRunAt" : {
                                "$lte" : ISODate("2018-11-30T11:46:34.650Z")
                            }
                        }
                    ]
                }
            ]
        },
        "winningPlan" : {
            "stage" : "SUBPLAN",
            "inputStage" : {
                "stage" : "SORT",
                "sortPattern" : {
                    "nextRunAt" : 1.0,
                    "priority" : -1.0
                },
                "inputStage" : {
                    "stage" : "SORT_KEY_GENERATOR",
                    "inputStage" : {
                        "stage" : "FETCH",
                        "inputStage" : {
                            "stage" : "OR",
                            "inputStages" : [ 
                                {
                                    "stage" : "FETCH",
                                    "filter" : {
                                        "$and" : [ 
                                            {
                                                "nextRunAt" : {
                                                    "$eq" : null
                                                }
                                            }, 
                                            {
                                                "lastFinishedAt" : {
                                                    "$eq" : null
                                                }
                                            }
                                        ]
                                    },
                                    "inputStage" : {
                                        "stage" : "IXSCAN",
                                        "keyPattern" : {
                                            "name" : 1,
                                            "nextRunAt" : 1,
                                            "priority" : -1,
                                            "disabled" : 1,
                                            "lockedAt" : 1,
                                            "lastFinishedAt" : 1
                                        },
                                        "indexName" : "findAndLockNextJobIndex1",
                                        "isMultiKey" : false,
                                        "isUnique" : false,
                                        "isSparse" : false,
                                        "isPartial" : false,
                                        "indexVersion" : 1,
                                        "direction" : "forward",
                                        "indexBounds" : {
                                            "name" : [ 
                                                "[\"eight\", \"eight\"]"
                                            ],
                                            "nextRunAt" : [ 
                                                "[null, null]"
                                            ],
                                            "priority" : [ 
                                                "[MaxKey, MinKey]"
                                            ],
                                            "disabled" : [ 
                                                "[false, false]"
                                            ],
                                            "lockedAt" : [ 
                                                "(true, new Date(1543577789650)]"
                                            ],
                                            "lastFinishedAt" : [ 
                                                "[null, null]"
                                            ]
                                        }
                                    }
                                }, 
                                {
                                    "stage" : "IXSCAN",
                                    "keyPattern" : {
                                        "name" : 1,
                                        "nextRunAt" : 1,
                                        "priority" : -1,
                                        "disabled" : 1,
                                        "lockedAt" : 1,
                                        "lastFinishedAt" : 1
                                    },
                                    "indexName" : "findAndLockNextJobIndex1",
                                    "isMultiKey" : false,
                                    "isUnique" : false,
                                    "isSparse" : false,
                                    "isPartial" : false,
                                    "indexVersion" : 1,
                                    "direction" : "forward",
                                    "indexBounds" : {
                                        "name" : [ 
                                            "[\"eight\", \"eight\"]"
                                        ],
                                        "nextRunAt" : [ 
                                            "(true, new Date(1543578394650)]"
                                        ],
                                        "priority" : [ 
                                            "[MaxKey, MinKey]"
                                        ],
                                        "disabled" : [ 
                                            "[false, false]"
                                        ],
                                        "lockedAt" : [ 
                                            "(true, new Date(1543577789650)]"
                                        ],
                                        "lastFinishedAt" : [ 
                                            "[MinKey, MaxKey]"
                                        ]
                                    }
                                }
                            ]
                        }
                    }
                }
            }
        },
        "rejectedPlans" : []
    },
    "executionStats" : {
        "executionSuccess" : true,
        "nReturned" : 180702,
        "executionTimeMillis" : 1056,
        "totalKeysExamined" : 180705,
        "totalDocsExamined" : 180702,
        "executionStages" : {
            "stage" : "SUBPLAN",
            "nReturned" : 180702,
            "executionTimeMillisEstimate" : 930,
            "works" : 361409,
            "advanced" : 180702,
            "needTime" : 180706,
            "needYield" : 0,
            "saveState" : 2823,
            "restoreState" : 2823,
            "isEOF" : 1,
            "invalidates" : 0,
            "inputStage" : {
                "stage" : "SORT",
                "nReturned" : 180702,
                "executionTimeMillisEstimate" : 920,
                "works" : 361408,
                "advanced" : 180702,
                "needTime" : 180706,
                "needYield" : 0,
                "saveState" : 2823,
                "restoreState" : 2823,
                "isEOF" : 1,
                "invalidates" : 0,
                "sortPattern" : {
                    "nextRunAt" : 1.0,
                    "priority" : -1.0
                },
                "memUsage" : 29093022,
                "memLimit" : 33554432,
                "inputStage" : {
                    "stage" : "SORT_KEY_GENERATOR",
                    "nReturned" : 0,
                    "executionTimeMillisEstimate" : 570,
                    "works" : 180706,
                    "advanced" : 0,
                    "needTime" : 3,
                    "needYield" : 0,
                    "saveState" : 2823,
                    "restoreState" : 2823,
                    "isEOF" : 1,
                    "invalidates" : 0,
                    "inputStage" : {
                        "stage" : "FETCH",
                        "nReturned" : 180702,
                        "executionTimeMillisEstimate" : 420,
                        "works" : 180705,
                        "advanced" : 180702,
                        "needTime" : 2,
                        "needYield" : 0,
                        "saveState" : 2823,
                        "restoreState" : 2823,
                        "isEOF" : 1,
                        "invalidates" : 0,
                        "docsExamined" : 180702,
                        "alreadyHasObj" : 0,
                        "inputStage" : {
                            "stage" : "OR",
                            "nReturned" : 180702,
                            "executionTimeMillisEstimate" : 320,
                            "works" : 180705,
                            "advanced" : 180702,
                            "needTime" : 2,
                            "needYield" : 0,
                            "saveState" : 2823,
                            "restoreState" : 2823,
                            "isEOF" : 1,
                            "invalidates" : 0,
                            "dupsTested" : 180702,
                            "dupsDropped" : 0,
                            "locsForgotten" : 0,
                            "inputStages" : [ 
                                {
                                    "stage" : "FETCH",
                                    "filter" : {
                                        "$and" : [ 
                                            {
                                                "nextRunAt" : {
                                                    "$eq" : null
                                                }
                                            }, 
                                            {
                                                "lastFinishedAt" : {
                                                    "$eq" : null
                                                }
                                            }
                                        ]
                                    },
                                    "nReturned" : 0,
                                    "executionTimeMillisEstimate" : 0,
                                    "works" : 1,
                                    "advanced" : 0,
                                    "needTime" : 0,
                                    "needYield" : 0,
                                    "saveState" : 2823,
                                    "restoreState" : 2823,
                                    "isEOF" : 1,
                                    "invalidates" : 0,
                                    "docsExamined" : 0,
                                    "alreadyHasObj" : 0,
                                    "inputStage" : {
                                        "stage" : "IXSCAN",
                                        "nReturned" : 0,
                                        "executionTimeMillisEstimate" : 0,
                                        "works" : 1,
                                        "advanced" : 0,
                                        "needTime" : 0,
                                        "needYield" : 0,
                                        "saveState" : 2823,
                                        "restoreState" : 2823,
                                        "isEOF" : 1,
                                        "invalidates" : 0,
                                        "keyPattern" : {
                                            "name" : 1,
                                            "nextRunAt" : 1,
                                            "priority" : -1,
                                            "disabled" : 1,
                                            "lockedAt" : 1,
                                            "lastFinishedAt" : 1
                                        },
                                        "indexName" : "findAndLockNextJobIndex1",
                                        "isMultiKey" : false,
                                        "isUnique" : false,
                                        "isSparse" : false,
                                        "isPartial" : false,
                                        "indexVersion" : 1,
                                        "direction" : "forward",
                                        "indexBounds" : {
                                            "name" : [ 
                                                "[\"eight\", \"eight\"]"
                                            ],
                                            "nextRunAt" : [ 
                                                "[null, null]"
                                            ],
                                            "priority" : [ 
                                                "[MaxKey, MinKey]"
                                            ],
                                            "disabled" : [ 
                                                "[false, false]"
                                            ],
                                            "lockedAt" : [ 
                                                "(true, new Date(1543577789650)]"
                                            ],
                                            "lastFinishedAt" : [ 
                                                "[null, null]"
                                            ]
                                        },
                                        "keysExamined" : 1,
                                        "dupsTested" : 0,
                                        "dupsDropped" : 0,
                                        "seenInvalidated" : 0
                                    }
                                }, 
                                {
                                    "stage" : "IXSCAN",
                                    "nReturned" : 180702,
                                    "executionTimeMillisEstimate" : 240,
                                    "works" : 180704,
                                    "advanced" : 180702,
                                    "needTime" : 1,
                                    "needYield" : 0,
                                    "saveState" : 2823,
                                    "restoreState" : 2823,
                                    "isEOF" : 1,
                                    "invalidates" : 0,
                                    "keyPattern" : {
                                        "name" : 1,
                                        "nextRunAt" : 1,
                                        "priority" : -1,
                                        "disabled" : 1,
                                        "lockedAt" : 1,
                                        "lastFinishedAt" : 1
                                    },
                                    "indexName" : "findAndLockNextJobIndex1",
                                    "isMultiKey" : false,
                                    "isUnique" : false,
                                    "isSparse" : false,
                                    "isPartial" : false,
                                    "indexVersion" : 1,
                                    "direction" : "forward",
                                    "indexBounds" : {
                                        "name" : [ 
                                            "[\"eight\", \"eight\"]"
                                        ],
                                        "nextRunAt" : [ 
                                            "(true, new Date(1543578394650)]"
                                        ],
                                        "priority" : [ 
                                            "[MaxKey, MinKey]"
                                        ],
                                        "disabled" : [ 
                                            "[false, false]"
                                        ],
                                        "lockedAt" : [ 
                                            "(true, new Date(1543577789650)]"
                                        ],
                                        "lastFinishedAt" : [ 
                                            "[MinKey, MaxKey]"
                                        ]
                                    },
                                    "keysExamined" : 180704,
                                    "dupsTested" : 0,
                                    "dupsDropped" : 0,
                                    "seenInvalidated" : 0
                                }
                            ]
                        }
                    }
                }
            }
        }
    },
    "serverInfo" : {
        "host" : "65499cfe051e",
        "port" : 27017,
        "version" : "3.2.21",
        "gitVersion" : "1ab1010737145ba3761318508ff65ba74dfe8155"
    },
    "ok" : 1.0
}

[simison]

Wohoo, thanks for checking it out!