feat: crons for all bin scripts, new jobs:handle-schedules script, more reliable job system crons (#13564)

## New jobs:handle-schedules bin script

Similarly to `payload jobs:run`, this PR adds a new
`jobs:handle-schedules` bin script which only handles scheduling.

## Allows jobs:run bin script to handle scheduling

Similarly to how [payload
autoRun](https://payloadcms.com/docs/jobs-queue/queues#cron-jobs)
handles both running and scheduling jobs by default, you can now set the
`payload jobs:run` bin script to also handle scheduling. This is opt-in:

```sh
pnpm payload jobs:run --cron "*/5 * * * *" --queue myQueue --handle-schedules # This will both schedule jobs according to the configuration and run them
```

## Cron schedules for all bin scripts

Previously, only the `payload jobs:run` bin script accepted a cron flag.
The `payload jobs:handle-schedules` would have required the same logic
to also handle a cron flag.

Instead of opting for this duplicative logic, I'm now handling cron
logic before we determine which script to run. This means: it's simpler
and requires less duplicative code.

**This allows all other bin scripts (including custom ones) to use the
`--cron` flag**, enabling cool use-cases like scheduling your own custom
scripts - no additional config required!

Example:

```sh
pnpm payload run ./myScript.ts --cron "0 * * * *"
```

Video Example:


https://github.com/user-attachments/assets/4ded738d-2ef9-43ea-8136-f47f913a7ba8

## More reliable job system crons

When using autorun or `--cron`, if one cron run takes longer than the
cron interval, the second cron would run before the first one finishes.

This can be especially dangerous when running jobs using a bin script,
potentially causing race conditions, as the first cron run will take
longer due to payload initialization overhead (only for first cron run,
consecutive ones use cached payload). Now, consecutive cron runs will
wait for the first one to finish by using the `{ protect: true }`
property of Croner.

This change will affect both autorun and bin scripts.

## Cleanup

- Centralized payload instance cleanup (payload.destroy()) for all bin
scripts
- The `getPayload` function arguments were not properly typed. Arguments
like `disableOnInit: true` are already supported, but the type did not
reflect that. This simplifies the type and makes it more accurate.

## Fixes

- `allQueues` argument for `payload jobs:run` was not respected


---
- To see the specific tasks where the Asana app for GitHub is being
used, see below:
  - https://app.asana.com/0/0/1211124797199077

Author: AlessioGr

docs/configuration/overview.mdx

@@ -319,3 +319,13 @@ Now you can run the command using:
 ```sh
 pnpm payload seed
 ```
+
+## Running bin scripts on a schedule
+
+Every bin script supports being run on a schedule using cron syntax. Simply pass the `--cron` flag followed by the cron expression when running the script. Example:
+
+```sh
+pnpm payload run ./myScript.ts --cron "0 * * * *"
+```
+
+This will use the `run` bin script to execute the specified script on the defined schedule.

docs/jobs-queue/queues.mdx

@@ -173,25 +173,31 @@ const results = await payload.jobs.runByID({
 Finally, you can process jobs via the bin script that comes with Payload out of the box. By default, this script will run jobs from the `default` queue, with a limit of 10 jobs per invocation:
 
 ```sh
-npx payload jobs:run
+pnpm payload jobs:run
 ```
 
 You can override the default queue and limit by passing the `--queue` and `--limit` flags:
 
 ```sh
-npx payload jobs:run --queue myQueue --limit 15
+pnpm payload jobs:run --queue myQueue --limit 15
 ```
 
 If you want to run all jobs from all queues, you can pass the `--all-queues` flag:
 
 ```sh
-npx payload jobs:run --all-queues
+pnpm payload jobs:run --all-queues
 ```
 
 In addition, the bin script allows you to pass a `--cron` flag to the `jobs:run` command to run the jobs on a scheduled, cron basis:
 
 ```sh
-npx payload jobs:run --cron "*/5 * * * *"
+pnpm payload jobs:run --cron "*/5 * * * *"
+```
+
+You can also pass `--handle-schedules` flag to the `jobs:run` command to make it schedule jobs according to configured schedules:
+
+```sh
+pnpm payload jobs:run --cron "*/5 * * * *" --queue myQueue --handle-schedules # This will both schedule jobs according to the configuration and run them
 ```
 
 ## Processing Order

docs/jobs-queue/schedules.mdx

@@ -35,6 +35,20 @@ Something needs to actually trigger the scheduling of jobs (execute the scheduli
 
 You can disable this behavior by setting `disableScheduling: true` in your `autorun` configuration, or by passing `disableScheduling=true` to the `/api/payload-jobs/run` endpoint. This is useful if you want to handle scheduling manually, for example, by using a cron job or a serverless function that calls the `/api/payload-jobs/handle-schedules` endpoint or the `payload.jobs.handleSchedules()` local API method.
 
+### Bin Scripts
+
+Payload provides a set of bin scripts that can be used to handle schedules. If you're already using the `jobs:run` bin script, you can set it to also handle schedules by passing the `--handle-schedules` flag:
+
+```sh
+pnpm payload jobs:run --cron "*/5 * * * *" --queue myQueue --handle-schedules # This will both schedule jobs according to the configuration and run them
+```
+
+If you only want to handle schedules, you can use the dedicated `jobs:handle-schedules` bin script:
+
+```sh
+pnpm payload jobs:handle-schedules --cron "*/5 * * * *" --queue myQueue # or --all-queues
+```
+
 ## Defining schedules on Tasks or Workflows
 
 Schedules are defined using the `schedule` property:

packages/payload/src/bin/index.ts

@@ -1,12 +1,11 @@
-// @ts-strict-ignore
 /* eslint-disable no-console */
 import { Cron } from 'croner'
 import minimist from 'minimist'
 import { pathToFileURL } from 'node:url'
 import path from 'path'
 
 import { findConfig } from '../config/find.js'
-import payload, { getPayload } from '../index.js'
+import { getPayload, type Payload } from '../index.js'
 import { generateImportMap } from './generateImportMap/index.js'
 import { generateTypes } from './generateTypes.js'
 import { info } from './info.js'
@@ -20,19 +19,61 @@ const availableScripts = [
   'generate:types',
   'info',
   'jobs:run',
+  'jobs:handle-schedules',
   'run',
   ...migrateCommands,
 ] as const
 
 export const bin = async () => {
   loadEnv()
+  process.env.DISABLE_PAYLOAD_HMR = 'true'
 
   const args = minimist(process.argv.slice(2))
   const script = (typeof args._[0] === 'string' ? args._[0] : '').toLowerCase()
 
+  if (args.cron) {
+    new Cron(
+      args.cron,
+      async () => {
+        // If the bin script initializes payload (getPayload), this will only happen once, as getPayload
+        // caches the payload instance on the module scope => no need to manually cache and manage getPayload initialization
+        // outside the Cron here.
+        await runBinScript({ args, script })
+      },
+      {
+        // Do not run consecutive crons if previous crons still ongoing
+        protect: true,
+      },
+    )
+
+    process.stdin.resume() // Keep the process alive
+
+    return
+  } else {
+    const { payload } = await runBinScript({ args, script })
+    if (payload) {
+      await payload.destroy() // close database connections after running jobs so process can exit cleanly
+    }
+    process.exit(0)
+  }
+}
+
+async function runBinScript({
+  args,
+  script,
+}: {
+  args: minimist.ParsedArgs
+  script: string
+}): Promise<{
+  /**
+   * Scripts can return a payload instance if it exists. The bin script runner can then safely
+   * shut off the instance, depending on if it's running in a cron job or not.
+   */
+  payload?: Payload
+}> {
   if (script === 'info') {
     await info()
-    return
+    return {}
   }
 
   if (script === 'run') {
@@ -58,7 +99,7 @@ export const bin = async () => {
       // Restore original process.argv
       process.argv = originalArgv
     }
-    return
+    return {}
   }
 
   const configPath = findConfig()
@@ -91,65 +132,70 @@ export const bin = async () => {
       console.error(err)
     }
 
-    return
+    return {}
   }
 
   if (script.startsWith('migrate')) {
-    return migrate({ config, parsedArgs: args }).then(() => process.exit(0))
+    await migrate({ config, parsedArgs: args })
+    return {}
   }
 
   if (script === 'generate:types') {
-    return generateTypes(config)
+    await generateTypes(config)
+    return {}
   }
 
   if (script === 'generate:importmap') {
-    return generateImportMap(config)
+    await generateImportMap(config)
+    return {}
   }
 
   if (script === 'jobs:run') {
     const payload = await getPayload({ config }) // Do not setup crons here - this bin script can set up its own crons
     const limit = args.limit ? parseInt(args.limit, 10) : undefined
     const queue = args.queue ? args.queue : undefined
-    const allQueues = !!args.allQueues
-
-    if (args.cron) {
-      new Cron(args.cron, async () => {
-        await payload.jobs.run({
-          allQueues,
-          limit,
-          queue,
-        })
-      })
-
-      process.stdin.resume() // Keep the process alive
+    const allQueues = !!args['all-queues']
+    const handleSchedules = !!args['handle-schedules']
 
-      return
-    } else {
-      await payload.jobs.run({
+    if (handleSchedules) {
+      await payload.jobs.handleSchedules({
         allQueues,
-        limit,
         queue,
       })
+    }
 
-      await payload.destroy() // close database connections after running jobs so process can exit cleanly
+    await payload.jobs.run({
+      allQueues,
+      limit,
+      queue,
+    })
 
-      process.exit(0)
-    }
+    return { payload }
+  }
+
+  if (script === 'jobs:handle-schedules') {
+    const payload = await getPayload({ config }) // Do not setup crons here - this bin script can set up its own crons
+    const queue = args.queue ? args.queue : undefined
+    const allQueues = !!args['all-queues']
+
+    await payload.jobs.handleSchedules({
+      allQueues,
+      queue,
+    })
+
+    return { payload }
   }
 
   if (script === 'generate:db-schema') {
     // Barebones instance to access database adapter, without connecting to the DB
-    await payload.init({
-      config,
-      disableDBConnect: true,
-      disableOnInit: true,
-    })
+    const payload = await getPayload({ config, disableDBConnect: true, disableOnInit: true }) // Do not setup crons here
 
     if (typeof payload.db.generateSchema !== 'function') {
       payload.logger.error({
         msg: `${payload.db.packageName} does not support database schema generation`,
       })
 
+      await payload.destroy()
       process.exit(1)
     }
 
@@ -158,7 +204,7 @@ export const bin = async () => {
       prettify: args.prettify === 'false' ? false : true,
     })
 
-    process.exit(0)
+    return { payload }
   }
 
   console.error(script ? `Unknown command: "${script}"` : 'Please provide a command to run')

packages/payload/src/index.ts

@@ -637,38 +637,45 @@ export class BasePayload {
 
       await Promise.all(
         cronJobs.map((cronConfig) => {
-          const jobAutorunCron = new Cron(cronConfig.cron ?? DEFAULT_CRON, async () => {
-            if (
-              _internal_jobSystemGlobals.shouldAutoSchedule &&
-              !cronConfig.disableScheduling &&
-              this.config.jobs.scheduling
-            ) {
-              await this.jobs.handleSchedules({
-                allQueues: cronConfig.allQueues,
-                queue: cronConfig.queue,
-              })
-            }
+          const jobAutorunCron = new Cron(
+            cronConfig.cron ?? DEFAULT_CRON,
+            async () => {
+              if (
+                _internal_jobSystemGlobals.shouldAutoSchedule &&
+                !cronConfig.disableScheduling &&
+                this.config.jobs.scheduling
+              ) {
+                await this.jobs.handleSchedules({
+                  allQueues: cronConfig.allQueues,
+                  queue: cronConfig.queue,
+                })
+              }
 
-            if (!_internal_jobSystemGlobals.shouldAutoRun) {
-              return
-            }
+              if (!_internal_jobSystemGlobals.shouldAutoRun) {
+                return
+              }
 
-            if (typeof this.config.jobs.shouldAutoRun === 'function') {
-              const shouldAutoRun = await this.config.jobs.shouldAutoRun(this)
+              if (typeof this.config.jobs.shouldAutoRun === 'function') {
+                const shouldAutoRun = await this.config.jobs.shouldAutoRun(this)
 
-              if (!shouldAutoRun) {
-                jobAutorunCron.stop()
-                return
+                if (!shouldAutoRun) {
+                  jobAutorunCron.stop()
+                  return
+                }
               }
-            }
 
-            await this.jobs.run({
-              allQueues: cronConfig.allQueues,
-              limit: cronConfig.limit ?? DEFAULT_LIMIT,
-              queue: cronConfig.queue,
-              silent: cronConfig.silent,
-            })
-          })
+              await this.jobs.run({
+                allQueues: cronConfig.allQueues,
+                limit: cronConfig.limit ?? DEFAULT_LIMIT,
+                queue: cronConfig.queue,
+                silent: cronConfig.silent,
+              })
+            },
+            {
+              // Do not run consecutive crons if previous crons still ongoing
+              protect: true,
+            },
+          )
 
           this.crons.push(jobAutorunCron)
         }),
@@ -942,6 +949,7 @@ export const reload = async (
   config: SanitizedConfig,
   payload: Payload,
   skipImportMapGeneration?: boolean,
+  options?: InitOptions,
 ): Promise<void> => {
   if (typeof payload.db.destroy === 'function') {
     // Only destroy db, as we then later only call payload.db.init and not payload.init
@@ -991,9 +999,11 @@ export const reload = async (
     })
   }
 
-  await payload.db.init?.()
+  if (payload.db?.init) {
+    await payload.db.init()
+  }
 
-  if (payload.db.connect) {
+  if (!options?.disableDBConnect && payload.db.connect) {
     await payload.db.connect({ hotReload: true })
   }
 
@@ -1037,7 +1047,7 @@ export const getPayload = async (
      * @default 'default'
      */
     key?: string
-  } & Pick<InitOptions, 'config' | 'cron' | 'disableOnInit' | 'importMap'>,
+  } & InitOptions,
 ): Promise<Payload> => {
   if (!options?.config) {
     throw new Error('Error: the payload config is required for getPayload to work.')
@@ -1080,7 +1090,7 @@ export const getPayload = async (
       // will reach `if (cached.reload instanceof Promise) {` which then waits for the first reload to finish.
       cached.reload = new Promise((res) => (resolve = res))
       const config = await options.config
-      await reload(config, cached.payload, !options.importMap)
+      await reload(config, cached.payload, !options.importMap, options)
 
       resolve()
     }