[12.x] Improve Queues and Horizon documentation #10596
Conversation
| @@ -1213,7 +1215,21 @@ class ProcessPodcast implements ShouldQueue | |||
| <a name="max-attempts"></a> | |||
| #### Max Attempts | |||
|
|
|||
| If one of your queued jobs is encountering an error, you likely do not want it to keep retrying indefinitely. Therefore, Laravel provides various ways to specify how many times or for how long a job may be attempted. | |||
There was a problem hiding this comment.
This section was confusing to me, as it mixed errors and attempts, so I tried to explain as clearly as possible what an ‘attempt’ is.
| > [!NOTE] | ||
| > A job marked failed will never be retried. | ||
There was a problem hiding this comment.
This might be redundant, but I added it for people who might not read the entire documentation as they should
horizon.md
Outdated
| ], | ||
| ``` | ||
|
|
||
| In this example, jobs in the `high` queue will be prioritized over those in the `default` or `low` queues. |
There was a problem hiding this comment.
I think this is only the case if balance=false.
There was a problem hiding this comment.
You are right, I need rewrite this section
There was a problem hiding this comment.
I've just completely rewritten the section about the strategies.
horizon.md
Outdated
| @@ -132,7 +176,12 @@ Unlike Laravel's default queue system, Horizon allows you to choose from three w | |||
|
|
|||
| The `auto` strategy, which is the configuration file's default, adjusts the number of worker processes per queue based on the current workload of the queue. For example, if your `notifications` queue has 1,000 pending jobs while your `render` queue is empty, Horizon will allocate more workers to your `notifications` queue until the queue is empty. | |||
|
|
|||
| When using the `auto` strategy, you may define the `minProcesses` and `maxProcesses` configuration options to control the minimum number of processes per queue and the maximum number of worker processes in total Horizon should scale up and down to: | |||
| When using the auto scaling strategy, you may define the `minProcesses` and `maxProcesses` configuration options: | |||
There was a problem hiding this comment.
In my testing these work with any balacing mode or at least also with balance=false.
There was a problem hiding this comment.
Yes, that's true, but these options do not have the same meaning. To make more sense of it, I've added the processes key, which is an alias for maxProcesses
There was a problem hiding this comment.
Yes, but they work the same regardless of the balancing strategy. So, if you set balance=false, you can still set minProcesses and maxProcesses. If you set minProcesses=1 and maxProcesses=4, 1 process will run in total, even if you have two queues because of balance being set to false. If there is load, Horizon will add up to 3 more workers (total of 4), as it would with any other strategy. This should be reflected somehow, as the current one is confusing to me.
There was a problem hiding this comment.
@peaklabs-dev Ok, I just realized that I didn't understand the false option correctly. So I guess it was really confusing.
Really appreciate your reviews here! I've updated section about this strategy, let me know if it's sounds good to you.
PS: I’m pretty sure I don’t fully understand the simple one either. From what you’re saying, I’m guessing that Horizon always assigns the same number of processes to each queue when scaling up?
There was a problem hiding this comment.
Did some testing and it seems that when using the simple strategy, there is no scaling. All worker processes defined by maxProcesses are always running. Let me know if I'm wrong
About
This PR is based on a recent post I shared on X, where I discussed some pain points around Horizon.
I did my best to avoid injecting strong personal opinions or turning the documentation into a blog post.
Queues:
Clarifies how job attempts are counted and how failed jobs are handled. Adds updated examples and notes for middleware and retries.
Horizon:
Adds sections on queue priorities, job attempts, timeouts, and backoff. Improves explanations for environment configuration and worker balancing.
There are still options left to document.
Reflections on
maxExceptionandtriesDefaultsWhile working on this PR, I strongly felt the need for an option to configure
maxExceptionsat the Horizon supervisor or queue worker level, similar to howtriescan currently be set.Additionally, I’m wondering whether the default value for
triesis appropriate. Many of Laravel’s advanced queue features, such asWithoutOverlappingandRateLimited, often require increasing thetriesvalue significantly, sometimes even setting it to0.In my opinion, the default configuration should be more aligned with these common use cases. A more sensible default might look like:
tries: 25maxException: 1This defaults would work better out of the box for most Laravel developers .