Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure Autoscaler never reduces processes to less than 1 #1221

Merged
merged 6 commits into from
Nov 25, 2022

Conversation

LasseRafn
Copy link
Contributor

@LasseRafn LasseRafn commented Nov 25, 2022

CleanShot 2022-11-25 at 22 25 39

Last week, we started noticing some of our queues having 0 Processes assigned, despite this being against how Horizon is designed to work. Processes must always be positive, even if you specifically set minProcesses = 0.

After much digging, I eventually went through and added Log::debug statements all over the AutoScaler:

[2022-11-25 14:40:20]: Horizon maxProcesses: 15  
[2022-11-25 14:40:20]: Horizon processPools: 2  
[2022-11-25 14:40:20]: Horizon minProcesses: 1  
[2022-11-25 14:40:20]: Horizon $totalProcessCount: 1            <---- 
[2022-11-25 14:40:20]: Horizon totalProcessCountMethod: 16  
[2022-11-25 14:40:20]: Horizon $maxUpShift: -1                  <----
[2022-11-25 14:40:20]: Horizon desiredProcessCount: 15  
[2022-11-25 14:40:20]: Horizon pool->scale: 0  

Somehow Horizon would spawn 1 more process than was allowed (another issue for another day) and when then subtracting maxProcesses with totalProcessCount(), we would get -1.

Later down in the code it then picks which value to use for scaling:

min(
    $totalProcessCount + $maxUpShift,
    $supervisor->options->maxProcesses - (($supervisor->processPools->count() - 1) * $supervisor->options->minProcesses),
    $desiredProcessCount
)

Conclusion

As we saw above, $totalProcessCount = 1 and $maxUpShift = -1, which means this will return 0, setting processes to zero!

So when it tries to scale UP, it actually ends up scaling DOWN. Once it then hits 0, it seems to never be able to recover.

Solution

It is luckily not a difficult problem to solve. Wrapping max(0, ...) around the code that calculates $maxUpShift will prevent it from ever being negative, thus solving our problem.

I am aware that this technically fixes a problem that you might think is not there (why is processes greater than maxProcesses?) and whilst I agree, I also don't see the harm in covering all the bases.

Similar issue(s)

My abbreviated config for reference

[
    "waits" => [
      "redis:default" => 60,
    ],
    "fast_termination" => false,
    "memory_limit" => 512,
    "defaults" => [
      "app" => [
        "connection" => "redis",
        "queue" => [
          "default",
          "test",
        ],
        "balance" => "auto",
        "maxProcesses" => 1,
        "memory" => 512,
        "tries" => 2,
        "nice" => 5,
        "timeout" => 1800,
      ],
    ],
    "environments" => [
      "production" => [
        "app" => [
          "maxProcesses" => 15,
          "balance" => "auto",
          "tries" => 10,
          "backoff" => [
            60,
            300,
            600,
            900,
          ],
        ],
      ],
    ],
  ]

@LasseRafn LasseRafn marked this pull request as draft November 25, 2022 14:53
# Conflicts:
#	src/AutoScaler.php
#	tests/Feature/Fakes/FakePool.php
# Conflicts:
#	src/AutoScaler.php
#	tests/Feature/Fakes/FakePool.php
# Conflicts:
#	tests/Feature/Fakes/FakePool.php
@LasseRafn LasseRafn marked this pull request as ready for review November 25, 2022 15:21
@LasseRafn LasseRafn changed the title Fix bug causing Processes to hit 0 despite Autoscaler scaling up Ensure Autoscaler never reduces processes to less than 1 Nov 25, 2022
@taylorotwell taylorotwell merged commit 01b26da into laravel:5.x Nov 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants