Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Puma plugin :solid_queue throws an error on start #77

Closed
raphaeljlps opened this issue Dec 20, 2023 · 12 comments · Fixed by #86
Closed

Puma plugin :solid_queue throws an error on start #77

raphaeljlps opened this issue Dec 20, 2023 · 12 comments · Fixed by #86

Comments

@raphaeljlps
Copy link

I'm trying to use puma plugin :solid_queue but I get an error

gems/solid_queue-0.1.1/lib/puma/plugin/solid_queue.rb:11:in `block in start': uninitialized constant SolidQueue (NameError)
      SolidQueue::Supervisor.start(mode: :all)
                ^^^^^^^^^^^^

this happens on my MacBook (M1) and in my linux server/container

my solid_queue.yml

production:
  dispatchers:
    - polling_interval: 1
      batch_size: 500
  workers:
    - queues:
      - "*"
      threads: 1
      polling_interval: 2
    - queues:
      - real_time
      - background
      threads: 1
      polling_interval: 0.1
      processes: 1

The full stack

Running 'bundle exec puma -t 5:5 -p ${PORT:-3000} -e ${RACK_ENV:-development}'
[62] Puma starting in cluster mode...
[62] * Puma version: 6.4.0 (ruby 3.2.2-p53) ("The Eagle of Durango")
[62] *  Min threads: 5
[62] *  Max threads: 5
[62] *  Environment: deployment
[62] *   Master PID: 62
[62] *      Workers: 4
[62] *     Restarts: (✔) hot (✔) phased
[62] * Listening on http://0.0.0.0:10000/
[62] Use Ctrl-C to stop
/opt/render/project/.gems/ruby/3.2.0/gems/solid_queue-0.1.1/lib/puma/plugin/solid_queue.rb:11:in `block in start': uninitialized constant SolidQueue (NameError)
      SolidQueue::Supervisor.start(mode: :all)
                ^^^^^^^^^^^^
	from /opt/render/project/.gems/ruby/3.2.0/gems/solid_queue-0.1.1/lib/puma/plugin/solid_queue.rb:9:in `fork'
	from /opt/render/project/.gems/ruby/3.2.0/gems/solid_queue-0.1.1/lib/puma/plugin/solid_queue.rb:9:in `start'
	from /opt/render/project/.gems/ruby/3.2.0/gems/puma-6.4.0/lib/puma/plugin.rb:24:in `block in fire_starts'
	from /opt/render/project/.gems/ruby/3.2.0/gems/puma-6.4.0/lib/puma/plugin.rb:22:in `each'
	from /opt/render/project/.gems/ruby/3.2.0/gems/puma-6.4.0/lib/puma/plugin.rb:22:in `fire_starts'
	from /opt/render/project/.gems/ruby/3.2.0/gems/puma-6.4.0/lib/puma/launcher.rb:188:in `run'
	from /opt/render/project/.gems/ruby/3.2.0/gems/puma-6.4.0/lib/puma/cli.rb:75:in `run'
	from /opt/render/project/.gems/ruby/3.2.0/gems/puma-6.4.0/bin/puma:10:in `<top (required)>'
	from /opt/render/project/.gems/bin/puma:27:in `load'
	from /opt/render/project/.gems/bin/puma:27:in `<top (required)>'
	from /opt/render/project/.gems/gems/bundler-2.4.22/lib/bundler/cli/exec.rb:58:in `load'
	from /opt/render/project/.gems/gems/bundler-2.4.22/lib/bundler/cli/exec.rb:58:in `kernel_load'
	from /opt/render/project/.gems/gems/bundler-2.4.22/lib/bundler/cli/exec.rb:23:in `run'
	from /opt/render/project/.gems/gems/bundler-2.4.22/lib/bundler/cli.rb:492:in `exec'
	from /opt/render/project/.gems/gems/bundler-2.4.22/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
	from /opt/render/project/.gems/gems/bundler-2.4.22/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
	from /opt/render/project/.gems/gems/bundler-2.4.22/lib/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
	from /opt/render/project/.gems/gems/bundler-2.4.22/lib/bundler/cli.rb:34:in `dispatch'
	from /opt/render/project/.gems/gems/bundler-2.4.22/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
	from /opt/render/project/.gems/gems/bundler-2.4.22/lib/bundler/cli.rb:28:in `start'
	from /opt/render/project/.gems/gems/bundler-2.4.22/exe/bundle:37:in `block in <top (required)>'
	from /opt/render/project/.gems/gems/bundler-2.4.22/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
	from /opt/render/project/.gems/gems/bundler-2.4.22/exe/bundle:29:in `<top (required)>'
	from /opt/render/project/.gems/bin/bundle:108:in `load'
	from /opt/render/project/.gems/bin/bundle:108:in `<main>'
[62] Detected Solid Queue has gone away, stopping Puma...
[62] - Gracefully shutting down workers...
[70] Early termination of worker
[68] Early termination of worker
[74] Early termination of worker
[72] Early termination of worker
[62] === puma shutdown: 2023-12-20 22:15:18 +0000 ===
[62] - Goodbye!
@djmb
Copy link
Contributor

djmb commented Dec 21, 2023

Hi @raphaeljlps,

I've tested this out on a fresh Rails app and it works fine when I have gem "solid_queue" in the gemfile. If however I do gem "solid_queue", require: false I get the same error as you have. Is that what you have in your setup?

If that's the issue here, adding require "solid_queue" to the plugin won't help as by the time its invoked we are too far on in the Rails initialisation process. You'll need to require the gem from the Gemfile instead.

If that's not what's going on here, do you have anything else you could share about your setup?

@raphaeljlps
Copy link
Author

raphaeljlps commented Dec 21, 2023

Hey @djmb

In my gem file I have

# background jobs backend
gem 'solid_queue', '~> 0.1.1'

I've added require: true to the gemfile and the same behavior continues

If I do rails s in my development machine, it works as expected (meaning, solid queue runs). However, if I run with bundle exec puma -t 5:5 -p ${PORT:-3000} -e ${RACK_ENV:-development} it crashes with the error.


# frozen_string_literal: true

source 'https://rubygems.org'

ruby '3.2.2'

# Bundle edge Rails instead: gem "rails", github: "rails/rails", branch: "main"
gem 'rails', '~> 7.1.2'

# The original asset pipeline for Rails [https://github.com/rails/sprockets-rails]
gem 'sprockets-rails'

# Use postgresql as the database for Active Record
gem 'pg', '~> 1.1'

# Use the Puma web server [https://github.com/puma/puma]
gem 'puma', '>= 5.0'

# Use JavaScript with ESM import maps [https://github.com/rails/importmap-rails]
gem 'importmap-rails'

# Hotwire's SPA-like page accelerator [https://turbo.hotwired.dev]
gem 'turbo-rails'

# Hotwire's modest JavaScript framework [https://stimulus.hotwired.dev]
gem 'stimulus-rails'

# Use Tailwind CSS [https://github.com/rails/tailwindcss-rails]
gem 'tailwindcss-rails'

# Build JSON APIs with ease [https://github.com/rails/jbuilder]
gem 'jbuilder'

# Use Redis adapter to run Action Cable in production
gem 'redis', '>= 4.0.1'

# Windows does not include zoneinfo files, so bundle the tzinfo-data gem
gem 'tzinfo-data', platforms: %i[jruby]

# Reduces boot times through caching; required in config/boot.rb
gem 'bootsnap', require: false

group :development, :test do
  gem 'brakeman'
  gem 'bundler-audit'
  gem 'debug', platforms: %i[mri]
  gem 'rubocop'
  gem 'rubocop-performance'
  gem 'rubocop-rails'
end

group :development do
  gem 'web-console'
  gem 'rack-mini-profiler'

  gem 'derailed_benchmarks'
  gem 'memory_profiler'
  gem 'stackprof'
end

group :test do
  gem 'capybara'
  gem 'selenium-webdriver'
end

# authentication
gem 'rodauth-rails', '~> 1.12'

# background jobs backend
gem 'solid_queue', '~> 0.1.1', require: true

# multitenancy
gem 'acts_as_tenant'

gem 'matrix', '~> 0.4.2'

# apm
gem 'appsignal'

# email provider
gem 'postmark-rails'

@mattpolito
Copy link

Also experiencing this. Works as expected in dev but throws the uninitialized constant SolidQueue error in production env.

@djmb
Copy link
Contributor

djmb commented Dec 21, 2023

Thanks @raphaeljlps - I can see that failing as well now.

Loading the application first in config/puma.rb looks like it fixes it:

# config/puma.rb
require File.expand_path("environment", File.dirname(__FILE__))
...

But that's not ideal. We'd maybe will want to shell out to the solid queue rake task or exe (once we add one) instead.

@koenhandekyn
Copy link

same issue here : fine with rails s but same exception with below

 bundle exec puma -C config/puma.rb --early-hints -e development

@atstockland
Copy link

atstockland commented Jan 11, 2024

Same issue here, on Heroku. Why was this closed with no solutions?

@raphaeljlps
Copy link
Author

@atstockland there was a solution, we have @djmb workaround and I see a PR by @nashby.

Are you still affected by this? I'm using the latests version on Render without issues and without the workaround.

@atstockland
Copy link

Sorry about that--I didn't realize that was the sanctioned work-around. I did try that, but the issue persisted for me. I'll try again, or try after the next update. Thanks so much!

@atstockland
Copy link

I must have had a compounding problems going on the first time I tried. I just updated my puma.rb to load the environment and everything works fine now.

Thanks!

@jules-w2
Copy link

Hey

the problem is still here (on devlopement & production mode).
The gem seems to be loaded but not all these modules.
do you have any idea how to load all the modules?

# Procfile
puma:   bundle exec puma -C config/puma/development.rb
# config/puma/development.rb
bind "ssl://0.0.0.0:4000"
workers 2
threads 1, 3
plugin :solid_queue
# solid_queue/lib/puma/plugin/solid_queue.rb

def start(launcher)
  @log_writer = launcher.log_writer
  @puma_pid = $$

  launcher.events.on_booted do
   puts "puma is booted..."
    puts defined?(SolidQueue)             ? "SolidQueue loaded" : "SolidQueue Not loaded"
    puts defined?(SolidQueue::VERSION)    ? "SolidQueue Version: #{SolidQueue::VERSION}" : "SolidQueue: Not loaded"
    puts defined?(SolidQueue::Supervisor) ? "SolidQueue::Supervisor loaded" : "SolidQueue::Supervisor: Not loaded"

    @solid_queue_pid = fork do
      Thread.new { monitor_puma }
      SolidQueue::Supervisor.start(mode: :all)
    end

    in_background do
      monitor_solid_queue
    end
  end

  launcher.events.on_stopped { stop_solid_queue }
  launcher.events.on_restart { stop_solid_queue }
end
01:45:06 puma.1    | [97629] [MacBook-Pro.local] 2024-04-17 01:45:06 -0500: Puma starting in cluster mode...
01:45:06 puma.1    | [97629] [MacBook-Pro.local] 2024-04-17 01:45:06 -0500: * Puma version: 6.4.2 (ruby 3.2.2-p53) ("The Eagle of Durango")
01:45:06 puma.1    | [97629] [MacBook-Pro.local] 2024-04-17 01:45:06 -0500: *  Min threads: 1
01:45:06 puma.1    | [97629] [MacBook-Pro.local] 2024-04-17 01:45:06 -0500: *  Max threads: 3
01:45:06 puma.1    | [97629] [MacBook-Pro.local] 2024-04-17 01:45:06 -0500: *  Environment: development
01:45:06 puma.1    | [97629] [MacBook-Pro.local] 2024-04-17 01:45:06 -0500: *   Master PID: 97629
01:45:06 puma.1    | [97629] [MacBook-Pro.local] 2024-04-17 01:45:06 -0500: *      Workers: 2
01:45:06 puma.1    | [97629] [MacBook-Pro.local] 2024-04-17 01:45:06 -0500: *     Restarts: (✔) hot (✔) phased
01:45:06 puma.1    | [97629] [MacBook-Pro.local] 2024-04-17 01:45:06 -0500: * Listening on ssl://0.0.0.0:4000
01:45:06 puma.1    | [97629] [MacBook-Pro.local] 2024-04-17 01:45:06 -0500: Use Ctrl-C to stop
01:45:07 puma.1    | [97629] [MacBook-Pro.local] 2024-04-17 01:45:07 -0500: - Worker 1 (PID: 97636) booted in 1.13s, phase: 0
01:45:07 puma.1    | [97629] [MacBook-Pro.local] 2024-04-17 01:45:07 -0500: - Worker 0 (PID: 97635) booted in 1.13s, phase: 0
01:45:07 puma.1    | puma is booted...
01:45:07 puma.1    | SolidQueue loaded
01:45:07 puma.1    | SolidQueue Version: 0.3.0
01:45:07 puma.1    | SolidQueue::Supervisor: Not loaded
01:45:07 puma.1    | /Users/test/solid_queue/lib/puma/plugin/solid_queue.rb:18:in `block (2 levels) in start': uninitialized constant SolidQueue::Supervisor (NameError)
01:45:07 puma.1    |
01:45:07 puma.1    |         SolidQueue::Supervisor.start(mode: :all)
01:45:07 puma.1    |                   ^^^^^^^^^^^^
01:45:07 puma.1    | 	from /Users/test/solid_queue/lib/puma/plugin/solid_queue.rb:16:in `fork'

@jczyzewski
Copy link

In case anyone still has this problem after the #86, here's what helped me:
For me, I got the same error as @jules-w2, but only when puma was running in a cluster mode. Error went away once I added preload_app! to puma.rb config (note that this disables phased restarts)

@jules-w2
Copy link

jules-w2 commented May 8, 2024

In my case i'm using cluster mode and phased restarts for Puma.
I was never able to get the plugin to work in development or production modes.

So I wrote a gem to manage Capistrano tasks. It manages SolidQueue as a service in production mode and in development I launch it in Procfile

https://github.com/IMMOSQUARE/immosquare-capistrano?tab=readme-ov-file#solidqueue-integration

[Unit]
Description=SolidQueue for <%= "#{fetch(:application)} (#{fetch(:stage)})" %>
After=syslog.target network.target

[Service]
Type=simple
Environment=RAILS_ENV=<%= fetch(:stage) %>
User=<%= fetch(:solid_queue_user) %>
WorkingDirectory=<%= current_path %>

ExecStart=<%= Capistrano::Immosquare::Helpers.expanded_bundle_command %> exec rake solid_queue:start
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/bin/kill -TERM $MAINPID


StandardOutput=append:<%= shared_path %>/log/solid_queue_access.log
StandardError=append:<%= shared_path %>/log/solid_queue_error.log

RestartSec=1
Restart=on-failure

SyslogIdentifier=<%= fetch(:solid_queue_service_unit_name) %>

[Install]
WantedBy=default.target

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants