New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Puma take about 2-3 minute to start in Production, how do I debug the issue? #1717

Closed
1c7 opened this Issue Jan 25, 2019 · 11 comments

Comments

Projects
None yet
3 participants
@1c7
Copy link

1c7 commented Jan 25, 2019

Steps to reproduce

  1. mina deploy --verbose

  2. ssh ubuntu@xx.xx.xx.xx

cd /var/www/example.com/current
bundle exec puma -C ./deploy/puma-server-production-9292.rb -d

puma-server-production-9292.rb

# frozen_string_literal: true

bind "tcp://0.0.0.0:9292"

threads 1, 5
environment "production"
workers 2

app_name = "example.com"
app_dir = File.expand_path("../..", __FILE__)
app_dir_parent = File.expand_path("../../", __FILE__)
app_path = "/var/www/#{app_name}/"

directory "#{app_path}current"
pidfile "#{app_path}/shared/tmp/pids/puma.pid"
state_path "#{app_path}/shared/tmp/sockets/puma.state"

UCloud_data_folder = "/var/www/wit_log"
stdout_redirect "#{UCloud_data_folder}/witticism/log/production9292.stdout.log", "#{UCloud_data_folder}/witticism/log/production9292.stderr.log", true

before_fork do
  ActiveRecord::Base.connection_pool.disconnect! if defined?(ActiveRecord)
end

on_worker_boot do
  ActiveRecord::Base.establish_connection if defined?(ActiveRecord)
end
  1. it take long time to start up

Expected behavior

puma start in 1 minute

Actual behavior

puma start take 2-3 minute sometime even more

System configuration

Ruby version:
2.5.1
Rails version:
5.2.2
Puma version:
3.12.0

@1c7

This comment has been minimized.

Copy link
Author

1c7 commented Jan 25, 2019

How big is our codebase?

60 model, 40 controller or so. still a fairly small project

How many gem in Gemfile?

# frozen_string_literal: true

source "https://rubygems.org"

ruby "2.5.1"

gem "mail"

gem "skylight"

gem 'bcrypt'

gem "rails-settings-cached"

gem "syslogger", "~> 1.6.0"
gem "lograge", "~> 0.3.1"
# https://github.com/roidrage/lograge

git_source(:github) { |repo_name| "https://github.com/#{repo_name}.git" }

gem "whenever", require: false

gem "httparty"

# Sentry Error Tracking
# https://sentry.io/for/rails/
gem "sentry-raven"

# Blockchain (Nebulas)
gem 'bitcoin-secp256k1', '~> 0.4.0' # gem 'neb' 依赖的库 # https://rubygems.org/gems/bitcoin-secp256k1
gem 'neb', '0.1.3'


gem "hashids"

gem "fast_jsonapi"
gem "devise-jwt", "~> 0.5.6"
gem "active_model_serializers"
gem "qiniu", ">= 6.8.0"
gem "aliyun-sms"   # 短信发送 https://github.com/VICTOR-LUO-F/aliyun-sms

gem "mina", require: false

gem 'bootsnap', '~> 1.3', '>= 1.3.2', require: false
gem 'rails', '~> 5.2', '>= 5.2.2'
gem "puma", "~> 3.12"

gem "pg"
gem "pghero" # https://github.com/ankane/pghero # A performance dashboard for Postgres

gem "rack-attack"
# https://github.com/kickstarter/rack-attack

gem "http_accept_language" # https://github.com/iain/http_accept_language

# OAuth Provider
gem "doorkeeper" # https://github.com/doorkeeper-gem/doorkeeper
gem "doorkeeper-i18n"

gem "bulk_insert" # https://github.com/jamis/bulk_insert

# gem "carrierwave"
# gem "carrierwave-aliyun"

# Lazy load
gem "mini_magick", require: false

gem "devise"
gem "devise-encryptable"

gem "notifications" # https://github.com/rails-engine/notifications

# Cache
# gem "second_level_cache"
# https://github.com/hooopo/second_level_cache

gem "action-store" # https://github.com/rails-engine/action-store
# https://ruby-china.org/topics/32262

gem "kaminari"

gem "omniauth"
gem "omniauth-github"

# Permission
gem "cancancan" # https://github.com/CanCanCommunity/cancancan

# Redis
gem "hiredis"
gem "redis"
gem "redis-namespace"
gem "redis-objects"
gem 'redis-rails', '~> 5.0', '>= 5.0.2' # https://github.com/redis-store/redis-rails

# Setting
# gem "rails-settings-cached"
# gem 'rails-settings-cached', '0.7.0'
# https://github.com/huacnlee/rails-settings-cached
# https://github.com/ledermann/rails-settings

# 队列
gem 'sidekiq', '~> 5.2', '>= 5.2.5'

gem "dalli" # High performance memcached client for Ruby
# https://github.com/petergoldstein/dalli

# API cors
gem "rack-cors", require: "rack/cors"
gem "rack-utf8_sanitizer"

gem "bundler-audit", require: false # https://github.com/rubysec/bundler-audit

group :development do
  gem "derailed" # https://github.com/schneems/derailed_benchmarks
  gem "better_errors"  # https://github.com/charliesome/better_errors
  gem "spring"
  gem "spring-commands-rspec"
  gem "faker"
  gem "binding_of_caller"
  gem "annotate"
  gem "listen"
end

group :development, :test do
  gem "capybara"
  gem "database_cleaner"
  gem "factory_bot_rails"
  gem "letter_opener"

  gem "rspec-rails"
  # gem "rubocop", ">= 0.49.0", require: false
end
@1c7

This comment has been minimized.

Copy link
Author

1c7 commented Jan 25, 2019

I want to solve this myself,
but I just don't know where to start.
what config I should fine tune.
which component eat up a lot of time etc.

If anyone can provide some insight that would be very helpful
Thanks you in advance

@venarius

This comment has been minimized.

Copy link

venarius commented Jan 27, 2019

Have the same problem! Puma is taking longer and longer with every cap deploy

@1c7

This comment has been minimized.

Copy link
Author

1c7 commented Jan 30, 2019

Ahh.. Still don't know how to fix
considering change to Unicorn / Passanger.

@1c7

This comment has been minimized.

Copy link
Author

1c7 commented Jan 30, 2019

A lot of Github Issue have no reply at all.
Seem like no one maintaining puma?

@1c7

This comment has been minimized.

Copy link
Author

1c7 commented Jan 30, 2019

One example:
I haven't build CI/CD system yet. (would get it done soon)
(and I haven't write any test, I know it's bad..)

sometime a bug happen in Production and I can fix it in 20 second.
but boot up puma alone take 4-5 minute

@evanphx

This comment has been minimized.

Copy link
Member

evanphx commented Jan 30, 2019

There are people maintaining it here, we're all just volunteers, so please be kind.

As for this issue, it's something about your app startup, not puma. Switch to another webserver won't change the situation.

@1c7

This comment has been minimized.

Copy link
Author

1c7 commented Jan 30, 2019

@evanphx oh my native language is not English.
so if there are tone problem I am sorry, I didn't mean it.

And Thanks for the suggestion!

@1c7

This comment has been minimized.

Copy link
Author

1c7 commented Jan 30, 2019

Let me try Unicorn / Passenger see if problem still exist.
If so. there must be something wrong in my code.
I not. probably because wrong server config / wrong puma config / both / some other weird thing

Thank again

@1c7

This comment has been minimized.

Copy link
Author

1c7 commented Jan 30, 2019

would close this issue within a week

@1c7

This comment has been minimized.

Copy link
Author

1c7 commented Jan 30, 2019

Remove bootsnap and puma
Add passenger, somehow seem working. ( take only couple second to start )
Doing further investigate. maybe somehow bootsnap case problem?

Update: 2019-2-15

because I use Docker now. the problem goes away.
I suspect that when I am not using Docker. the problem come from my server.
probably environment variable wrong or some config wrong or my code wrong. who knows

anyway. now the problem gone.

@1c7 1c7 closed this Feb 6, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment