Slow deployments when the server is running for long periods

[darkcode85]

Hi,

After having a server running for 2/3 weeks the deployment process is very slow, when the server is initially created the deployment takes 1-2 minutes to finish, after 3 weeks the deployment process takes +10 minutes to finish. This process is incremental so after 2 weeks it takes +5 minutes to finish and getting worse if the server is running for more time.

Reviewing the log files I have noticed that the set user to deploy is taking +1 minute each time it is executed when the server is running for 3 weeks, initially the set user to deploy was completed in 10/20 seconds.

Deployment log when the server is created - (set user to deploy completed in 10/20 seconds)

...
[2019-02-20T11:55:28-06:00] INFO: deploy[appname] updated to revision d3a2c74eb1299f57f4393f2910974bab3e61830d
[2019-02-20T11:55:38-06:00] INFO: deploy[appname] set user to deploy
[2019-02-20T11:55:38-06:00] INFO: deploy[appname] set group to www-data
[2019-02-20T11:55:39-06:00] INFO: deploy[appname] copied the cached checkout to /srv/www/appname/releases/20190220175526
[2019-02-20T11:55:58-06:00] INFO: deploy[appname] set user to deploy
[2019-02-20T11:55:58-06:00] INFO: deploy[appname] set group to www-data
[2019-02-20T11:55:58-06:00] INFO: deploy[appname] running callback before_migrate
[2019-02-20T11:55:58-06:00] INFO: deploy[appname] queueing checkdeploy hook /srv/www/appname/releases/20190220175526/deploy/before_migrate.rb
[2019-02-20T11:55:58-06:00] INFO: Processing execute[bundle_install] action run (/var/chef/runs/fc475635-f399-426c-bb86-a7493b9e13cc/local-mode-cache/cache/cookbooks/opsworks_ruby/libraries/helpers.rb line 97)
[2019-02-20T11:55:59-06:00] INFO: execute[bundle_install] ran successfully
[2019-02-20T11:55:59-06:00] INFO: deploy[appname] made pre-migration symlinks
[2019-02-20T11:56:09-06:00] INFO: deploy[appname] set user to deploy
[2019-02-20T11:56:09-06:00] INFO: deploy[appname] set group to www-data
[2019-02-20T11:56:09-06:00] INFO: deploy[appname] migrating deploy with environment ...
[2019-02-20T11:56:24-06:00] INFO: deploy[appname] running callback before_symlink
[2019-02-20T11:56:24-06:00] INFO: deploy[appname] queueing checkdeploy hook /srv/www/appname/releases/20190220175526/deploy/before_symlink.rb
[2019-02-20T11:56:24-06:00] INFO: deploy[appname] purged directories in checkout log, tmp/cache, tmp/pids, public/system, public/assets
[2019-02-20T11:56:24-06:00] INFO: deploy[appname] created directories before symlinking: tmp,public,config,../../shared/cache,../../shared/assets
[2019-02-20T11:56:24-06:00] INFO: deploy[appname] linked shared paths into current release: system => public/system, assets => public/assets, cache => tmp/cache, pids => tmp/pids, log => log, config/application.yml => config/application.yml
[2019-02-20T11:56:24-06:00] INFO: deploy[appname] made pre-migration symlinks
[2019-02-20T11:56:34-06:00] INFO: deploy[appname] set user to deploy
[2019-02-20T11:56:34-06:00] INFO: deploy[appname] set group to www-data
[2019-02-20T11:56:34-06:00] INFO: deploy[appname] linked release /srv/www/appname/releases/20190220175526 into production at /srv/www/appname/current
[2019-02-20T11:56:44-06:00] INFO: deploy[appname] set user to deploy
[2019-02-20T11:56:44-06:00] INFO: deploy[appname] set group to www-data
[2019-02-20T11:56:44-06:00] INFO: deploy[appname] updated symlinks
...

Deployment log after 3 weeks running - (set user to deploy completed in +1 minutes)

...
[2019-03-05T16:40:02-06:00] INFO: deploy[appname] updated to revision a35ad9d17389f0a287c5edcd062ce5271a506e92
[2019-03-05T16:41:07-06:00] INFO: deploy[appname] set user to deploy
[2019-03-05T16:41:07-06:00] INFO: deploy[appname] set group to www-data
[2019-03-05T16:41:14-06:00] INFO: deploy[appname] copied the cached checkout to /srv/www/appname/releases/20190305224000
[2019-03-05T16:42:20-06:00] INFO: deploy[appname] set user to deploy
[2019-03-05T16:42:20-06:00] INFO: deploy[appname] set group to www-data
[2019-03-05T16:42:20-06:00] INFO: deploy[appname] running callback before_migrate
[2019-03-05T16:42:20-06:00] INFO: deploy[appname] queueing checkdeploy hook /srv/www/appname/releases/20190305224000/deploy/before_migrate.rb
[2019-03-05T16:42:20-06:00] INFO: Processing execute[bundle_install] action run (/var/chef/runs/447f4193-8063-4abd-a867-f4f78d8990d8/local-mode-cache/cache/cookbooks/opsworks_ruby/libraries/helpers.rb line 97)
[2019-03-05T16:42:20-06:00] INFO: execute[bundle_install] ran successfully
[2019-03-05T16:42:20-06:00] INFO: deploy[appname] made pre-migration symlinks
[2019-03-05T16:43:25-06:00] INFO: deploy[appname] set user to deploy
[2019-03-05T16:43:25-06:00] INFO: deploy[appname] set group to www-data
[2019-03-05T16:43:25-06:00] INFO: deploy[appname] migrating deploy with environment ...
[2019-03-05T16:43:41-06:00] INFO: deploy[appname] running callback before_symlink
[2019-03-05T16:43:41-06:00] INFO: deploy[appname] queueing checkdeploy hook /srv/www/appname/releases/20190305224000/deploy/before_symlink.rb
[2019-03-05T16:43:41-06:00] INFO: deploy[appname] purged directories in checkout log, tmp/cache, tmp/pids, public/system, public/assets
[2019-03-05T16:43:41-06:00] INFO: deploy[appname] created directories before symlinking: tmp,public,config,../../shared/cache,../../shared/assets
[2019-03-05T16:43:41-06:00] INFO: deploy[appname] linked shared paths into current release: system => public/system, assets => public/assets, cache => tmp/cache, pids => tmp/pids, log => log, config/application.yml => config/application.yml
[2019-03-05T16:43:41-06:00] INFO: deploy[appname] made pre-migration symlinks
[2019-03-05T16:44:45-06:00] INFO: deploy[appname] set user to deploy
[2019-03-05T16:44:45-06:00] INFO: deploy[appname] set group to www-data
[2019-03-05T16:44:45-06:00] INFO: deploy[appname] linked release /srv/www/appname/releases/20190305224000 into production at /srv/www/appname/current
[2019-03-05T16:45:50-06:00] INFO: deploy[appname] set user to deploy
[2019-03-05T16:45:50-06:00] INFO: deploy[appname] set group to www-data
[2019-03-05T16:45:50-06:00] INFO: deploy[appname] updated symlinks
...

Any idea/reason why the set user to deploy is taking so long to be completed each time it is exucuted after having the server running for +3 weeks?

[ajgon]

Honestly - I have no idea. It shouldn't take that long. Does your project grows over time (i.e. it's adding new logs or a lot of other small files)? Have you tried to set ownership manually by SSHing to the server? Does it take so much time as well?

[darkcode85]

Finally I found the cause of this issue, this was caused by the bootsnap gem creating multiple files on tmp folder. I have just removed the bootsnap from production env.