Not a Passenger issue (was: Passenger 5.3.0 raise error on startup)

[PikachuEXE]

Issue report

Fill in as much as possible so that we can understand, find and fix the problem.

Are you sure this is a bug in Passenger?
Yes

Please try with the newest version of Passenger to avoid issues that have already been fixed

Question 1: What is the problem?

• What is the expected behavior?
  No error occurred when running passenger start
• What is the actual behavior?

=============== Phusion Passenger Standalone web server started ===============
PID file: /users/pikachuexe/projects/spacious/spacious-rails/tmp/pids/passenger.3000.pid
Log file: /users/pikachuexe/projects/spacious/spacious-rails/log/passenger.3000.log
Environment: development
Accessible via: http://0.0.0.0:3000/

You can stop Phusion Passenger Standalone by pressing Ctrl-C.
Problems? Check https://www.phusionpassenger.com/library/admin/standalone/troubleshooting/
===============================================================================
App 46775 stdout:
App 46775 stderr: This program may only be invoked from Passenger (error: $PASSENGER_SPAWN_WORK_DIR not set).
[ E 2018-05-10 09:40:12.0225 46761/Tq age/Cor/App/Implementation.cpp:231 ]: Cannot render an error page: Cannot open '/users/pikachuexe/.rvm/gems/ruby-2.5.1/gems/passenger-5.3.0/resources/templates/error_layout.css' for reading: No such file or directory (errno=2)
     in 'void Passenger::ApplicationPool2::processAndLogNewSpawnException(Passenger::SpawnException &, const Passenger::ApplicationPool2::Options &, const SpawningKit::ConfigPtr &)' (Implementation.cpp:207)
     in 'void Passenger::SpawningKit::SmartSpawner::throwPreloaderSpawnException(const string &, SpawnException::ErrorKind, BackgroundIOCapturerPtr &, const Options &, const DebugDirPtr &)' (SmartSpawner.h:149)
     in 'string Passenger::SpawningKit::SmartSpawner::negotiatePreloaderStartup(Passenger::SpawningKit::SmartSpawner::StartupDetails &)' (SmartSpawner.h:630)
     in 'void Passenger::SpawningKit::SmartSpawner::startPreloader()' (SmartSpawner.h:278)
     in 'virtual Passenger::SpawningKit::Result Passenger::SpawningKit::SmartSpawner::spawn(const Options &)' (SmartSpawner.h:803)
     in 'void Passenger::ApplicationPool2::Group::spawnThreadRealMain(const SpawningKit::SpawnerPtr &, const Passenger::ApplicationPool2::Options &, unsigned int)' (SpawningAndRestarting.cpp:95)

[ E 2018-05-10 09:40:12.0225 46761/Tq age/Cor/App/Implementation.cpp:305 ]: Could not spawn process for application /users/pikachuexe/projects/spacious/spacious-rails: An error occurred while starting up the preloader. It exited before signalling successful startup back to Phusion Passenger.
  Error ID: daa98c25
  Message from application: An error occurred while starting up the preloader. It exited before signalling successful startup back to Phusion Passenger. Please read <a href="https://github.com/phusion/passenger/wiki/Debugging-application-startup-problems">this article</a> for more information about this problem.<br>
<h2>Raw process output:</h2>
<pre>
This program may only be invoked from Passenger (error: $PASSENGER_SPAWN_WORK_DIR not set).
</pre>

[ E 2018-05-10 09:40:12.0396 46761/T7 age/Cor/Con/CheckoutSession.cpp:285 ]: [Client 1-1] Cannot checkout session because a spawning error occurred. The identifier of the error is daa98c25. Please see earlier logs for details about the error.
[ E 2018-05-10 09:40:12.0397 46761/T7 age/Cor/Con/CheckoutSession.cpp:361 ]: [Client 1-1] Cannot render an error page: Cannot open '/users/pikachuexe/.rvm/gems/ruby-2.5.1/gems/passenger-5.3.0/resources/templates/error_layout.css' for reading: No such file or directory (errno=2)
     in 'void Passenger::Core::Controller::endRequestWithErrorResponse(Passenger::Core::Client **, Request **, const Passenger::StaticString &, const Passenger::SpawnException *)' (CheckoutSession.cpp:350)
     in 'void Passenger::Core::Controller::writeSpawnExceptionErrorResponse(Passenger::Core::Client *, Request *, const boost::shared_ptr<SpawnException> &)' (CheckoutSession.cpp:282)
     in 'void Passenger::Core::Controller::reportSessionCheckoutError(Passenger::Core::Client *, Request *, const ExceptionPtr &)' (CheckoutSession.cpp:235)
     in 'void Passenger::Core::Controller::sessionCheckedOutFromEventLoopThread(Passenger::Core::Client *, Request *, const AbstractSessionPtr &, const ExceptionPtr &)' (CheckoutSession.cpp:148)
     in 'void Passenger::Core::Controller::sessionCheckedOutFromAnotherThread(Passenger::Core::Client *, Request *, AbstractSessionPtr, ExceptionPtr)' (CheckoutSession.cpp:114) -- Client 1-1

[ N 2018-05-10 09:40:12.0554 46761/T5 age/Cor/SecurityUpdateChecker.h:517 ]: Security update check: no update found (next check in 24 hours)
[ E 2018-05-10 09:40:12.0697 46761/T5 Crypto.cpp:990 ]: Removing Passenger private key from keychain failed: write permissions error. Please remove the private key from the certificate labeled Phusion Passenger Open Source in your keychain.

• How can we reproduce it? Please try to provide a sample application (or Virtual Machine) demonstrating the issue. Otherwise, if we can't reproduce it, we might have to ask you a number of followup questions or run certain commands to try and figure out the problem.

$ gem install passenger
$ passenger start

Question 2: Passenger version and integration mode:
open source 5.3.0 standalone

Question 3: OS or Linux distro, platform (including version):
OS X 10.12.6 Sierra, x86_64

Question 4: Passenger installation method:

Your answer:
[x] RubyGems + Gemfile
[ ] RubyGems, no Gemfile
[ ] Phusion APT repo
[ ] Phusion YUM repo
[ ] OS X Homebrew
[ ] source tarball
[ ] Other, please specify:

Question 5: Your app's programming language (including any version managers) and framework (including versions):
Ruby 2.5.1, RVM, Rails 5.1.5

Question 6: Are you using a PaaS and/or containerization? If so which one?
Nope

Question 7: Anything else about your setup that we should know?
Nope

---

We strive for quality and appreciate you taking the time to submit a report! Please note that if you want guaranteed response times and priority issue support we encourage you to join our enterprise customer base. They also provide us with the means to continue our high level of open source support!

[FooBarWidget]

The behavior exhibited by the output suggests that you are somehow running a PassengerAgent belonging to a previous version of Passenger, in combination with a PassengerRoot belonging to version 5.3.

Can you tell me what you see if you run this?

passenger start --debug-nginx-config

Also, if you look at the output, you will find a passenger_root directive there. Can you tell me what you see if you run this?

$VALUE_OF_PASSENGER_ROOT/buildout/support-binaries/PassengerAgent --version

[remomueller]

Hi, I'm also getting various errors in starting applications with Ruby 2.5.1, Rails 5.2.0, on both MacOS 10.13.4 and CentOS 7.3. They are also in the "Spawn subprocess section" either failing at "Basic initialization before exec()" or "Setup environment". The strange thing is, sometimes certain apps will load, and others won't, but it's sporadic in which load (no particular order).

Here's a look at one of the errors I'm seeing:

The Phusion Passenger application server tried to start the web application. In doing so, Passenger had to first start an internal helper tool called the "preloader", which in turn had to be started through another internal tool called the "SpawnEnvSetupper". But the SpawnEnvSetupper encountered an operating system error.

The error reported by the operating system is:

setgroups(26, ...) failed: Invalid argument (errno=22)

The stdout/stderr output of the subprocess so far is:

Error: setgroups(26, ...) failed: Invalid argument (errno=22)

Let me know if you'd like me to keep following this issue (which may be related), or if you'd like me to open a new issue. Passenger 5.2.3 (for reference) has been working as expected.

[CamJN]

@remomueller I think your problem is actually this: #2077

[PikachuEXE]

@FooBarWidget

$ passenger start --debug-nginx-config
##########################################################################
#  Passenger Standalone is built on the same technology that powers
#  Passenger for Nginx, so any configuration option supported by Passenger
#  for Nginx can be applied to Passenger Standalone as well. You can do
#  this by direct editing the Nginx configuration template that is used by
#  Passenger Standalone.
#
#  This file is the original template. DO NOT EDIT THIS FILE DIRECTLY.
#  Instead, make a copy of this file and pass the `--nginx-config-template`
#  parameter to Passenger Standalone.
#
#  Learn more about using the Nginx configuration template at:
#  https://www.phusionpassenger.com/library/config/standalone/intro.html#nginx-configuration-template
#
#  *** NOTE ***
#  If you customize the template file, make sure you keep an eye on the
#  original template file and merge any changes. New Phusion Passenger
#  features may require changes to the template file.
##############################################################

master_process on;
daemon on;
error_log '/users/pikachuexe/projects/spacious/spacious-rails/log/passenger.3000.log' ;
pid '/users/pikachuexe/projects/spacious/spacious-rails/tmp/pids/passenger.3000.pid';


worker_processes 1;
events {
    worker_connections 4096;
}

http {
    log_format debug '[$time_local] $msec  "$request" $status conn=$connection sent=$bytes_sent body_sent=$body_bytes_sent';
    include '/users/pikachuexe/.rvm/gems/ruby-2.5.1/gems/passenger-5.3.0/resources/mime.types';
    passenger_root '/users/pikachuexe/.rvm/gems/ruby-2.5.1/gems/passenger-5.3.0';
    passenger_abort_on_startup_error on;
    passenger_ctl pidfiles_to_delete_on_exit '["/var/folders/xb/rb9b_7y12b9d12swg8lwzd6m0000gn/T/passenger-standalone.1f14epa/temp_dir_toucher.pid"]';
    passenger_ctl integration_mode standalone;
    passenger_ctl standalone_engine nginx;
    passenger_user_switching off;
    passenger_log_level 3;


    ### BEGIN your own configuration options ###
    # This is a good place to put your own config
    # options. Note that your options must not
    # conflict with the ones Passenger already sets.
    # Learn more at:
    # https://www.phusionpassenger.com/library/config/standalone/intro.html#nginx-configuration-template

    ### END your own configuration options ###

    default_type application/octet-stream;
    types_hash_max_size 2048;
    server_names_hash_bucket_size 64;
    client_max_body_size 1024m;
    access_log off;
    keepalive_timeout 60;
    underscores_in_headers on;
    gzip on;
    gzip_comp_level 3;
    gzip_min_length 150;
    gzip_proxied any;
    gzip_types text/plain text/css text/json text/javascript
        application/javascript application/x-javascript application/json
        application/rss+xml application/vnd.ms-fontobject application/x-font-ttf
        application/xml font/opentype image/svg+xml text/xml;




    server {
        server_name _;
        listen 0.0.0.0:3000;
        listen 0.0.0.0:3001 ssl http2;
        ssl_certificate /Users/PikachuEXE/Projects/spacious/spacious-rails/.ssl/development.crt;
        ssl_certificate_key /Users/PikachuEXE/Projects/spacious/spacious-rails/.ssl/development.key;
        root '/users/pikachuexe/projects/spacious/spacious-rails/public';
        passenger_app_root '/users/pikachuexe/projects/spacious/spacious-rails';
        passenger_enabled on;
        passenger_app_env 'development';
        passenger_spawn_method 'smart';
        passenger_load_shell_envvars off;

        # Rails asset pipeline & webpacker support.
        location ~ "^/(assets|packs)/.+-([0-9a-f]{32}|[0-9a-f]{64}|[0-9a-f]{20})\..+" {
            error_page 490 = @static_asset;
            error_page 491 = @dynamic_request;
            recursive_error_pages on;

            if (-f $request_filename) {
                return 490;
            }
            if (!-f $request_filename) {
                return 491;
            }
        }
        location @static_asset {
            gzip_static on;
            expires max;
            add_header Cache-Control public;
            add_header ETag "";
        }
        location @dynamic_request {
            passenger_enabled on;
        }


        ### BEGIN your own configuration options ###
        # This is a good place to put your own config
        # options. Note that your options must not
        # conflict with the ones Passenger already sets.
        # Learn more at:
        # https://www.phusionpassenger.com/library/config/standalone/intro.html#nginx-configuration-template

        ### END your own configuration options ###
    }
    passenger_pre_start http://0.0.0.0:3000/;
}

$VALUE_OF_PASSENGER_ROOT/buildout/support-binaries/PassengerAgent --version
which means /users/pikachuexe/.rvm/gems/ruby-2.5.1/gems/passenger-5.3.0/buildout/support-binaries/PassengerAgent --version since the file/folder does not exist

$ tree /users/pikachuexe/.rvm/gems/ruby-2.5.1/gems/passenger-5.3.0/buildout
/users/pikachuexe/.rvm/gems/ruby-2.5.1/gems/passenger-5.3.0/buildout
└── ruby
    └── ruby-2.5.1-x86_64-macosx
        ├── Makefile
        ├── mkmf.log
        ├── passenger_native_support.bundle
        └── passenger_native_support.o

2 directories, 4 files

[FooBarWidget]

What do you see if you run these?

~/.passenger/support-binaries/5.3.0/PassengerAgent --version
md5 ~/.passenger/support-binaries/5.3.0/PassengerAgent

[PikachuEXE]

$ ~/.passenger/support-binaries/5.3.0/PassengerAgent version
Usage: PassengerAgent <SUBCOMMAND> [options...]
Phusion Passenger version 5.3.0.
Type '/users/pikachuexe/.passenger/support-binaries/5.3.0/PassengerAgent <SUBCOMMAND> --help' for help on a specific subcommand.

Daemon subcommands:
  core
  watchdog

Utility subcommands:
  system-metrics
  exec-helper

$ md5 ~/.passenger/support-binaries/5.3.0/PassengerAgent
MD5 (/users/pikachuexe/.passenger/support-binaries/5.3.0/PassengerAgent) = 7506d9b3af114d9c9eec5f82701f5fcd

[FooBarWidget]

I am unable to reproduce the issue with gem install.

Could it be that you were developing Passenger through its Git repo? If you used 'passenger start' back then, then you may have ended up installing a prerelease version of PassengerAgent into ~/.passenger.

Try wiping ~/.passenger/standalone. Does that fix things?

[PikachuEXE]

$ ls ~/.passenger/standalone
4.0.50 4.0.52 4.0.53 4.0.55 4.0.56 4.0.57 4.0.58 4.0.59 4.0.60

$ rm -rf ~/.passenger/standalone

But passenger start throws the same error as reported above

So I try renaming ~/.passenger/support-binaries to something else and now it works

$ ls ~/.passenger/support-binaries
5.0.30 5.1.10 5.1.11 5.1.12 5.1.2  5.1.3  5.1.4  5.1.7  5.1.8  5.2.0  5.2.1  5.2.2  5.2.3  5.3.0

$ mv ~/.passenger/support-binaries ~/.passenger/support-binaries-old

I guess it's left the local version of passenger when I am using my branch of passenger for #2047

TL;DR
Clear/Rename ~/.passenger/support-binaries for MacOS when in doubt