Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Process aborted! signo=SIGABRT(6) on FreeBSD 11.2-STABLE #2097

Closed
iron-udjin opened this Issue Jun 23, 2018 · 33 comments

Comments

Projects
None yet
9 participants
@iron-udjin
Copy link

iron-udjin commented Jun 23, 2018

Hello,

-- OS: FreeBSD 11.2-STABLE r335568M, boost-libs-1.67.0, ruby-2.3, nginx-devel-1.15.0
-- Passenger version and integration mode: open source 5.3.1 with nginx as a builtin module.
-- Passenger installation method: from ports, as rubygem
-- Application: Redmine

After passenger upgrade from 5.2.3 to 5.3.1 on FreeBSD 11.2 when I try to restart nginx I see:

# service nginx restart
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Stopping nginx.
Waiting for PIDS: 53797.
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Starting nginx.
nginx: [alert] Unable to start the Phusion Passenger watchdog: it seems to have been killed with signal SIGABRT during startup (-1: Unknown error)

passenger-crash-log:

[ pid=65135, timestamp=1529724482 ] Process aborted! signo=SIGABRT(6), reason=#65543, si_addr=0x0, randomSeed=1529724482
[ pid=65135 ] Crash log dumped to /var/tmp/passenger-crash-log.1529724482
[ pid=65135 ] Date, uname and ulimits:
Sat Jun 23 06:28:02 +03 2018
FreeBSD 11.2-STABLE FreeBSD 11.2-STABLE #0 r335568M: Sat Jun 23 01:44:21 +03 2018     root@mm:/usr/obj/usr/src/sys/MM  amd64 amd64
cpu time               (seconds, -t)  unlimited
file size           (512-blocks, -f)  unlimited
data seg size           (kbytes, -d)  33554432
stack size              (kbytes, -s)  524288
core file size      (512-blocks, -c)  unlimited
max memory size         (kbytes, -m)  unlimited
locked memory           (kbytes, -l)  131072
max user processes              (-u)  89999
open files                      (-n)  3773268
virtual mem size        (kbytes, -v)  unlimited
swap limit              (kbytes, -w)  unlimited
socket buffer size       (bytes, -b)  unlimited
pseudo-terminals                (-p)  unlimited
kqueues                         (-k)  unlimited
umtx shared locks               (-o)  unlimited
[ pid=65135 ] Phusion Passenger version: 5.3.1
[ pid=65135 ] libc backtrace not available.
--------------------------------------
[ pid=65135 ] Open files and file descriptors:
*** ERROR: cannot execute lsof: No such file or directory (errno=2)
Falling back to another mechanism for dumping file descriptors.
ls: illegal option -- v
usage: ls [-ABCFGHILPRSTUWZabcdfghiklmnopqrstuwxy1,] [-D format] [file ...]
ERROR: Could not run 'ls' to dump file descriptor information!
--------------------------------------
[ pid=65135 ] Dumping a backtrace with crash-watch...
Found gdb at: /usr/bin/gdb
/usr/bin/gdb is broken on FreeBSD. Looking for an alternative...
*** ERROR ***: '/usr/bin/gdb' is broken on FreeBSD. Please install the one from the devel/gdb port instead. If you want to use another gdb

Coredump backtrace:

(gdb) bt
#0  0x0000000802a6dd9a in thr_kill () from /lib/libc.so.7
#1  0x0000000802a6dd64 in __raise (s=6) at /usr/src/lib/libc/gen/raise.c:52
#2  0x0000000802a6dcd9 in abort () at /usr/src/lib/libc/stdlib/abort.c:65
#3  0x0000000000418ca3 in boost::throw_exception<boost::bad_function_call> (e=...) at src/cxx_supportlib/vendor-modified/boost/throw_exception.hpp:69
#4  0x000000000041fa28 in boost::function1<Passenger::Json::Value, Passenger::Json::Value const&>::operator() (this=<optimized out>, a0=...) at src/cxx_supportlib/vendor-modified/boost/function/function_template.hpp:757
#5  Passenger::ConfigKit::Store::applyInspectFilters (this=<optimized out>, doc=...) at src/cxx_supportlib/ConfigKit/Store.h:219
#6  0x0000000000416932 in Passenger::ConfigKit::Store::inspect (this=<optimized out>) at src/cxx_supportlib/ConfigKit/Store.h:542
#7  0x00000000004116bb in Passenger::LoggingKit::Context::inspectConfig (this=<optimized out>) at src/cxx_supportlib/LoggingKit/Implementation.cpp:592
#8  0x000000000048dc0a in Passenger::Agent::Fundamentals::initializeLoggingKit (processName=0x954cc8 "Passenger watchdog", config=..., loggingKitTranslator=..., loggingKitPreInitFunc=@0x7fffffffc578: 0x0)
    at src/agent/Shared/Fundamentals/Initialization.cpp:492
#9  0x000000000048c7fd in Passenger::Agent::Fundamentals::initializeAgent (argc=3, argv=0x7fffffffc5b0, processName=0x954cc8 "Passenger watchdog", config=..., loggingKitTranslator=..., 
    optionParser=0x4c22e0 <parseOptions(int, char const**, Passenger::ConfigKit::Store&)>, loggingKitPreInitFunc=@0x7fffffffc578: 0x0, argStartIndex=2) at src/agent/Shared/Fundamentals/Initialization.cpp:610
#10 0x00000000004b2680 in initializeBareEssentials (argc=3, argv=0x7fffffffec88, wo=...) at src/agent/Watchdog/WatchdogMain.cpp:804
#11 0x00000000004b1358 in watchdogMain (argc=3, argv=0x7fffffffec88) at src/agent/Watchdog/WatchdogMain.cpp:1296
#12 0x000000000048ba18 in dispatchSubcommand (argc=3, argv=0x7fffffffec88) at src/agent/AgentMain.cpp:82
#13 0x000000000048b8f6 in main (argc=3, argv=0x7fffffffec88) at src/agent/AgentMain.cpp:105

nginx configuration:

passenger_root /usr/local/lib/ruby/gems/2.3/gems/passenger;
passenger_ruby /usr/local/bin/ruby23;
passenger_user redmine;
passenger_group redmine;

server {
    listen  xxx.xxx.xxx.xxx:80;
    # rcvbuf=8k sndbuf=8k;
    # accept_filter=httpready accept_filter=dataready;
    server_name  example.com;
    listen  xxx.xxx.xxx.xxx:443 ssl http2;
    ssl_certificate      /usr/local/etc/nginx/ssl/mm.crt;
    ssl_certificate_key  /usr/local/etc/nginx/ssl/mm.key;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;    
    ssl_prefer_server_ciphers on;
    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_buffer_size 8k;
    ssl_ciphers "EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH:EDH+aRSA:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS:!RC4";
    ssl_session_timeout         10m;
    ssl_dhparam /usr/local/etc/nginx/ssl/mm-dhparams.pem;
    passenger_enabled on;
    root   /home/redmine/redmine/public;
    if ($ssl_protocol = "") {
        rewrite ^   https://$server_name$request_uri? permanent;
    }
    access_log  /var/log/nginx/redmine-proxy-access;
    error_log   /var/log/nginx/redmine-proxy-error;
}

P.S: I've tried version 5.3.2, but result is the same.

How can I fix this problem?
Thank you!

@diemer25

This comment has been minimized.

Copy link

diemer25 commented Jul 3, 2018

Hi, I have the exact same problem.

Here's the gdb full bt from 5.3.2, should it help:

(gdb) bt full
#0  0x0000000802a32e8a in thr_kill () from /lib/libc.so.7
No symbol table info available.
#1  0x0000000802a32e54 in __raise (s=6) at /usr/src/lib/libc/gen/raise.c:52
        id = 100625
#2  0x0000000802a32dc9 in abort () at /usr/src/lib/libc/stdlib/abort.c:65
        act = <optimized out>
#3  0x0000000000419493 in boost::throw_exception<boost::bad_function_call> (e=...) at src/cxx_supportlib/vendor-modified/boost/throw_exception.hpp:69
No locals.
#4  0x0000000000420218 in boost::function1<Passenger::Json::Value, Passenger::Json::Value const&>::operator() (this=<optimized out>, a0=...) at src/cxx_supportlib/vendor-modified/boost/function/function_template.hpp:757
No locals.
#5  Passenger::ConfigKit::Store::applyInspectFilters (this=<optimized out>, doc=...) at src/cxx_supportlib/ConfigKit/Store.h:219
        subdoc = @0x8050b55d0: {_vptr$Value = 0x969948 <vtable for Passenger::Json::Value+16>, static null = @0xcca5c0, static nullRef = @0xcca5c0, static minLargestInt = -9223372036854775808, static maxLargestInt = 9223372036854775807, static maxLargestUInt = 18446744073709551615, static minInt = -2147483648, 
          static maxInt = 2147483647, static maxUInt = 4294967295, static minInt64 = -9223372036854775808, static maxInt64 = 9223372036854775807, static maxUInt64 = 18446744073709551615, value_ = {int_ = 34443755360, uint_ = 34443755360, real_ = 1.7017476237136297e-313, bool_ = 96, 
            string_ = 0x80501ff60 "@U\v\005\b", map_ = 0x80501ff60}, type_ = Passenger::Json::objectValue, allocated_ = 0, comments_ = 0x0, start_ = 0, limit_ = 0}
        effectiveValue = <optimized out>
        key = <optimized out>
        entry = <optimized out>
        userValue = <optimized out>
        it = <optimized out>
#6  0x0000000000417122 in Passenger::ConfigKit::Store::inspect (this=<optimized out>) at src/cxx_supportlib/ConfigKit/Store.h:542
        result = {_vptr$Value = 0x969948 <vtable for Passenger::Json::Value+16>, static null = @0xcca5c0, static nullRef = @0xcca5c0, static minLargestInt = -9223372036854775808, static maxLargestInt = 9223372036854775807, static maxLargestUInt = 18446744073709551615, static minInt = -2147483648, 
          static maxInt = 2147483647, static maxUInt = 4294967295, static minInt64 = -9223372036854775808, static maxInt64 = 9223372036854775807, static maxUInt64 = 18446744073709551615, value_ = {int_ = 34444380544, uint_ = 34444380544, real_ = 1.7017785119073027e-313, bool_ = 128, 
            string_ = 0x8050b8980 "`\352\001\005\b", map_ = 0x8050b8980}, type_ = Passenger::Json::objectValue, allocated_ = 0, comments_ = 0x0, start_ = 0, limit_ = 0}
        it = <optimized out>
#7  0x0000000000411eab in Passenger::LoggingKit::Context::inspectConfig (this=<optimized out>) at src/cxx_supportlib/LoggingKit/Implementation.cpp:592
        l = <optimized out>
#8  0x000000000048e3fa in Passenger::Agent::Fundamentals::initializeLoggingKit (processName=0x972248 "Passenger watchdog", config=..., loggingKitTranslator=..., loggingKitPreInitFunc=@0x7fffffffc388: 0x0) at src/agent/Shared/Fundamentals/Initialization.cpp:492
        initialConfig = {_vptr$Value = 0x969948 <vtable for Passenger::Json::Value+16>, static null = @0xcca5c0, static nullRef = @0xcca5c0, static minLargestInt = -9223372036854775808, static maxLargestInt = 9223372036854775807, static maxLargestUInt = 18446744073709551615, static minInt = -2147483648, 
          static maxInt = 2147483647, static maxUInt = 4294967295, static minInt64 = -9223372036854775808, static maxInt64 = 9223372036854775807, static maxUInt64 = 18446744073709551615, value_ = {int_ = 34443753088, uint_ = 34443753088, real_ = 1.701747511461915e-313, bool_ = 128, 
            string_ = 0x80501f680 "\200\312\001\005\b", map_ = 0x80501f680}, type_ = Passenger::Json::objectValue, allocated_ = 0, comments_ = 0x0, start_ = 0, limit_ = 0}
        dump = {_vptr$Value = 0x969948 <vtable for Passenger::Json::Value+16>, static null = @0xcca5c0, static nullRef = @0xcca5c0, static minLargestInt = -9223372036854775808, static maxLargestInt = 9223372036854775807, static maxLargestUInt = 18446744073709551615, static minInt = -2147483648, 
          static maxInt = 2147483647, static maxUInt = 4294967295, static minInt64 = -9223372036854775808, static maxInt64 = 9223372036854775807, static maxUInt64 = 18446744073709551615, value_ = {int_ = 34444380544, uint_ = 34444380544, real_ = 1.7017785119073027e-313, bool_ = 128, 
            string_ = 0x8050b8980 "`\352\001\005\b", map_ = 0x8050b8980}, type_ = Passenger::Json::objectValue, allocated_ = 0, comments_ = 0x0, start_ = 0, limit_ = 0}
#9  0x000000000048cfed in Passenger::Agent::Fundamentals::initializeAgent (argc=2, argv=0x7fffffffc3c0, processName=0x972248 "Passenger watchdog", config=..., loggingKitTranslator=..., optionParser=0x4c2ad0 <parseOptions(int, char const**, Passenger::ConfigKit::Store&)>, 
    loggingKitPreInitFunc=@0x7fffffffc388: 0x0, argStartIndex=2) at src/agent/Shared/Fundamentals/Initialization.cpp:610
        seedStr = 0x0
        __p = {function = 0x96f1e6 "void Passenger::Agent::Fundamentals::initializeAgent(int, char ***, const char *, ConfigKit::Store &, const ConfigKit::Translator &, Passenger::Agent::Fundamentals::OptionParserFunc, const Passenger::"..., source = 0x96f2e1 "src/agent/Shared/Fundamentals/Initialization.cpp", 
          u = {data = 0x0, dataFunc = {func = 0x0, userData = 0x805064db8}}, line = 582, m_detached = false, m_hasDataFunc = false}
        e = <error reading variable>
#10 0x00000000004b2e70 in initializeBareEssentials (argc=2, argv=0x7fffffffea90, wo=...) at src/agent/Watchdog/WatchdogMain.cpp:804
        oldOomScore = {<std::__1::__basic_string_common<true>> = {<No data fields>}, static __short_mask = 1, static __long_mask = 1, __r_ = {<std::__1::__compressed_pair_elem<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::__rep, 0, false>> = {__value_ = {{__l = {
                    __cap_ = 0, __size_ = 0, __data_ = 0x0}, __s = {{__size_ = 0 '\000', __lx = 0 '\000'}, __data_ = '\000' <repeats 22 times>}, __r = {__words = {0, 0, 
                      0}}}}}, <std::__1::__compressed_pair_elem<std::__1::allocator<char>, 1, true>> = {<std::__1::allocator<char>> = {<No data fields>}, <No data fields>}, <No data fields>}, static npos = 18446744073709551615}
#11 0x00000000004b1b48 in watchdogMain (argc=2, argv=0x7fffffffea90) at src/agent/Watchdog/WatchdogMain.cpp:1296
        wo = {px = 0x0, pn = {pi_ = 0x0}}
        instanceDirToucher = {px = 0x800cd4689 <matched_symbol+457>, pn = {pi_ = 0x0}}
        watchers = {<std::__1::__vector_base<boost::shared_ptr<AgentWatcher>, std::__1::allocator<boost::shared_ptr<AgentWatcher> > >> = {<std::__1::__vector_base_common<true>> = {<No data fields>}, __begin_ = 0x118, __end_ = 0x0, 
            __end_cap_ = {<std::__1::__compressed_pair_elem<boost::shared_ptr<AgentWatcher>*, 0, false>> = {
                __value_ = 0x0}, <std::__1::__compressed_pair_elem<std::__1::allocator<boost::shared_ptr<AgentWatcher> >, 1, true>> = {<std::__1::allocator<boost::shared_ptr<AgentWatcher> >> = {<No data fields>}, <No data fields>}, <No data fields>}}, <No data fields>}
        uidBeforeLoweringPrivilege = 8
        e = <error reading variable>
        e = <error reading variable>
#12 0x000000000048c208 in dispatchSubcommand (argc=2, argv=0x7fffffffea90) at src/agent/AgentMain.cpp:82
No locals.
#13 0x000000000048c0e6 in main (argc=2, argv=0x7fffffffea90) at src/agent/AgentMain.cpp:105
No locals.
(gdb) 

Also, this might be an operating system issue because I cannot reproduce under FreeBSD 11.1 (we're running 11.2 now).

Open for tests, just ask. Thank you.

@iron-udjin

This comment has been minimized.

Copy link
Author

iron-udjin commented Jul 3, 2018

I don't think that the bug is OS related. Passenger was upgraded from previous version which was working fine on 11.2-STABLE. Possibly it's boost related issue.

@diemer25

This comment has been minimized.

Copy link

diemer25 commented Jul 3, 2018

Hi, you are right: Passenger 5.0.30 works with FreeBSD 11.2, so it has to be something in Passenger itself. I'll keep on looking into this tomorrow.

@mkastner

This comment has been minimized.

Copy link

mkastner commented Jul 5, 2018

Yep, ran into the same problem. Tried to downgrade nginx-1.12.2_12,2 and passenger-5.1.12, which is running on my other machines, but I can't get i running again. It would be great if someone would cast an eye on the issue.

@mkastner

This comment has been minimized.

Copy link

mkastner commented Jul 6, 2018

I might be wrong, but to me it looks as as if this line and the entire following block simply can't work in FreeBSD 11.2

["/usr/local/bin/gdb76", "/usr/local/bin/gdb66"].each do |candidate|

There simply is no devel/gdb version with these binaries. The current 11.2 package and port provide a binary named gdb81 but that's not contained in the binaries array.

Furthermore I've tried to monkey patch the array and added gdb81 to it. This creates a hell of a crash.

As I wrote, I might be wrong, but it looks to me like passenger is not ready yet for FreeBSD 11.2

@iron-udjin

This comment has been minimized.

Copy link
Author

iron-udjin commented Jul 6, 2018

mkastner, there is no need to specify gdb81 because port devel/gdb creates symlink: /usr/local/bin/gdb -> gdb81

@mkastner

This comment has been minimized.

Copy link

mkastner commented Jul 6, 2018

@iron-udjin But in the the code of the gdb_controller.rb, if I understand it correctly, /usr/local/bin/gdb is never being checked.

The program on FreeBSD looks for these gdb paths:

  • /usr/bin/gdb
  • /usr/local/bin/gdb76
  • /usr/local/bin/gdb66

And therefore it throws an error complaining that there's no gdb and stops.

@diemer25

This comment has been minimized.

Copy link

diemer25 commented Jul 7, 2018

Hi,
ok, glad someone is looking into this.

The lsof and gdb part is not really relevant to this bug because it gets triggered AFTER the watchdog cannot be started.

Anyhow, if you are really interested in improving the code there are the following bugs that are FreeBSD related:

  1. lsof is not detected, so in any case control goes down to ls;
  2. ls is called with "-v" option which is not supported is BSDs, despite a comment nearby stating differently. Even if you make ls to work you need to have /proc mounted, which is not the standard setup in FreeBSD;
  3. stock gdb is considered broken just because reasons;
  4. attempt to locate a different gdb is hardcoded to gdb version 6.6 and 7.6. Current port version is 8.1 which is not detected. Furthermore this part of the code does not look into /usr/local/bin/gdb which is a link to the installed version.

If you fix things so that ls works, /proc is mounted, or lsof gets detected or ports gdb kicks in, you still have the root problem there. Plus, gdb81 bombs the machine.

Back to the relevant part I found this is a very subtle bug because it only triggers on some versions of Passenger and on recent FreeBSD.

In particular if you run 11.1 stable + Passenger 5.3.2 it DOES work, but when you upgrade to 11.2 stable you get the bug.

I think the problem lies in the fact that Passenger runs a tailor made version of boost that is probably a spin off from an old (and buggy?) version. I would not personally invest time and efforts in debugging old software so the current status for Passenger and FreeBSD is that it is supported only up to 11.1.

Passenger people shall be interested over this problem and take action. If an old version of boost is used, maybe they want to upgrade to a newer version or make sure Passenger can work with the latest boost, stock version and not a tailor made one.

Do any of you know how to attract Passenger people attention to this? I think it a serious bug affecting many people. The number of FreeBSD supported systems is going to shrink as people update to 11.2 and beyond.

On top of that users usually find out this bug the hard way: i.e. when they updated their server and get the nice surprise of have Passenger not working. Usually a downgrade is not desirable and many people do not know how to it.

Regards,

@mkastner

This comment has been minimized.

Copy link

mkastner commented Jul 8, 2018

@diemer25 Thanks for your systematic and precise description of the problem. I guess FreeBSD is simply considered exotic compared to linux. Although, there should still be a hell of a lot more people running into this when updgrading to 11.2 than just the few posting here.

What I also noticed, I have another machine which I upgraded from 11.1 to 11.2. At the moment, I haven't upgraded the applications yet. So it's a an 11.2 Machine with 11.1 built apps. It gives me the creeps when I think about it, but it's running for the time being.

So, but I haven't tested this yet, the workaround for the time being might be to build nginx and passenger-nginx packages on an 11.1 machine and then install them on the 11.2 machine. A I said, I haven't tested it yet but I will give it a shot. Not a good idea.

addendum

I am quite surprised how few interest there is on this problem. I seems like there are comparatively few FressBSD servers running with passenger/nginx/apache. Considering how many problems I had to solve the past few years just get the systems up and running, it's most likely no surprise.

That said, since nobody from the passenger community seems to bother, I will check for alternatives. http://pm2.keymetrics.io/ seems to look promising, at least if you're a node developer.

Don't get me wrong here: passenger is a great product, it's performant and it's open source, but if you're using FreeBSD, then it can become a big black hole, where a lot of time and nightshifts get drawn into.

@skillcoder

This comment has been minimized.

Copy link

skillcoder commented Jul 9, 2018

Upgrade today from 11.0-RELEASE-p9 to

FreeBSD 11.2-RELEASE #0

# pkg info | g rubygem-passenger 
rubygem-passenger-nginx-5.3.3  Modules for running Ruby on Rails and Rack applications

# pkg info | g gdb
gdb-8.1_2                      GNU GDB of newer version than comes with the system

# pkg info | g lsof
lsof-4.92.b,8                  Lists information about open files (similar to fstat(1))

Result:
Unable to start the Phusion Passenger watchdog: it seems to have been killed with signal SIGABRT during startup (-1: Unknown error)

[ pid=64230, timestamp=1531101181 ] Process aborted! signo=SIGABRT(6), reason=#65543, si_addr=0x0, randomSeed=1531101181
[ pid=64230 ] Crash log dumped to /var/tmp/passenger-crash-log.1531101181
[ pid=64230 ] Date, uname and ulimits:
Mon Jul  9 04:53:01 MSK 2018
FreeBSD 11.2-RELEASE FreeBSD 11.2-RELEASE #0 r335510: Fri Jun 22 04:32:14 UTC 2018     root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64 amd64
cpu time               (seconds, -t)  unlimited
file size           (512-blocks, -f)  unlimited
data seg size           (kbytes, -d)  33554432
stack size              (kbytes, -s)  524288
core file size      (512-blocks, -c)  unlimited
max memory size         (kbytes, -m)  unlimited
locked memory           (kbytes, -l)  unlimited
max user processes              (-u)  19558
open files                      (-n)  22500
virtual mem size        (kbytes, -v)  unlimited
swap limit              (kbytes, -w)  unlimited
socket buffer size       (bytes, -b)  unlimited
pseudo-terminals                (-p)  unlimited
kqueues                         (-k)  unlimited
umtx shared locks               (-o)  unlimited
[ pid=64230 ] Phusion Passenger version: 5.3.3
[ pid=64230 ] libc backtrace not available.
--------------------------------------
[ pid=64230 ] Open files and file descriptors:
*** ERROR: cannot execute lsof: No such file or directory (errno=2)
Falling back to another mechanism for dumping file descriptors.
ls: illegal option -- v
usage: ls [-ABCFGHILPRSTUWZabcdfghiklmnopqrstuwxy1,] [-D format] [file ...]
ERROR: Could not run 'ls' to dump file descriptor information!
--------------------------------------
[ pid=64230 ] Dumping a backtrace with crash-watch...
Found gdb at: /usr/bin/gdb
/usr/bin/gdb is broken on FreeBSD. Looking for an alternative...
*** ERROR ***: '/usr/bin/gdb' is broken on FreeBSD. Please install the one from the devel/gdb port instead. If you want to use another gdb

Also i cant just compile port www/rubygem-passenger
cuz it only compile apache version, so i change Makefile to compile nginx version

UPD:
Reinstall with
sudo portmaster -d www/rubygem-passenger@nginx
Same problem

@bartmichu

This comment has been minimized.

Copy link

bartmichu commented Jul 10, 2018

if you are really interested in improving the code there are the following bugs that are FreeBSD related
...

I would also add one thing to that list. Passenger fails to find PCRE in standalone mode due to the fact that FreeBSD uses /usr/local/ path.

@diemer25

This comment has been minimized.

Copy link

diemer25 commented Jul 11, 2018

Hi,
we need to have @FooBarWidget into this, really, as the number of people left behind is growing day by day.
Anyone know how to get his attention? Thank you.

@mkastner

This comment has been minimized.

Copy link

mkastner commented Jul 11, 2018

@diemer25 This is about the only place I would know, where the passenger guys actually might look into. It seriously scares me, because there's a whole bunch of servers running with nginx/passenger I need to upgrade.

@iron-udjin

This comment has been minimized.

Copy link
Author

iron-udjin commented Jul 11, 2018

@diemer25 I guess it's easier to move to any other application server than wait for fix. Personally I already migrated to uWSGI which works great.
For developers more interesting spend attention on bugs of commercial version because they bring them money. "Only business, nothing more".

@FooBarWidget

This comment has been minimized.

Copy link
Member

FooBarWidget commented Jul 11, 2018

Hey guys sorry about the delay. I just looked at the issue and it indeed seems to be something related to Boost and FreeBSD 11.2.

So the stack trace says that this line fails:

userValue = entry.schemaEntry->inspectFilter(userValue);

entry.schemaEntry->inspectFilter is a function object. But if I go one level deeper it tells me that it fails here:

 if (this->empty())
     boost::throw_exception(bad_function_call());

So it looks like inspectFilter refers to a null function object.

But earlier in the code I see this:

if (entry.schemaEntry->inspectFilter == NULL) {
   it.next();
   continue;
}

...so the code already checks for null. It seems this check doesn't work?

Not sure why. But can anyone running FreeBSD 11.2 try this change and tell me whether it fixes the problem? Edit src/cxx_supportlib/ConfigKit/Store.h, look for

if (entry.schemaEntry->inspectFilter == NULL) {

Change it to

if (entry.schemaEntry->inspectFilter.empty()) {

As for the gdb issue: years ago I found that /usr/bin/gdb is broken. I don't remember the exact details, such as the version of FreeBSD. Maybe this has changed nowadays and the blacklist should be removed.

@mkastner

This comment has been minimized.

Copy link

mkastner commented Jul 11, 2018

@FooBarWidget I would do it immediately, but I need to match a deadline until 16.00 CET. But if nobody has tested your proposed change until then I will check it. Sorry!

@diemer25

This comment has been minimized.

Copy link

diemer25 commented Jul 11, 2018

Hi @FooBarWidget, it works for me. Thank you for taking the time and solve the issue.

If you push this upstream, I'd appreciate if you could let me know and I'll ask the FreeBSD community to upgrade the ports so we do not need to patch 5.3.3. Thank you.

@FooBarWidget

This comment has been minimized.

Copy link
Member

FooBarWidget commented Jul 11, 2018

Thanks for confirming.

I would like to push a fix upstream, but likely not this one. My change fixed it, but we should also figure out why.

We vendor Boost 1.64. I've taken a look at the changes in Boost since then, but the changelogs do not mention any changes in Boost.Function, nor do they mention any relevant FreeBSD-specific fixes.

I'll spin up a FreeBSD 11.2 VM myself and see whether I can reproduce the problem, and check whether upgrading Boost "magically" fixes it.

@iron-udjin

This comment has been minimized.

Copy link
Author

iron-udjin commented Jul 11, 2018

@diemer25
Here is my FreeBSD bug report: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229392
Please use it for patches/fixes.

@FooBarWidget

This comment has been minimized.

Copy link
Member

FooBarWidget commented Jul 11, 2018

Here is what I found so far. The following test program...

boost::function<void ()> f;
printf("is null: %d\n", f == NULL);

...prints 1 on macOS (as it should) and 0 on FreeBSD 11.2. Digger deeper, on FreeBSD the == operator calls this function (in boost/function/function_base.hpp):

#  if !(defined(__GNUC__) && __GNUC__ == 3 && __GNUC_MINOR__ <= 3)
// Comparisons between boost::function objects and arbitrary function
// objects. GCC 3.3 and before has an obnoxious bug that prevents this
// from working.
template<typename Functor>
  BOOST_FUNCTION_ENABLE_IF_NOT_INTEGRAL(Functor, bool)
  operator==(const function_base& f, Functor g)
  {
    if (const Functor* fp = f.template target<Functor>())
      return function_equal(*fp, g);
    else return false;
  }

...and returns false (the else branch). While on macOS it calls this function:

inline bool operator!=(const function_base& f,
                       detail::function::useless_clear_type*)
{
  return !f.empty();
}

Just looking at the #if there already gives me the feeling that it is not right. Boost is trying to work around a GCC 3 bug but it wrongly detects Clang as GCC 3.

@FooBarWidget

This comment has been minimized.

Copy link
Member

FooBarWidget commented Jul 11, 2018

Upgrading to Boost 1.67 does not fix the problem. This problem must be reported to Boost, it likely affects more than just Passenger.

@diemer25

This comment has been minimized.

Copy link

diemer25 commented Jul 11, 2018

@FooBarWidget and @iron-udjin,
in order to fix the issue we need to have an updated gem, yes or yes.

Boring explanation:
FreeBSD ports, as it is currently implemented, is not very DRY because you need to patch both nginx source AND the gem. Plus, many people still might relay on the gem itself which is not patched.

Proposed fix:
Please push this upstream asap and update the gem, then we can have FreeBSD relay on the new gem (5.3.4?) and close the issue.

@FooBarWidget I understand there are more open issues with Passenger on FreeBSD, such as the ones I mention in a previous post (lsof, ls, gdb) but these are non blocking and can be solved at a later time.
We might actually start a new thread and take it from there.

Re: Boost issue, thank you for digging into it. Are you reporting this to Boost?

@FooBarWidget

This comment has been minimized.

Copy link
Member

FooBarWidget commented Jul 11, 2018

Reported bug to Boost: https://svn.boost.org/trac10/ticket/13632

Let's see what they say before I fix anything inside Passenger.

@mkastner

This comment has been minimized.

Copy link

mkastner commented Jul 11, 2018

@FooBarWidget Thanks so much for your efforts. I really appreciate it.

@VVD

This comment has been minimized.

Copy link

VVD commented Jul 11, 2018

FreeBSD 11.1 have clang 4.0.0, but FreeBSD 11.2 have clang 6.0.0.
Passenger compiled with GCC 6.4.0 from ports on FreeBSD 11.2 work fine.

Probably boost don't support clang 6.0.0.

@jbeich

This comment has been minimized.

Copy link

jbeich commented Jul 15, 2018

@FooBarWidget wrote:

boost::function<void ()> f;
printf("is null: %d\n", f == NULL);

FreeBSD and OpenBSD promote NULL to nullptr for -std=c++11 or greater. Clang 6 (like GCC 6) switched to -std=gnu++14 by default. So, the comparison fails because f is not a pointer.

@FooBarWidget

This comment has been minimized.

Copy link
Member

FooBarWidget commented Jul 15, 2018

@jbeich Indeed, that seems to have something to do with it. The following test program...

boost::function<void ()> f;
printf("is nullptr: %d\n", f == nullptr);
printf("is 0: %d\n", f == 0);

...prints the same result on macOS and on FreeBSD 11.12, namely 0 and 1.

The Boost.Function documentation says this about comparisons:

Invoking a function object wrapper that does not actually contain a function object is a precondition violation, much like trying to call through a null function pointer, and will throw a bad_function_call exception). We can check for an empty function object wrapper by using it in a boolean context (it evaluates true if the wrapper is not empty) or compare it against 0.

It says nothing about nullptr or NULL. This is a good case for arguing that the problem is in Passenger after all. Let me dive around in the source code a bit more.

FooBarWidget added a commit that referenced this issue Jul 15, 2018

Replace NULL with nullptr
Because nullptr did not exist before C++11, we introduce a Preamble.h
which is included when compiling .cpp files. Preamble.h #defines
nullptr to `(void *) 0` if the compiler does not support C++11.

By standardizing our codebase on using nullptr instead of NULL, we
have a higher probability of detecting issues such as
#2097 (comment)

FooBarWidget added a commit that referenced this issue Jul 15, 2018

Do not compare boost::function with nullptr
This always returns false. Closes #2097.

FooBarWidget added a commit that referenced this issue Jul 15, 2018

Do not compare boost::function with NULL
On some platforms such as FreeBSD 11.12, NULL is
an alias for nullptr. Comparing a boost::function
to nullptr always returns false.

Closes #2097.
@FooBarWidget

This comment has been minimized.

Copy link
Member

FooBarWidget commented Jul 16, 2018

The problem has been fixed now. I also spent time looking for other instances of the same problem and fixed those too. A new version will be released soon.

@FooBarWidget FooBarWidget added this to the 5.3.4 milestone Jul 16, 2018

@mkastner

This comment has been minimized.

Copy link

mkastner commented Jul 17, 2018

@FooBarWidget Thanks. Great Job!

@FooBarWidget

This comment has been minimized.

Copy link
Member

FooBarWidget commented Jul 17, 2018

I wrote a blog post that details my debugging journey in case anyone is interested: https://www.joyfulbikeshedding.com/blog/2018-07-17-do-not-compare-boost-function-with-null.html

@VVD

This comment has been minimized.

Copy link

VVD commented Jul 17, 2018

@FooBarWidget, you wrote a lot of times FreeBSD 11.12, but it's FreeBSD 11.2.

@FooBarWidget

This comment has been minimized.

Copy link
Member

FooBarWidget commented Jul 18, 2018

Thanks for pointing that out.

@PeterMozesMerl

This comment has been minimized.

Copy link

PeterMozesMerl commented Aug 22, 2018

"On top of that users usually find out this bug the hard way: i.e. when they updated their server and get the nice surprise of have Passenger not working. Usually a downgrade is not desirable and many people do not know how to it."

True. To me, the salvation was that Passenger 5.3.1 that I compiled on FreeBSD 11.1 worked fine on 11.2. Anything 5.x (before 5.3.4) didn’t work "only" if I compiled it with 11.2.

Fortunately, ruby 2.4.2 and all gems I use and which compile binaries kept working from 11.1 to 11.2. (Of course, before I started upgrading my production servers, I’ve had tested the whole process on two FreeBSD test servers, except for Passenger).

Although it was painful, I could restore working binaries from other production or CI servers. When I upgrade FreeBSD, even if it’s only certain packages, I prefer to reinstall everything Rails-related from scratch. I used to start it with rm -rf .rvm/. I learned it in the hard way that mv .rvm/ rvm_old/ saves lives.

I have no idea what I would have done if the old binaries didn’t work. (Ideas: crying, installing Puma).

But it did.

5.3.4 works fine on FreeBSD 11.2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.