Starting riak fails on latest SmartOS

[RyanGordon]

It appears that running the latest available tarball (2.0pre11) for SmartOS fails to start.

[root@qriak1 ~] # /opt/local/sbin/riak start
/opt/local/sbin/riak[189]: [: argument expected
^C[root@qriak1 ~] # /opt/local/sbin/riak console
18:23:53.850 [warning] cuttlefish_util:filter_by_variable_starts_with/2 has been deprecated. use cuttlefish_variable:filter_by_prefix/2
config is OK
-config /var/db/riak/generated.configs/app.2014.03.07.18.23.53.config -args_file /var/db/riak/generated.configs/vm.2014.03.07.18.23.53.args -vm_args /var/db/riak/generated.configs/vm.2014.03.07.18.23.53.args
/opt/local/sbin/riak[359]: [: argument expected
Exec: /opt/local/lib/riak/erts-5.10.3/bin/erlexec -boot /opt/local/lib/riak/releases/2.0.0pre11/riak -config /var/db/riak/generated.configs/app.2014.03.07.18.23.53.config -args_file /var/db/riak/generated.configs/vm.2014.03.07.18.23.53.args -vm_args /var/db/riak/generated.configs/vm.2014.03.07.18.23.53.args -pa /opt/local/lib/riak/lib/basho-patches -- console
Root: /opt/local/lib/riak
Erlang R16B02-basho3 (erts-5.10.3) [source] [64-bit] [smp:24:24] [async-threads:64] [kernel-poll:true] [dtrace]

[os_mon] memory supervisor port (memsup): Erlang has closed
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
{"Kernel pid terminated",application_controller,"{application_start_failure,riak_api,{{shutdown,{failed_to_start_child,"https://127.0.0.1:8098\",eaddrinuse}},{riak_api_app,start,[normal,[]]}}}"}

Crash dump was written to: /var/log/riak/erl_crash.dump
Kernel pid terminated (application_controller) ({application_start_failure,riak_api,{{shutdown,{failed_to_start_child,"https://127.0.0.1:8098",eaddrinuse}},{riak_api_app,start,[normal,[]]}}})

This is because the following command doesn't work:

[root@node1 ~] # prctl -n process.max-file-descriptor -t basic -P $$ | awk '/max-file-descriptor/ { print $3 }'
[root@node1 ~] # prctl -n process.max-file-descriptor -t priv -P $$ | awk '/max-file-descriptor/ { print $3 }'
65536
[root@node1 ~] #

The first prctl returns nothing so in the code: "if [ ${ULIMIT_S} -lt ${ULIMIT_H} ]; then" is evaluated as "if [ -lt 65536 ]; then"

https://github.com/basho/node_package/blob/a6b6fefed53588970a3649ab71afc96b264758c2/priv/templates/smartos/runner.patch

Here is what shows up without the -t flag:

[root@node1 ~] # prctl -n process.max-file-descriptor -P $$
process: 25862: -bash
process.max-file-descriptor privileged 65536 - deny -
process.max-file-descriptor system 2147483647 max deny -

Should "-t system" be used in place of "-t basic"?

[jaredmorrow]

Well this is very unfortunate, and another case of Joyent changing things in their datasets that breaks backwards compatibility. This same command is running on a customer who runs on Joyent and has three different versions of SmartOS and it works. Can you tell me what global zone and dataset you are using?

Thanks again for reporting this, I'm very sad with this news.

[RyanGordon]

We're running "sdc:sdc:base64:13.3.1" with "joyent_20131218T184706Z" for the platform image. Thanks :)

[jaredmorrow]

I don't know if I'll get to this before the next Riak 2.0 pre build, since this might be something different in the global zone and not the dataset. Sorry for the troubles.

[jaredmorrow]

I will fix the vm.args thing, we switch to an easy riak.conf method (that generates erlang-y config) to make everyone's lives easier with the config files, and I obviously missed some hidden corners.

The nice thing is, it is easily parsed now and is standard config file syntax rather than erlang terms http://docs.basho.com/riak/2.0.0pre11/ops/advanced/configs/configuration-files/

[RyanGordon]

Hey Jared, I just ran this command across all of servers which run several different PI and image versions and it doesn't work on any of them - however I did notice all of our servers run on the base64 dataset instead of something else. Maybe the difference lies in using base64 versus another type of dataset?

[RyanGordon]

Hm, I just tested it on a 'base' dataset instead of a 'base64' dataset and it still doesn't return anything - I wonder how it's working for the other customer

[jaredmorrow]

So the change is probably in the Global Zone. So no matter what dataset you use, the change is in the global zone. I've had varying level of problems dealing with SmartOS because some userland / tools changes are in the global zone (and our customer has mixed global zones) and some userland / tools are in the datasets. So I get X times Y combinations.

[jaredmorrow]

I have a spare machine by my desk, on Monday I'll put the new global zone image on it and try to figure out where things went wrong.

[jaredmorrow]

What does man prctl show you?

I see

SunOS 5.10                User Commands                  PRCTL(1)

NAME
     prctl - get or set the resource controls of running processes,
     tasks, and projects

SYNOPSIS
     prctl [-P] [-t [basic | privileged | system]]
          [-n name [-srx] [-v value] [-e | -d action] [-p pid]]
          [-i idtype] id...

At the top, where -t basic is fine.

[RyanGordon]

That's odd, it shows the same thing:

[root@node1 ~] # man prctl
SunOS 5.10 User Commands PRCTL(1)

NAME
prctl - get or set the resource controls of running processes,
tasks, and projects

SYNOPSIS
prctl [-P] [-t [basic | privileged | system]]
[-n name [-srx] [-v value] [-e | -d action] [-p pid]]
[-i idtype] id...

Maybe a bug in prctl?

[RyanGordon]

Hi,

I've been talking to Joyent about this issue and it would be helpful if we could know the dataset image(s)/Platform Images that you can reproduce -t basic working on?

Joyent also says that using -t system can be used in place of -t basic and that should do the same thing.

[jaredmorrow]

Well it works on their cloud, b/c our customer who is on Joyent and has machines with the 1.6, 1.8, and 13.1 datasets all have '-t basic' still working. The man pages still document '-t basic' as well.

My global zones are all at: SmartOS Live Image v0.147+ build: 20130629T040542Z

[jaredmorrow]

If I simply change it to -t system all is well according to them?

[RyanGordon]

Hey Jared,

I've passed that information along to them

And yes, that is what they are telling me and it does appear to produce the right values from my end as well 👍