Trinidad/Rails application does not know its root path on Ubuntu 14.04 when started using trinidad_init_services

[kgx]

I recently upgraded some application servers from Ubuntu 12.04 to 14.04 LTS.

I have noticed an unusual behavior: my app works fine when started manually using "trinidad" command, but when I start it using the jsvc wrapper it does not know its root path.

For example, when started via command line "trinidad" command:
Rails.root returns "/opt/app/myapp/releases/20140512165627"

When started via jsvc wrapper,
Rails.root return "/" , which results in a variety of errors...

At first I thought it might have something to do with the current release symlink (/opt/apps/myapp/current => /opt/app/myapp/releases/20140512165627) , but I have ruled that out by switching the references in /etc/init.d/trinidad to point at the actual release folder.

I am using trinidad 1.4.6, trinidad_jars 1.4.0, jruby-rack 1.1.14, trinidad_init_services 1.2.3, rails 3.2.18, and openjdk-7-amd64 7u55 (all the latest). The only component in my stack which has changed is Ubuntu.

[kgx]

Here is my startup script:

#!/usr/bin/env bash
# Trinidad init.d script using `jsvc` customized from the jsvc-wrapper.sh
# http://github.com/nicobrevin/jruby-jsvc/blob/master/bin/jsvc-wrapper.sh
#
# set on a per script/daemon basis (optional) :
# APP_NAME - name of your application (defaults to "trinidad")

# https://github.com/trinidad/trinidad_init_services/wiki/Installing-JSVC
JSVC=/usr/bin/jsvc
JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/jre
JRUBY_HOME=/opt/jruby
APP_PATH=/opt/apps/myapp/current
RUBY_SCRIPT=/opt/jruby/lib/ruby/gems/shared/gems/trinidad_init_services-1.2.3/lib/trinidad/daemon.rb
# The user rights for the running daemon, if you don't bind directly to 80 than
# it's always a good idea to run trinidad with non-root user rights.
# Make sure this user has rights on PIDFILE and LOG_FILE paths.
# Leaving this empty will cause the daemon to always run as whichever user calls
# the script (which during init is equivalent to running it as root).
RUN_USER="vagrant"
# Set this to "1" to echo commands before running them (for troubleshooting) :
ECHO_COMMAND=""

# Add here the options that Trinidad needs to run your application,
# but DO NOT delete the -d option, i.e -e production
TRINIDAD_OPTS="-d /opt/apps/myapp/current -e production"

PIDFILE=/opt/apps/rentallimo/shared/pids/trinidad.pid
LOG_FILE=/opt/apps/rentallimo/shared/log/trinidad.log

# Create pidfile directory if missing
# Todo: Remove PIDFILE_DIR creation once jsvc can be started
# as root using the -user option without any issues.
PIDFILE_DIR=$(dirname $PIDFILE)
if [ ! -d "$PIDFILE_DIR" ] ; then
  mkdir -p $PIDFILE_DIR
  if [ ! -z "$RUN_USER" ] ; then
    chown -R $RUN_USER $PIDFILE_DIR
  fi
fi

# Implements the jsvc Daemon interface.
MAIN_CLASS=com.msp.jsvc.JRubyDaemon

CLASSPATH=/opt/jruby/lib/ruby/gems/shared/gems/trinidad_init_services-1.2.3/trinidad-libs/jruby-jsvc.jar:/opt/jruby/lib/ruby/gems/shared/gems/trinidad_init_services-1.2.3/trinidad-libs/commons-daemon.jar:/opt/jruby/lib/jruby.jar

JRUBY_SHELL=/bin/sh

JRUBY_NATIVE_PATH=""
if [ -d "$JRUBY_HOME/lib/native/" ]; then
  for d in $JRUBY_HOME/lib/native/*`uname -s`; do
    if [ -z "$JRUBY_NATIVE_PATH" ]; then
      JRUBY_NATIVE_PATH="$d"
    else
      JRUBY_NATIVE_PATH="$JRUBY_NATIVE_PATH:$d"
    fi
  done
fi

# JAVA_MEM, JAVA_MEM_MIN & JAVA_STACK are compatible with jruby's binary

if [ -z "$JAVA_MEM" ] ; then
  JAVA_MEM=-Xmx500m
fi
# backwards compatibility we accept the non-full option (e.g. JAVA_MEM=500m)
if [[ "$JAVA_MEM" != "-Xmx"* ]] ; then
  JAVA_MEM="-Xmx$JAVA_MEM"
fi

# we accept the non-full option (e.g. JAVA_MEM_MIN=500m)
if [[ -n "$JAVA_MEM_MIN" && "$JAVA_MEM_MIN" != "-Xms"* ]] ; then
  JAVA_MEM_MIN="-Xms$JAVA_MEM_MIN"
fi
if [ -z "$JAVA_MEM_MIN" ] ; then
  JAVA_MEM_MIN=""
fi

if [ -z "$JAVA_STACK" ] ; then
  JAVA_STACK=-Xss2048k
fi
# backwards compatibility we accept the non-full option (e.g. JAVA_STACK=2048k)
if [[ "$JAVA_STACK" != "-Xss"* ]] ; then
  JAVA_STACK="-Xss$JAVA_STACK"
fi

# A custom (daemon) script will be started (not jruby -S) thus JRUBY_OPTS
# won't be used here, feel free to hand tune JAVA_OPTS here instead ...

JAVA_OPTS="$JAVA_MEM $JAVA_MEM_MIN $JAVA_STACK -Xbootclasspath/a:$JRUBY_HOME/lib/jruby.jar"

# force file.encoding to UTF-8 :
if [[ -z "$JAVA_ENCODING" ]]; then
  JAVA_OPTS="$JAVA_OPTS -Dfile.encoding=UTF-8"
fi

JAVA_PROPS="$JAVA_PROPS \
  -Djruby.shell=$JRUBY_SHELL \
  -Djffi.boot.library.path=$JRUBY_NATIVE_PATH \
  -Djruby.home=/opt/jruby -Djruby.lib=/opt/jruby/lib -Djruby.script=jruby -Djruby.daemon.module.name=Trinidad -Djruby.compat.version=RUBY1_9 "

PROC_NAME=${SCRIPT_NAME:-${APP_NAME:-"trinidad"}}

JSVC_ARGS="-home $JAVA_HOME \
  $JSVC_ARGS_EXTRA \
  -wait 20 \
  -pidfile $PIDFILE \
  -procname jsvc-$PROC_NAME \
  -jvm server
  -outfile $LOG_FILE \
  -errfile &1"

#
# Stop/Start
#

if [[ -n "$RUN_USER" && $EUID -eq 0 ]]; then
  JSVC="sudo -u $RUN_USER $JSVC"
fi

STOP_COMMAND="$JSVC $JSVC_ARGS -stop $MAIN_CLASS"
START_COMMAND="$JSVC $JSVC_ARGS -cp $CLASSPATH $JAVA_PROPS $JAVA_OPTS $MAIN_CLASS $RUBY_SCRIPT $TRINIDAD_OPTS"

cd $APP_PATH || exit 1

case "$1" in
    start)
      if [ -e "$PIDFILE" ]; then
          CUR_PID=$(<"$PIDFILE")
          COM_NAME=$(ps -p $CUR_PID -o command= | cut -d ' ' -f 1)
          PROCESS_NAME="jsvc-$PROC_NAME"
          if [ "$COM_NAME" != "$PROCESS_NAME" ]; then
              rm $PIDFILE
              $START_COMMAND
          else
              echo "Pidfile already exists for pid $CUR_PID, not starting."
              exit 1
          fi
      else
          echo "Starting $PROC_NAME daemon..."
          if [ -n "$ECHO_COMMAND" ]; then echo $START_COMMAND; fi
          $START_COMMAND
          EXIT_CODE=$?
          if [ "$EXIT_CODE" != 0 ]; then
              echo "Daemon exited with status: $EXIT_CODE. Check pidfile and log"
          fi
      fi
      ;;
    stop)
      if [ -e "$PIDFILE" ]; then
          echo "Stopping $PROC_NAME daemon..."
          if [ -n "$ECHO_COMMAND" ]; then echo $STOP_COMMAND; fi
          $STOP_COMMAND
      else
          echo "No pid file, not stopping."
          exit 1
      fi
      ;;
    restart)
      if [ -e "$PIDFILE" ]; then
          echo "Stopping $PROC_NAME daemon..."
          if [ -n "$ECHO_COMMAND" ]; then echo $STOP_COMMAND; fi
          $STOP_COMMAND
      fi
      if [ -e "$PIDFILE" ]; then
          echo "Pidfile still present, $PROC_NAME hasn't stopped"
          exit 1
      else
          if [ -n "$ECHO_COMMAND" ]; then echo $START_COMMAND; fi
          $START_COMMAND
          EXIT_CODE=$?
          if [ "$EXIT_CODE" != 0 ]; then
              echo "Daemon exited with status: $EXIT_CODE. Check pidfile and log"
          fi
      fi
      ;;
    status)
      if [ "$PIDFILE" ]; then
          PID=`cat $PIDFILE`
          OUTPUT=`ps -o pid $PID | grep "$PID"`
          if [ ${#OUTPUT} -gt 0 ]; then
              echo "Service running with pid: $PID"
          else
              echo "Pidfile present, but process not running"
          fi
      else
          echo "No pidfile present"
      fi
      ;;
    *)
      echo "Unrecognised command. Usage trinidad [ start | stop | restart ]"
      ;;
esac

[kares]

thanks a lot - could you try if this still happens with 1.5.0.B1 (simply gem install trinidad --pre) ?

[kgx]

I just tried 1.5.0.B1, and I now get the following error. I uninstalled 1.4.6 to make sure it wasn't picking anything up from the old version.

Exception in thread "jsvc-daemon-main" org.jruby.exceptions.RaiseException: (NameError) uninitialized constant Trinidad::CommandLineParser
    at org.jruby.RubyModule.const_missing(org/jruby/RubyModule.java:2690)
    at opt.jruby.lib.ruby.gems.shared.gems.trinidad_init_services_minus_1_dot_2_dot_3.lib.trinidad.daemon.start(/opt/jruby/lib/ruby/gems/shared/gems/trinidad_init_services-1.2.3/lib/trinidad/daemon.rb:24)

[kares]

I see - thanks ... this broke compatibility with init services but I've just fixed it.
would it be possible for you to build the trinidad gem from master and install it to try it out ?

[kgx]

@kares Your fix got init services working again, but the original issue remains.

[kares]

@kgx thanks a lot will need to look at it than ... really thought it's been a dir resolution on Trinidad's side ...

you can probably (hopefully - feel free to let us know here) work aroud by root_dir: /abs/path in trinidad.yml

[kgx]

Thank you for looking into this. I tried the root_dir workaround and it did not work. I added the startup parameter "--config /opt/apps/myapp/current/config/trinidad.yml" just to make sure it was finding the config file, but that did not help either.

[kares]

changing the work directory to the app's path should resolve this for you (although I was unable to reproduce, actually reproduced another issue) add: JSVC_ARGS="$JSVC_ARGS -cwd $APP_PATH" after the JSVC_ARGS definition ... also this issue is likely resolved on JRuby-Rack's side as well (not yet released - expected as 1.1.15)

[kgx]

@kares Thank you! This workaround works for now. Do you know what in jruby-rack is causing this?

[kares]

yes