Worker Start-up Issue

[martingrayson]

I'm attempting to have the various airflow services run on startup of the airflow host (a ubuntu box).

They all work fine except I'm have an issue where by the worker doesn't seem to want run at startup but works fine when called manually (using the same init.d script).

I'm getting the following error in my worker logs:

Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 17, in <module>
    args.func(args)
  File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 376, in worker
    sp = subprocess.Popen(['airflow', 'serve_logs'], env=env)
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

Do you have any pointers as to what environment variables etc could be causing this?

[bolkedebruin]

Basically "airflow" is not in the path of your worker. And your have to make sure it is.

Is your airflow installed globally? What user are you running your worker under? If not root (which I would suggest) you will need to change to this user fully (ie. su - airflow). Did you test your script under this user?

[martingrayson]

My startup script runs as root but changes to the airflow user. I've added a couple of print statements and the AIRFLOW_HOME env variable seems to be present which is rather confusing...

Commenting out the serve logs call seems to allow the worker to start, but this isn't a great solution 🙁

On 4 Dec 2015, at 19:38, bolkedebruin notifications@github.com wrote:

Basically "airflow" is not in the path of your worker. And your have to make sure it is.

Is your airflow installed globally? What user are you running your worker under? If not root (which I would suggest) you will need to change to this user fully (ie. su - airflow). Did you test your script under this user?

—
Reply to this email directly or view it on GitHub.

[bolkedebruin]

You are looking in the wrong place I think. Like I wrote: the airflow binary (/bin/airflow?) does not seem the be in $PATH for the airflow user. Make sure it is.

[pedrorjbr]

@bolkedebruin : hello! I am facing the same problem.

I installed airflow under a user called ubuntu. I clone the repository and installed airflow with python setup.py install.
Thenn create a init.d script with:
start-stop-daemon --start --chuid ubuntu --quiet --pidfile $PIDFILE --exec $DAEMON

See below the boot.log file :

Running the Gunicorn server with 4 syncworkers on host 0.0.0.0 and port 8080...
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 4, in
import('pkg_resources').run_script('airflow==1.6.1', 'airflow')
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 742, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 1667, in run_script
exec(code, namespace, namespace)
File "/usr/local/lib/python2.7/dist-packages/airflow-1.6.1-py2.7.egg/EGG-INFO/scripts/airflow", line 17, in
args.func(args)
File "/usr/local/lib/python2.7/dist-packages/airflow-1.6.1-py2.7.egg/airflow/bin/cli.py", line 338, in webserver
'airflow.www.app:cached_app()'])
File "/usr/lib/python2.7/subprocess.py", line 710, in init
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
t.log file:

How can I install it under root user?
What do u recommend?

[martingrayson]

You've pretty much described my exact setup, I'm running Ubuntu with a similar daemon script.

My Ubuntu user has all the airflow env variables in the profile and the airflow binary should be on the path. I've yet to do any further digging...

On 9 Dec 2015, at 17:26, pedrorjbr notifications@github.com wrote:

@bolkedebruin : hello! I am facing the same problem.

I installed airflow under a user called ubuntu. I clone the repository and installed airflow with python setup.py install.
Thenn create a init.d script with:
start-stop-daemon --start --chuid ubuntu --quiet --pidfile $PIDFILE --exec $DAEMON

Looking the booRunning the Gunicorn server with 4 syncworkers on host 0.0.0.0 and port 8080...
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 4, in
import('pkg_resources').run_script('airflow==1.6.1', 'airflow')
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 742, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/local/lib/python2.7/dist-packages/pkg_resources/init.py", line 1667, in run_script
exec(code, namespace, namespace)
File "/usr/local/lib/python2.7/dist-packages/airflow-1.6.1-py2.7.egg/EGG-INFO/scripts/airflow", line 17, in
args.func(args)
File "/usr/local/lib/python2.7/dist-packages/airflow-1.6.1-py2.7.egg/airflow/bin/cli.py", line 338, in webserver
'airflow.www.app:cached_app()'])
File "/usr/lib/python2.7/subprocess.py", line 710, in init
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
t.log file:

How can I install it under root user?
What do u recommend?

—
Reply to this email directly or view it on GitHub.

[pedrorjbr]

I solved using:
sudo start-stop-daemon --start --chuid ubuntu --quiet --pidfile $PIDFILE --exec $DAEMON

[martingrayson]

@pedrorjbr what has changed in your daemon script? Those two commands look the same to me excluding sudo. Is sudo even required as on boot the init.d scripts will be run as root (unless manually run)?

My script looks something like the following

PROG="/usr/local/bin/airflow"
PROG_ARGS="worker" 
PID_PATH="/var/run"
PID_NAME="airflow-worker.pid"
AIRFLOW_USER="ubuntu"
LOG_FILE="/var/log/airflow/airflow_worker.log"

source "/home/$AIRFLOW_USER/.profile"

start() {
    start-stop-daemon --start --background --pidfile "$PID_PATH/$PID_NAME" --make-pidfile --user $AIRFLOW_USER --chuid $AIRFLOW_USER --startas /bin/bash -- -c "exec $PROG $PROG_ARGS > $LOG_FILE 2>&1"
}

stop() {
    start-stop-daemon --stop --pidfile "$PID_PATH/$PID_NAME" --retry 10
}

check_user() {
    if [ "$(id -u)" != "0" ]; then
        echo "This script must be run as root" 1>&2
        exit 1
    fi
}

# Other stuff here

case "$1" in
    start)
        check_user
        start
        exit 0
    ;;
    stop)
        check_user
        stop
        exit 0
    ;;
    reload|restart|force-reload)
        stop
        start
        exit 0
    ;;
    status)
        status
        exit $?
    ;;
    **)
        echo "Usage: $0 {start|stop|reload|status}" 1>&2
        exit 1
    ;;
esac

Heres an extract of my users profile too (note the PATH hasn't been changed as /usr/local/bin should already be on the path)

export AIRFLOW_HOME=/home/ubuntu/airflow
export AIRFLOW_DAGS=${AIRFLOW_HOME}/dags

[martingrayson]

This was a PATH issue :) In case anyone experiences this, my path was correctly set in /etc/environment/ but this wasn't picked up at boot hence not being able to find the airflow binary.

I amended my users profile to set the path.

[toryhaavik]

The problem I am running into is similar. The command "airflow webserver" starts gunicorn worker children, and I'm having a hard time getting the stop command to kill all the worker children. My script looks like martin's above, except it starts the webserver rather than the worker. Does anyone know how to kill the entire process group so that the stop command actually stops everything?

EDIT: have since found scripts/systemd/airflow-webserver.service but I think L12 should be KillMode=control-group ?

[cin]

I think I found a solution to daemonize airflow's webserver and scheduler under CentOS. I tried using the scripts provided in scripts/systemd, but I couldn't get the webserver and scheduler to work on the same box for some reason. The child processes for gunicorn weren't dying consistently either. I am using a configuration similar to those above but CentOS doesn't have the start-stop-daemon calls. I am not a bash expert or anything so feel free to make it better.

I tried to make the solution generic so it's broken up into 3 files that live under /etc/init.d

/etc/init.d/daemonize

. /etc/rc.d/init.d/functions

if [ -z ${NAME+x} ]; then
  echo "NAME is not set"
  exit 1001
fi

if [ -z ${APPDIR+x} ]; then
  echo "APPDIR is not set"
  exit 1001
fi

if [ -z ${OWNER+x} ]; then
  echo "OWNER is not set"
  exit 1001
fi

if [ -z ${PIDDIR+x} ]; then
  echo "PIDDIR is not set"
  exit 1001
fi

if [ -z ${PIDFILE+x} ]; then
  echo "PIDFILE is not set"
  exit 1001
fi

if [ -z ${BASECMD+x} ]; then
  echo "BASECMD is not set"
  exit 1001
fi

if [ -z ${CMD+x} ]; then
  echo "CMD is not set"
  exit 1001
fi

if [ ! -d ${APPDIR} ]; then
  echo "${APPDIR} does not exist"
  exit 1001
fi

if [ ! -d "${APPDIR}/logs" ]; then
  (mkdir ${APPDIR}/logs && chown ${OWNER}:${OWNER} ${APPDIR}/logs)
fi

if [ ! -d "${PIDDIR}" ]; then
  mkdir -p ${PIDDIR}
fi

daemonize() {
  case "$1" in
    start)
      start
      ;;
    stop)
      stop
      ;;
    reload|restart)
      $0 stop
      $0 start
      ;;
    status)
      status -p ${PIDFILE} ${NAME}
      exit $?
      ;;
    *)
      echo "Usage: `basename $0` start|stop|status|restart|reload"
      exit 1
  esac
}

make_pid_file() {
  pid=`ps -ef | grep -v grep | grep "${BASECMD}" | tr -s ' ' | cut -d' ' -f2`
  if [ -n "$pid" ]; then
    echo "$pid" > "$PIDFILE"
  fi
}

start() {
  echo -n "Starting $NAME"
  daemon --user="${OWNER}" --pidfile="${PIDFILE}" "${CMD}" || return 2
  make_pid_file
  echo
  return 0
}

stop() {
  echo -n "Stopping $NAME"
  pid=`cat ${PIDFILE}`
  if [ "${USEPKILL}" = true ]; then
    pkill -INT -P ${pid}
  else
    kill -INT ${pid}
  fi
  RETVAL="$?"
  echo
  [ "$RETVAL" = 2 ] && return 2
  rm -f "$PIDFILE"
  return "$RETVAL"
}

/etc/init.d/airflow-webserver

#!/bin/bash

OWNER=hdfs
APPDIR=/var/lib/hdfs/airflow
NAME=airflow-webserver
PIDDIR="/var/run/airflow"
PIDFILE="${PIDDIR}/webserver.pid"
LOGFILE="${APPDIR}/logs/airflow.log"
BASECMD="airflow webserver"
CMD="${BASECMD} > ${LOGFILE} 2>&1 &"
USEPKILL=true

. /etc/rc.d/init.d/daemonize
daemonize "$@"
exit 0

/etc/init.d/airflow-scheduler

#!/bin/bash

OWNER=hdfs
APPDIR=/var/lib/hdfs/airflow
NAME=airflow-scheduler
PIDDIR="/var/run/airflow"
PIDFILE="${PIDDIR}/scheduler.pid"
LOGFILE="${APPDIR}/logs/airflow-scheduler.log"
SCHEDULER_RUNS=5
BASECMD="airflow scheduler"
CMD="${BASECMD} -n ${SCHEDULER_RUNS} > ${LOGFILE} 2>&1 &"
USEPKILL=false

. /etc/rc.d/init.d/daemonize
daemonize "$@"
exit 0