Create flask upstart script in puppet manifest

[jeff1evesque]

We need to create start_webserver.pp. This script will be responsible for installing flask, and defining the necessary upstart script, to ensure that our flask server is running each time the Ubuntu Server has started.

The following need to be removed (if present):

• flask in $packages_flask_pip = ['flask', 'requests'] from install_packages.pp

[jeff1evesque]

After #2001 has been resolved, we will move corresponding python package installation from install_packages.pp to start_webserver.pp:

• flask
• requests

Then, we can safely remove python-pip package from install_packages.pp. However, we need to add include python at the top of install_packages.pp. Also, we will need to add corresponding python include statements at the top of start_webserver.pp.

[jeff1evesque]

We need to create the upstart script, /etc/init/start_flask.conf:

#!upstart
description 'start flask server'

## start job defined in this file after system services, and processes have already loaded
#       (to prevent conflict).
#
#  @filesystem, ensure job in this file executes after filesystems have been mounted
#  @[2345], represents all configuration states with general linux, and networking access
start on filesystem or runlevel [2345]

## stop flask server when machine gracefully shuts down
stop on shutdown

## start flask server (via bash shell)
exec python /vagrant/app.py

This will allow us the run the following command:

sudo service <servicename> <control>

with the following <command> options:

• restart: this will stop, then start a service
• start: this will start a service, if it's not running
• stop: this will stop a service, if it's running
• status: this will display the status of a service

Note: according to the Ubuntu upstart documentation, the following can ensure that the job defined in start_flask.conf starts each time the machine boots up:

• filesystem: event signalling that filesystems have been mounted
• local-filesystems: event signalling that local filesystems have been
  mounted

[jeff1evesque]

4f4f564: we forgot that redhat based distros may have a different format for upstart scripts.

[jeff1evesque]

We will make a minor change to the docblock.

[jeff1evesque]

We need to fix a minor language typo.

[jeff1evesque]

Although, our current upstart script runs manually (and on first system boot, via vagrant up), it does not automatically start on successive system start-up (i.e. vagrant up).

[jeff1evesque]

aae5e9b: since our upstart script should have the ability to stop, as well as start the corresponding service, we've renamed start_flask.conf to flask.conf.

[jeff1evesque]

Since our custom logs defined in flask.conf are not executed (because the service doesn't work on boot up), we need to check /var/log/upstart/flask.log, and compare the logs for the following cases (in sequence):

• initial vagrant up
• successive vagrant up (second instance suffices)
• manual sudo reboot
• manual sudo service flask start
• manual sudo reboot

[jeff1evesque]

After initial vagrant up, we have a custom flask_server.pid that has one line containing the process id of the flask service (as defined by our flask.conf).

For example:

19534

Also, after initial vagrant up we have a custom flask_server.log that generates a log of times when our flask service was started (as defined by our flask.conf).

For example:

[Thu Jul  9 02:32:51 UTC 2015] flask server starting

However, if we check our upstart logs in /var/log/upstart we don't find a corresponding flask.log:

vagrant@vagrant-ubuntu-trusty-64:/var/log/upstart$ ls
console-setup.log     network-interface-eth0.log      rsyslog.log
container-detect.log  pollinate.log                   statd.log
cryptdisks.log        procps-static-network-up.log    systemd-logind.log
gssd.log              procps-virtual-filesystems.log  ttyS0.log

This means, our flask service did not generate a default log. However, we see our flask service exists, but not running after initial vagrant up:

vagrant@vagrant-ubuntu-trusty-64:/var/log/upstart$ initctl list | grep flask
flask stop/waiting

[jeff1evesque]

On the successive vagrant halt, followed by vagrant up, we have the /var/log/upstart/flask.log generated, with the following content:

/bin/sh: 1: /bin/sh: cannot create /vagrant/log/flask_server.log: Directory nonexistent
/bin/sh: 1: /bin/sh: cannot create /vagrant/log/flask_server.log: Directory nonexistent

This means our flask.conf (upstart script) is attempting to create corresponding log files, into a directory that doesn't exists. Therefore, we need to create the /vagrant/log/ directory within the upstart script (flask.conf), and not in our puppet manifest, start_webserver.pp.

Note: we will assume for now, the other three above cases:

• manual sudo reboot
• manual sudo service flask start
• manual sudo reboot

will yield the same /var/log/upstart/flask.log.

[jeff1evesque]

We need to implement the following to start_webserver.pp:

...
pre-start script
    [ "$MOUNTPOINT" == "/vagrant" ] || stop
end script
...

If that is not successful, we can implement the following in our Vagrantfile:

...
config.vm.provision :shell, :inline => "sudo initctl emit vagrant-ready"
...

followed by the following snippet within start_webserver.pp:

...
start on vagrant-ready

## stop flask server when machine gracefully shuts down
...

[jeff1evesque]

e67c5d0: our upstart script, /etc/init/flask.conf properly starts with initial vagrant up, as displayed in /var/log/upstart/flask.log:

*Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
* Restarting with stat

On successive vagrant halt, followed by vagrant up, we have the following in /var/log/upstart/flask.log:

>/proc/self/fd/9: 2: /proc/self/fd9: cannot create /vagrant/log/flask_server.log: Protocol error
*Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
* Restarting with stat

So, we need to attempt, and research the following:

• run our flask app.py as a background service in our upstart script, flask.conf
  • this will allow successive commands in the corresponding stanza to execute (if syntax is correct)
• determine why we can't create flask_server.log in our script stanza
• determine the logic components of the script stanza:
  • is an exec statement, with corresponding start (on the same line) required
• ensure service <service_name> stop, and service <service_name> restart works

[jeff1evesque]

80001ca: after numerous vagrant halt, followed by vagrant up, we notice that our /etc/init/flask.conf properly creates a background service:

vagrant@vagrant-ubuntu-trusty-64:~$ ps aux | grep python
root      1818  3.2  9.0 316912 45328 ?        Ss   09:16   0:00 python /vagrant
/app.py
root      1877  6.0  9.1 391032 45864 ?        Sl   09:16   0:01 /usr/bin/python
 /vagrant/app.py
vagrant   2123  0.0  0.1  10460   936 pts/1    S+   09:16   0:00 grep --color=au
to python

Note: since the system has been rebooted more than once, we know that previous service processes, associated with flask.conf, is properly terminated. The only process that exists with respect to our flask.conf is the current running upstart process.

[jeff1evesque]

If we clear the contents of /var/log/upstart/flask.conf, and proceed with vagrant halt, then vagrant up, we get the following within /var/log/upstart/flask.conf:

/bin/sh: 1: /bin/sh: cannot create /vagrant/log/flask_server.log
: Protocol error
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat

This means our post-stop script stanza is unable to log when our upstart service terminates.

Note: changing the post-stop script stanza, to pre-stop script has the same outcome.

[jeff1evesque]

We need to define the following:

• setuid
• setgid

Otherwise, we get the following within /var/log/upstart/flask.log during vagrant halt:

/proc/self/fd/9: 2: /proc/self/fd/9: /vagrant: permission denied
Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
  File "/usr/lib/python2.7/threading.py", line 763, in run
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 603 in inner
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 462 in serve_forever
  File "/usr/lib/python2.7/SocketServer.py", line 241, in serve_forever
  File "/usr/lib/python2.7/threading.py", line 585, in set
  File "/usr/lib/python2.7/threading.py", line 406, in notifyAll
<type 'exceptions.TypeError'>: 'NoneType' object is not callable
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat

[jeff1evesque]

f8036c2, 2d93aed: after numerous vagrant halt, followed by vagrant up we get the following within /var/log/upstart/flask.log:

 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat

Similarly, our /vagrant/log/flask_server.log contains the following:

[Sat Jul 11 17:47:19 UTC 2015] flask server starting
[Sat Jul 11 17:52:11 UTC 2015] flask server stopping
[Sat Jul 11 17:54:00 UTC 2015] flask server starting
[Sat Jul 11 17:54:46 UTC 2015] flask server stopping
[Sat Jul 11 17:56:42 UTC 2015] flask server starting

[jeff1evesque]

After vagrant destroy, vagrant up, vagrant halt, and vagrant up we notice the following error:

 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
/proc/self/fd/9: 2: [: =: unexpected operator

The reason for this, is because we are running a windows 7 host, and when setup_webserver.pp creates /etc/init/flask.conf (on the guest VM), it does not convert windows line endings to linux line endings (on the guest VM). If we implement the following within the Ubuntu virtual machine, our problems are eliminated:

sudo apt-get install dos2unix
sudo dos2unix /etc/init/flask.conf

Note: the dos2unix package exists in the yum repository, as well as apt-get.

[jeff1evesque]

We need to remove an unnecessary notify => Exec['dos2unix-line-endings'],.

[jeff1evesque]

We need to ensure that our pre-stop script stanza properly notifies the series of chained puppet commands. Otherwise, only after the second (and successive) series of vagrant up, followed by vagrant halt implementation, will correctly implement the pre-stop script stanza, which means the first vagrant up, followed by vagrant halt, will contain the following error within /var/log/upstart/flask.log:

 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
/proc/self/fd/9: 2: [: =: unexpected operator

One solution is the implement the following within start_webserver.pp:

...
                       ## log shut-down date, remove process id from log before '/vagrant' is unmounted
                       #
                       #  @[`date`], current date script executed
                       pre-stop script
                           echo "[`date`] flask server stopping" >> /vagrant/log/flask_server.log
                       end script
                       | EOT
            notify  => Exec['dos2unix-line-endings'],
        }

        ## convert clrf (windows to linux) in case host machine is windows.
        exec {'dos2unix-line-endings':
            command => 'dos2unix /etc/init/flask.conf',
            refreshonly => true,
            notify => Service['flask'],
        }

        ## start webserver
        service {'flask':
            ensure => 'running',
            enable => 'true',
        }
    }
    default: {
    }
}

[jeff1evesque]

We need to determine if the following from start_webserver.pp has some syntax error:

...
            notify => Service['flask'],
        }

        ## start webserver
        service {'flask':
            ensure => 'running',
            enable => 'true',
        }
...

The motivation for the above snippet, is because our generated /etc/init/flask.conf does not automatically run on initial build, nor successive vagrant up. However, if we manually vagrant up, then run sudo service flask start, our flask server, app.py is guaranteed to be running.

[jeff1evesque]

We need to adjust our flask upstart script by implementing another workaround upstart script in order to ensure the vagrant-mounted event.

[jeff1evesque]

The workaround has been implemented in #2046. However, we will close this issue, since the creation of the upstart script has been resolved, and handle all other bugs, within their own corresponding issue.