Ensure flask server starts on system boot via upstart

[jeff1evesque]

After successfully building the machine learning repository via vagrant up, the web application is not capable of executing a python script, containing an import statement to another python script, relative to the flask app.py script.

One test approach, is to create a simple python script on the host machine, relative to the flask app.py. This script would contain import sys, a standard python library on the first line. On the following line, it would print 'hello world'. After the file is created, we would make sure that vagrant mounts this file from our host machine (windows), to the guest Ubuntu VM:

$ vagrant halt
$ vagrant up

Then, we execute this python script, python /vagrant/test.py from the guest Ubuntu VM. If this script doesn't run, we vagrant halt, and implement dos2unix to convert the line endings of this file, before executing this file again (after vagrant up).

If the above two cases do not work, we may consider the following:

• check validity of upstart logic within start_webserver.pp, between script stanza, and the exec command
  • we know that a process ID corresponding to the flask upstart script is present on vagrant up
• test case: adjust above tests by importing another trivial python script instead of import sys

However, if the base case test.py works, we need to check the following additional cases:

• change import sys to import test_directory.test2 in test.py. Then, create the following:
  • /test_directory/test2.py
    • contains two lines of code: import sys, followed by print 'hello from test2.py'
  • /test_directory/__init__.py
    • can be as trivial as an empty file
• check integrity of all __init__.py, and determine if the cwd is relative to the flask server, app.py.
• check validity of upstart logic within start_webserver.pp, between script stanza, and the exec command
  • we know that a process ID corresponding to the flask upstart script is present on vagrant up

[jeff1evesque]

We created test.py on the host (windows 7) machine, with the suggested lines of code:

import sys
print 'hello world'

Then, we executed the script, on our guest (ubuntu 14.04) machine:

vagrant@vagrant-ubuntu-trusty-64:/vagrant$ ls -l
total 47
-rwxrwxrwx 1 vagrant vagrant   264 Aug  8 12:35 app.py
drwxrwxrwx 1 vagrant vagrant  4096 Aug  8 12:46 brain
drwxrwxrwx 1 vagrant vagrant     0 Aug  8 12:41 build
-rwxrwxrwx 1 vagrant vagrant   138 Aug  8 12:35 __init__.py
-rwxrwxrwx 1 vagrant vagrant  1580 Aug  8 12:35 license.md
drwxrwxrwx 1 vagrant vagrant  4096 Aug  8 12:46 log
drwxrwxrwx 1 vagrant vagrant  4096 Aug  8 12:36 puppet
-rwxrwxrwx 1 vagrant vagrant 19976 Aug  8 12:35 README.md
-rwxrwxrwx 1 vagrant vagrant   287 Aug  8 12:35 settings.py
-rwxrwxrwx 1 vagrant vagrant   243 Aug  8 12:46 settings.pyc
drwxrwxrwx 1 vagrant vagrant     0 Aug  8 12:35 src
-rwxrwxrwx 1 vagrant vagrant    31 Aug  9 22:01 test.py
-rwxrwxrwx 1 vagrant vagrant  5754 Aug  8 12:35 Vagrantfile
drwxrwxrwx 1 vagrant vagrant  4096 Aug  8 12:46 web_interface
vagrant@vagrant-ubuntu-trusty-64:/vagrant$ python test.py
hello world

Based on our initial strategy, we will consider the following approaches:

• change import sys to import test_directory.test2 in test.py. Then, create the following:
  • /test_directory/test2.py
    • contains two lines of code: import sys, followed by print 'hello from test2.py'
  • /test_directory/__init__.py
    • can be as trivial as an empty file
• check validity of upstart logic within start_webserver.pp, between script stanza, and the exec command
  • we know that a process ID corresponding to the flask upstart script is present on vagrant up

[jeff1evesque]

We proceeded with the above strategy, by creating test scripts on the host windows 7 machine:

• change import sys to import test_directory.test2 in test.py. Then, create the following:
  • /test_directory/test2.py
    • contains two lines of code: import sys, followed by print 'hello from test2.py'
  • /test_directory/__init__.py
    • can be as trivial as an empty file

Then, we proceeded by running the test script, test.py from the vagrant guest machine:

vagrant@vagrant-ubuntu-trusty-64:~$ cd /vagrant
vagrant@vagrant-ubuntu-trusty-64:/vagrant$ ls
app.py  __init__.py  puppet       settings.pyc    test.py
brain   license.md   README.md    src             Vagrantfile
build   log          settings.py  test_directory  web_interface
vagrant@vagrant-ubuntu-trusty-64:/vagrant$ python test.py
hello from test2.py
hello world

Therefore, we know that we can import other python scripts across directories. Specifically, python scripts created on Windows 7 machines, later mounted on Linux machines, will not have a problem with conflicting line endings during execution.

We will investigate the following case:

• check integrity of all __init__.py, and determine if the cwd is relative to the flask server, app.py.

[jeff1evesque]

We noticed on initial build (first time of vagrant up), we are able to access http://localhost:8080/. However, when attempting to store a dataset on a Data new session, nothing was stored into our MariaDB database. However, if we vagrant halt, and restart with vagrant up, we notice that http://localhost:8080/ does not automatically start on the host. Therefore, we need to vagrant ssh into our vagrant guest, and manually run the following:

vagrant@vagrant-ubuntu-trusty-64:~$ cd /vagrant
vagrant@vagrant-ubuntu-trusty-64:/vagrant$ ls
app.py  __init__.py  puppet       settings.pyc    test.py
brain   license.md   README.md    src             Vagrantfile
build   log          settings.py  test_directory  web_interface
vagrant@vagrant-ubuntu-trusty-64:/vagrant$ python app.py

Then, we notice that http://localhost:8080/ is now available on the host machine. However, proceeding with the Data New session, we get the following errors when attempting to store data into our MariaDB database:

vagrant@vagrant-ubuntu-trusty-64:/vagrant$ python app.py
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
 * Restarting with stat
10.0.2.2 - - [12/Aug/2015 08:41:04] "GET / HTTP/1.1" 200 -
10.0.2.2 - - [12/Aug/2015 08:41:04] "GET /static/js/jquery-2.1.1.min.js HTTP/1.1" 304 -
10.0.2.2 - - [12/Aug/2015 08:41:04] "GET /static/js/jquery-validate.min.js HTTP/1.1" 304 -
10.0.2.2 - - [12/Aug/2015 08:41:04] "GET /static/js/html_form.min.js HTTP/1.1" 304 -
10.0.2.2 - - [12/Aug/2015 08:41:04] "GET /static/js/ajax_graphic.min.js HTTP/1.1" 304 -
10.0.2.2 - - [12/Aug/2015 08:41:04] "GET /static/js/form_validator.min.js HTTP/1.1" 304 -
10.0.2.2 - - [12/Aug/2015 08:41:04] "GET /static/js/html_form_delegator.min.js HTTP/1.1" 304 -
10.0.2.2 - - [12/Aug/2015 08:41:04] "GET /static/js/ajax_data.min.js HTTP/1.1" 304 -
10.0.2.2 - - [12/Aug/2015 08:41:04] "GET /static/js/ajax_session.min.js HTTP/1.1" 304 -
10.0.2.2 - - [12/Aug/2015 08:41:04] "GET /static/js/ajax_model.min.js HTTP/1.1"304 -
10.0.2.2 - - [12/Aug/2015 08:41:04] "GET /static/js/ajax_feature.min.js HTTP/1.1" 304 -
10.0.2.2 - - [12/Aug/2015 08:41:04] "GET /static/css/style.min.css HTTP/1.1" 304
 -
10.0.2.2 - - [12/Aug/2015 08:41:42] "POST /load-data/ HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/vagrant/web_interface/views.py", line 40, in load_data
    if session_type == 'data_new': response = loader.load_data_new()
  File "/vagrant/brain/load_data.py", line 33, in load_data_new
    session_entity = session.save_svm_entity('data_new')
  File "/vagrant/brain/session/base_data.py", line 72, in save_svm_entity
    db_return  = db_save.save()
  File "/vagrant/brain/database/save_entity.py", line 49, in save
    response      = self.sql.sql_command(sql_statement, 'insert', args)
  File "/vagrant/brain/database/db_query.py", line 56, in sql_command
    if sql_type in ['insert', 'delete', 'update']: return {'status': False, 'error': self.list_error, 'id': self.cursor.lastrowid}
AttributeError: 'SQL' object has no attribute 'cursor'
10.0.2.2 - - [12/Aug/2015 08:41:42] "GET /static/img/loading.gif HTTP/1.1" 304 -

Therefore, our problem is two parts:

• our upstart script, flask.conf is not properly executing
• our mysql-python binding is not properly executing

[jeff1evesque]

We will assume the following case is a result of #1962 not being complete:

• our mysql-python binding is not properly executing

[jeff1evesque]

We have resolved the following case from #1962:

• our mysql-python binding is not properly executing

This means, if we manually execute the following on the guest VM, at any point in time (even on initial build), the application will work on the host machine:

cd /vagrant
python app.py

However, our upstart script, flask.conf is still not properly executing. This means, that our flask server does not automatically start up when our guest VM is booted up.

[jeff1evesque]

We will assume the following case is a result of the #1997 issue being reopened:

• our upstart script, flask.conf is not properly executing

[jeff1evesque]

With #2046 merged, we need to run sudo service flask stop (if it hangs ctrl-c), on the vagrant VM machine, then manually run python /vagrant/app.py in order for the web application to successfully port forward on localhost:8080 on the host machine.

[jeff1evesque]

In order to ensure our /vagrant/app.py flask server runs via the upstart script, generated by /puppet/manifests/start_webserver.pp, we will research one of the following implementations:

• apache2 + mod_wsgi
• apache2 + nginx
• nginx + uwsgi

Otherwise, we will have to resort to manually vagrant ssh, then python /vagrant/app.py. However, we will take ngnix + uwsgi, as a first approach.

[jeff1evesque]

We have closed #1997, and have decided to resolve the remaining flask-upstart related bug, within this overall issue.

[jeff1evesque]

The following provides a generalization of how to implement uwsgi, and nginx with respect to a flask script:

• http://flask.pocoo.org/docs/0.10/deploying/uwsgi/

[jeff1evesque]

The following is a error traceback when trying to build the vagrant VM:

...
==> default: Processing triggers for ureadahead (0.100.0-16) ...
==> default: Running provisioner: puppet...
==> default: Running Puppet with install_packages.pp...
==> default: stdin: is not a tty
==> default: Warning: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations
==> default:    (at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1139:in `issue_deprecation_warning')
==> default: Warning: Config file /etc/puppet/hiera.yaml not found, using Hiera defaults
==> default: Notice: Compiled catalog for vagrant-ubuntu-trusty-64.home in environment production in 2.34 seconds
==> default: Notice: /Stage[main]/Main/Exec[enable-multiverse-repository-2]/returns: executed successfully
==> default: Notice: /Stage[main]/Main/Package[redis-server]/ensure: ensure changed 'purged' to 'present'
==> default: Notice: /Stage[main]/Main/Exec[enable-multiverse-repository-1]/returns: executed successfully
==> default: Notice: /Stage[main]/Python/Package[python-setuptools]/ensure: ensure changed 'purged' to 'present'
==> default: Notice: /Stage[main]/Python/Package[python-pip]/ensure: ensure changed 'purged' to 'present'
==> default: Notice: /Stage[main]/Apt/Apt::Setting[conf-update-stamp]/File[/etc/apt/apt.conf.d/15update-stamp]/content: content changed '{md5}b9de0ac9e2c9854b1bb213e362dc4e41' to '{md5}12bcd8b4e91e9d715290ae16ba3c85fb'
==> default: Notice: /Stage[main]/Apt/File[preferences]/ensure: created
==> default: Notice: /Stage[main]/Main/Exec[build-package-dependencies-1]: Triggered 'refresh' from 2 events
==> default: Notice: /Stage[main]/Main/Package[dos2unix]/ensure: ensure changed 'purged' to 'present'
==> default: Notice: /Stage[main]/Main/Package[ruby-dev]/ensure: ensure changed 'purged' to 'present'
==> default: Notice: /Stage[main]/Main/Package[inotify-tools]/ensure: ensure changed 'purged' to 'present'
==> default: Notice: /Stage[main]/Main/Package[xmltodict]/ensure: created
==> default: Notice: /Stage[main]/Main/Package[redis]/ensure: created
==> default: Notice: /Stage[main]/Main/Package[jsonschema]/ensure: created
==> default: Notice: /Stage[main]/Nodejs::Repo::Nodesource::Apt/Apt::Source[nodesource]/Apt::Key[Add key: 9FD3B784BC1C6FC31A8A0A1C1655A0AB68576280 from Apt::Source nodesource]/Apt_key[Add key: 9FD3B784BC1C6FC31A8A0A1C1655A0AB68576280 from Apt::Source nodesource]/ensure: created
==> default: Notice: /Stage[main]/Nodejs::Repo::Nodesource::Apt/Apt::Source[nodesource]/Apt::Setting[list-nodesource]/File[/etc/apt/sources.list.d/nodesource.list]/ensure: created
==> default: Notice: /Stage[main]/Apt::Update/Exec[apt_update]: Triggered 'refresh' from 1 events
==> default: Notice: /Stage[main]/Nodejs::Install/Package[nodejs]/ensure: ensure changed 'purged' to 'present'
==> default: Notice: /Stage[main]/Main/Package[uglify-js]/ensure: created
==> default: Notice: /Stage[main]/Main/Package[node-sass]/ensure: created
==> default: Notice: /Stage[main]/Main/Package[imagemin]/ensure: created
==> default: Notice: Finished catalog run in 371.55 seconds
==> default: Running provisioner: puppet...
==> default: Running Puppet with install_sklearn.pp...
==> default: stdin: is not a tty
==> default: Warning: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations
==> default:    (at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1139:in `issue_deprecation_warning')
==> default: Warning: Config file /etc/puppet/hiera.yaml not found, using Hiera defaults
==> default: Notice: Compiled catalog for vagrant-ubuntu-trusty-64.home in environment production in 0.89 seconds
==> default: Notice: /Stage[main]/Git/Package[git]/ensure: ensure changed 'purged' to 'present'
==> default: Notice: Finished catalog run in 18.06 seconds
==> default: Running provisioner: puppet...
==> default: Running Puppet with vagrant_mounted.pp...
==> default: stdin: is not a tty
==> default: Warning: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations
==> default:    (at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1139:in `issue_deprecation_warning')
==> default: Notice: Compiled catalog for vagrant-ubuntu-trusty-64.home in environment production in 0.83 seconds
==> default: Notice: /Stage[main]/Main/File[vagrant-startup-script]/ensure: created
==> default: Notice: /Stage[main]/Main/Exec[dos2unix-upstart-vagrant]/returns: executed successfully
==> default: Notice: /Stage[main]/Main/Exec[dos2unix-upstart-vagrant]: Triggered 'refresh' from 1 events
==> default: Notice: /Stage[main]/Main/Service[workaround-vagrant-bug-6074]/ensure: ensure changed 'stopped' to 'running'
==> default: Notice: Finished catalog run in 0.08 seconds
==> default: Running provisioner: puppet...
==> default: Running Puppet with start_webserver.pp...
==> default: stdin: is not a tty
==> default: Warning: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations
==> default:    (at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1139:in `issue_deprecation_warning')
==> default: Error: Could not parse for environment production: Syntax error at '::app' at /tmp/vagrant-puppet/manifests-846018e2aa141a5eb79a64b4015fc5f3/start_webserver.pp:12:16 on node vagrant-ubuntu-trusty-64.home
==> default: Error: Could not parse for environment production: Syntax error at '::app' at /tmp/vagrant-puppet/manifests-846018e2aa141a5eb79a64b4015fc5f3/start_webserver.pp:12:16 on node vagrant-ubuntu-trusty-64.home
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.

[jeff1evesque]

We are now getting the following traceback error:

...
==> default: Running provisioner: puppet...
==> default: Running Puppet with start_webserver.pp...
==> default: stdin: is not a tty
==> default: Warning: Setting templatedir is deprecated. See http://links.puppet
labs.com/env-settings-deprecations
==> default:    (at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1139:in `issue_
deprecation_warning')
==> default: Error: Could not parse for environment production: Syntax error at
'webserver' at /tmp/vagrant-puppet/manifests-846018e2aa141a5eb79a64b4015fc5f3/st
art_webserver.pp:13:5 on node vagrant-ubuntu-trusty-64.home
==> default: Error: Could not parse for environment production: Syntax error at
'webserver' at /tmp/vagrant-puppet/manifests-846018e2aa141a5eb79a64b4015fc5f3/st
art_webserver.pp:13:5 on node vagrant-ubuntu-trusty-64.home
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.

[jeff1evesque]

We are now receiving the following traceback error:

...
==> default: Running provisioner: puppet...
==> default: Running Puppet with start_webserver.pp...
==> default: stdin: is not a tty
==> default: Warning: Setting templatedir is deprecated. See http://links.puppetlabs.com/env-settings-deprecations
==> default:    (at /usr/lib/ruby/vendor_ruby/puppet/settings.rb:1139:in `issue_deprecation_warning')
==> default: Warning: Config file /etc/puppet/hiera.yaml not found, using Hiera defaults
==> default: Error: Invalid parameter ensure on Class[Uwsgi::App] at /tmp/vagrant-puppet/manifests-846018e2aa141a5eb79a64b4015fc5f3/start_webserver.pp:12 on node vagrant-ubuntu-trusty-64.home
==> default: Error: Invalid parameter ensure on Class[Uwsgi::App] at /tmp/vagrant-puppet/manifests-846018e2aa141a5eb79a64b4015fc5f3/start_webserver.pp:12 on node vagrant-ubuntu-trusty-64.home
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.

[jeff1evesque]

We've tried to ensure that the following lines was present, as well as absent (without the implementation of uswgi, and nginx):

## run upstart job as a background process
expect fork

in start_webserver.pp. However, we received the same errors as shown above.

[jeff1evesque]

We can simply change the following lines in start_webserver.pp, without the implementation of uwsgi, or nginx:

...
## start upstart job
exec python '/vagrant/app.py'
...

[jeff1evesque]

5e0165cj, 05992b2: does not work on the initial build, but, on successive vagrant up. So, we need to determine how to modify start_webserver.pp, such that the flask app.py runs on initial vagrant up build (not just successive vagrant up).