Move towards virtualenv

[lucasvickers]

Mycodo Issue Report:

• All versions

Problem Description

Currently you install python versions via the global python and pip. You should move towards installing a virtualenv in your install dir and running the daemon with an isolated python instance.

[kizniche]

Yes! I've been wanting to do this for a while, but lacked the knowhow to do it properly.

[lucasvickers]

It's pretty straight forward.

• Install pip for any version of python you want to use.
• Install virtualenv using pip install virtualenv
  • If you have multiple python versions you'll get the likes of pip-2.7, pip-3.5, etc
• You use virtualenv to create an isolate env using this
  • virtualenv env --no-site-packages

If you are using it from command line you can activate your env by doing
activate env/bin/activate
and deactivate using
deactivate env/bin/activate

To install stuff into your env via pip, use
env/bin/pip install -r requirements.txt
If you want to then start up a process, instead of calling
python ./script.py you would call env/bin/python ./script.py and it will get called from within your env.

You can also have a virtualenv per python version so you can choose what version of python you want to run it with.

[kizniche]

Well, I get that part, and have done that from the command line. I was referring more to how I would integrate into being used by Mycodo.

[lucasvickers]

:) (sorry for stating the obvious)

When the daemon launches, the python script gets called. You could just call a wrapper that says ~/Mycodo/env/bin/python ~/Mycodo/mycodo/mycodo_daemon.py

Then it should run out of your env. The only trick would be if you manually kick off other python scripts from within your script, i.e. something like os.process('python somescript.py'), then you'd just need to wrap it as well so you called ~/Mycodo/env/bin/python somescript.py

Am I helping at all?

[kizniche]

That helps, but I'm still at a loss of how to start and how involved it will be to get the frontend and backend both running in a virtualenv.

[lucasvickers]

So let's pretend you use your setup script to create a folder ~/Mycodo/env that has the python env, and you use ~/Mycodo/env/bin/pip to install all your stuff in there.

For any python call, you need to call the script with ~/Mycodo/env/bin/python script.py. This should be pretty straight forward for a daemon process. Just googling I see a lot of stuff.

As for your frontend, you need to install and run uwsgi from within your python env. Take a look at this tutorial for setting up nginx:
https://www.digitalocean.com/community/tutorials/how-to-serve-flask-applications-with-uwsgi-and-nginx-on-ubuntu-14-04

So on a system I have that deals w/ sms messages, if I check my service I see this, meaning it runs from my virtuale env.

$ systemctl status sms.service
● sms.service - uWSGI instance
...
   CGroup: /system.slice/sms.service
           ├─2393 /home/.../flask/env/bin/uwsgi --ini uwsgi.ini --py-autoreload 2

[kizniche]

That seems pretty easy. I've also wanted to experiment with nginx performance. I'll check it out once I get this new install going.

[kizniche]

Here's my progress so far:

• Created virtualenv (needs --system-site-packages for opencv from apt, pyopencv won't install through pip)
• Experimented with nginx in virtualenv (w/ uwsgi + ssl) but gave up after some difficulty. I'll stick with apache2 until figuring out getting everything in the virtualenv has been completed.
• Got upgrade_commands.sh script setting up virtualenv, activating, and installing pip packages from requirements.txt
• Got apache2 to host the flask app in the virtualenv
• Currently working on getting the mycodo daemon to run in the virtualenv

[kizniche]

I think I have everything working with the virtual environment, from setup to updating. I'll run a fresh install sometime in the next few days to test it. In the System info page, I added to the system checks at the top to indicate whether the frontend and backend are in a virtualenv, just to make sure it's working as I test.

[lucasvickers]

very cool!

Yeah wouldn't worry much about the NGINX stuff, you're likely just stuck on a configuration issue, and to be honest the type of hosting mycodo is doing can be run on any webserver. I prefer NGINX cuz I find it to be a lighter weight deploy.

[kizniche]

It's all running great right now. I'm going to go for a long uptime on the new Pi3 I installed dev-5.0 on.

I wasn't able to get through the errors with the nginx config. It started but the app would crash at the first request. I believe it was a config/database issue. I'll revisit it in the future. If you have experience with nginx and would like to see if you can get a stable config, please do ;)

[kizniche]

Just remembered the last issue I was having with the nginx config was SSL, because the flask app forces SSL by default.