Selective dependency installation

[kizniche]

v5.6.0 proposal

As Mycodo grows in device support, so too does the number of dependencies that need to be met during the install process. To prepare for future expansion and save users from installing components they will never use, the Mycodo install should only include those packages required to run the frontend (web UI) and backend (daemon) without any devices added. Also, by selectively installing Mycodo components for only what features are used, there will be a smaller/faster base install.

Upon adding a device that requires a Python package that's not installed, the user is prompted to install the dependency. If the user says yes, the package is installed and the device is added. If the user says no, the user is taken back to the previous page without adding the device.

Users should also be prompted during the install whether they would like to perform a minimal, custom, or full install. Custom would allow the user to select individual packages to install. This would be useful if the Pi will not have internet access at a future time when a new device is desired to be added.

Steps

• Make Python modules conditionally imported (controller_input.py)
• Add dependencies to devices in config.py
• Build a frontend system to check for dependency issues and prompt the user to install the requirements
• Implement a dependency installation system
• Implement the dependency install system from the web UI
• Implement an install process with the new dependency system
• Implement an upgrade process with the new dependency system
• Test everything

[kizniche]

First iteration of the intervention page when an Input is attempted to be installed when there's an unmet dependency.

[kizniche]

I finally got a dependency install system working in the web UI.

[kizniche]

@zsole2 Will you be willing to test the new minimal install feature on your Pi Zero and provide feedback about improving the system? I have everything except the install/upgrade scripts worked out for this new system.

[kizniche]

It seems to be working really well in the web UI now.

[zsole2]

Of course, I will do the testing... this is a feature that I am looking forward to use all the tome on my Pi Zeros.
I cannot promise to be as fast as you were to implement it, though, since I'm still quite sick, and even sitting up can hurt. But I'll check on my test-Pi (for the memory leak issue) and then after it will be free to play with.
So tell me what to do, and I will try my best.

[kizniche]

Sorry to hear. Get well.

There's no hurry at all. I've just been on a roll with this, which is why there's so many recent commits. I just got the interactive install system working. I'm going to test a new install very soon.

[kizniche]

So far so good...

[kizniche]

Here are some initial install times on a Pi 2 using the new dependency system:

Minimal Install (required dependencies)

Total install time: 6 minutes and 27 seconds

Custom Install (required dependencies + quick2wire + 1wthermsensor)

Total install time: 6 minutes and 51 seconds

Full Install (required dependencies + all optional dependencies)

Total install time: 9 minutes and 32 seconds

The difference between a Full and Minimum install was 3 min. 5 sec. (32% less time).

Edit: Updated times to be more accurate

[zsole2]

Did a Minimum install, it took 37m 52s. Then I checked two older log files, one of them took about 72 mins, the other 78 mins. Great improvement! I'll try to add some sensors and see that part.

[zsole2]

Another fresh minimum install also finished in the 38 mins range. Adding the DS18 sensor went flawlessly, too. A few issues came to my mind:

• if pigpiod is an optional component, the sampling frequency selection should be checked accordingly
• adding w1thermsensor as dependency is not enough, 1-wire interface should be enabled, and a restart is also required
• how backup/export/import will be affected?
• why Mycodo requires wiringpi? Originally I thought that it was needed for some wireless controllers, but the new requirement rpi-rf only depends on Rpi.GPIO. Only other thing I could find is the gpio readall call for the System Information. If I have to select between this fancy readout and the 7 minutes it takes to compile it on the PiZero, it is an easy choice.
  I've seen somewhere that wiringpi is available through apt-get, but could not find much on this except that the author suggests to remove it. Or there is wiringpi2, but the references seem quite ancient.

As an even better news, it looks that the total resource usage is much much better now, still checking it, but on the sous-vide controller with two sensors, one output and one PID controller, the processor usage is 5-6% and the memory usage is 5%.

[kizniche]

if pigpiod is an optional component, the sampling frequency selection should be checked accordingly

I'll get to this soon.

adding w1thermsensor as dependency is not enough, 1-wire interface should be enabled, and a restart is also required

I can also put a check in, when the user attempts to add a 1-wire device, to see if the 1-wire interface has been enabled, and if not, redirect the user to the Pi Config page to enable it. In that same line, I can have all interfaces checked and the user redirected to the Pi Config page if a particular interface hasn't been enabled. I should also add an option to disable that checking (for power users) in case they enabled an interface on a non-default location (not raspi-config), so I don't lock them out from adding Inputs if they choose not to use the default interface provided by raspi-config.

how backup/export/import will be affected?

It operates the same as before.

why Mycodo requires wiringpi?

Good point. It appears gpio readall is all that uses this. I'll add it to the optional install. If it's not installed, that part of the System Information page will simply not appear.

it looks that the total resource usage is much much better now

Good to hear, though I have no idea why, lol

[zsole2]

I did a third install, on an older smaller SD card, it took 53 mins.
As for the performance improvement, I must add that the above numbers were only for influxd, but this was the man bottleneck on the Zero.
The improvement seems quite real, I'm just running the sous-vide controller with 5 sec sampling of the temperature and 10 sec on the PID period, and the and the CPU is around 50% idle.

[zsole2]

What I meant for the export/import is that what happens if someone loads the config to another Pi where the dependencies were not properly set up. The mycodo database will have the device(s) but their dependencies may be missing.

[kizniche]

I should add a function to scan the new database and check for unmet dependencies. As of now, the user will have to ensure the dependencies are met.

[kizniche]

I have a feeling removing libav-tools and libffi-dev from the installation will also speed it up: b36e6d6

[zsole2]

I did an upgrade from v5.6.0 to Master, it went smooth and fast, about 15 mins on the Zero (it used to be around 40 mins).
I had no better thing to do than watching the upgrade, and noticed a few things. First, the segment

[2018-02-28 08:33:47] #### Creating mycodo user
[2018-02-28 08:33:48] The user `mycodo' is already a member of `adm'.
[2018-02-28 08:33:49] The user `mycodo' is already a member of `dialout'.
[2018-02-28 08:33:49] The user `mycodo' is already a member of `gpio'.
[2018-02-28 08:33:51] The user `mycodo' is already a member of `i2c'.
[2018-02-28 08:33:53] The user `mycodo' is already a member of `video'.
[2018-02-28 08:33:53] 
[2018-02-28 08:33:53] #### Compiling mycodo_wrapper
[2018-02-28 08:33:55] 
[2018-02-28 08:33:55] #### Creating symlinks to Mycodo executables
[2018-02-28 08:33:55] 
[2018-02-28 08:33:55] #### Creating files and directories
[2018-02-28 08:33:56] 
[2018-02-28 08:33:56] #### Setting permissions
[2018-02-28 08:34:00] 

appears more than once, in fact, it appears 3 times in the log.

This seems redundant, too:

[2018-02-28 08:25:28] #### Updating pip
[2018-02-28 08:25:48] Requirement already up-to-date: pip in ./env/lib/python3.5/site-packages
[2018-02-28 08:25:52] 
[2018-02-28 08:25:52] #### Installing pip requirements from requirements.txt
[2018-02-28 08:26:13] Requirement already up-to-date: pip in ./env/lib/python3.5/site-packages

Finally, there upgrade of optional components seems OK:

[2018-02-28 08:32:26] #### Checking for updates to optional dependencies
[2018-02-28 08:32:34] 
[2018-02-28 08:32:34] #### No install candidate for smbus
[2018-02-28 08:32:34] 
[2018-02-28 08:32:34] #### No install candidate for serial
[2018-02-28 08:32:34] 
[2018-02-28 08:32:34] #### Installing/updating w1thermsensor with pip
[2018-02-28 08:32:56] Requirement already up-to-date: w1thermsensor==1.0.5 in ./env/lib/python3.5/site-packages
[2018-02-28 08:32:57] Requirement already up-to-date: click in ./env/lib/python3.5/site-packages (from w1thermsensor==1.0.5)
[2018-02-28 08:33:00] 
[2018-02-28 08:33:00] #### No install candidate for RPi.GPIO
[2018-02-28 08:33:00] 
[2018-02-28 08:33:00] #### No install candidate for numpy
[2018-02-28 08:33:00] 
[2018-02-28 08:33:00] #### Installing/updating install-wiringpi with pip
[2018-02-28 08:33:00] 
[2018-02-28 08:33:00] Unrecognized dependency: install-wiringpi
[2018-02-28 08:33:00] #### Ensuring compatible version of influxdb is installed ####

Although wiringpi behaves differently than the others...

[kizniche]

appears more than once, in fact, it appears 3 times in the log.

Part of that short sequence is setting proper file/folder permissions, which needs to be done after a few key events to make sure permissions are set correctly. The execution should be relatively quick, so it's not a big deal to me that there may be some redundancy in it. I'll check it out again and see if it can be improved with just permission setting instead of the other code being executed.

This seems redundant, too

One is the updating of pip itself, the next is checking for python package updates. Both are necessary.

wiringpi behaves differently than the others

This is because it's the first time you updated to a version with wiringpi being optional. Any future upgrade it should either update it (if it was previously installed) or state there's no install candidate and not install it.

[kizniche]

I hope the latest commit, above, doesn't break the upgrade process 😬

[zsole2]

I can test it...

[zsole2]

Thinking of testing... first I need upgrade to master to have the new upgrade script installed, then in a second run is needed to test it. Would it work to the same state of the master, or should I wait for at least 1 new commit?

[kizniche]

You can perform two upgrades to master to use the latest code to test an upgrade.

[zsole2]

Apparently both upgrades went without any problem.

[kizniche]

Thanks for testing. I just fixed some issues, improved the dependency install with a display (and auto-update) of the dependency install log, and added this log to the Mycodo Log page. So, I believe this issue is completely solved.