Set a minimum version policy for dependencies at a global or project-wide level

[PeterBowman]

There seems to be an unwritten rule about always developing against the latest version of YARP. After having read robotology/community#173, I concluded that we may benefit from defining such policies as projects grow larger and more people get involved in their development. I perceive this as a bit of a pressing matter since several of my past commits were mostly aimed at dealing with deprecations, code no longer working due to certain upgrade and achieving backwards compatibility where possible. For example, I'm unsure about the minimum version of CMake I should be targeting at and in consequence I will miss some of the newer features. Additionally, it's not clear to which extent hacks (sometimes dirty ones) must be applied to support outdated software.

The YARP team has a nice page that covers all dependencies:
http://wiki.icub.org/wiki/YARP_Supported_Distributions

Seeing this from another perspective, care should also be taken to conform to the latest releases of upstream software - in case such policy applies, too (as stated in the first sentence, YARP could be a candidate in this respect). To list some of these issues, developers should be aware of incoming deprecations (see roboticslab-uc3m/yarp-devices@efb8962) and keep their documentation up-to-date (roboticslab-uc3m/installation-guides#1).

[PeterBowman]

An example of a breaking change in YARP, should we conform to the always-latest-release-policy: the new EXTRA_CONFIG parameter in 2.3.68.

[jgvictores]

I agree on the fact that this is an important issue.

Where would this go? Perhaps on an individual file in the root of installation-guides?

Alternatives would include https://github.com/roboticslab-uc3m/best-practices and perhaps a developer manual as suggested in #12, but maybe there is where it can have most visibility.

[David-Estevez]

If I understood correctly, installation-guides only specifies how a dependency should be installed, and then each repo specifies somewhere within that repo which dependencies are needed for that concrete repo.

In that case, the minimum version should be specified along each dependency within that repo. Note that several entries for the same dependency might be required in installation-guides, one per version.

[PeterBowman]

Another YARP breaking change: roboticslab-uc3m/yarp-devices@b342044.

[PeterBowman]

In a recent meeting, @jgvictores and I concluded that we could just specify the minimum version for each dependency along its name on each repo's installation guide (see David's comment), dropping the idea of a master table on a separate .md file that would hold this information. That is, for teo-body/doc/teo_body_install_on_debian_6.md#install-the-software, this would simply result in:

• Install CMake (Debian 6.0): v2.8.12+
• Install YARP (Debian 6.0): v2.3.64+

(version numbers were chosen only for illustrative purposes) The final format may vary (I'd rather use parentheses and no v prefix, but in this case it just didn't fit otherwise).

[jgvictores]

Okay, I've been populating teo_body_install_on_debian_6.md.

• On TEO, we use the most recent Debian squeeze, 6.0.10.
• I've updated APT sources for squeeze and squeeze-backports on the same page.
• The use of CMake 2.8.9 (Debian 6.0.10) from squeeze-backports is now clear.

Regarding YARP, here are the facts:

• Both 2.3.66 and 2.3.66.2 compile on Debian 6.0.10 with no additional CMake flags.
• Both 2.3.68 and the latest commit on master (e1221283abb1abc77619429bb4d9408cbb1cf6c8) at time of writing compile with the CMake flag -DYARP_NO_DEPRECATED_WARNINGS=ON.
  Options:

1. Freeze on 2.3.66.2.
2. Freeze on 2.3.68 with -DYARP_NO_DEPRECATED_WARNINGS=ON.
3. Do not freeze; recommend e1221283abb1abc77619429bb4d9408cbb1cf6c8 with -DYARP_NO_DEPRECATED_WARNINGS=ON.
   My vote goes to (3), always pushing towards the latest mastercommit so we get any improvement we do upstream. I'd also be okay with (2), but I'm afraid with what will happen with some patches I have in mind.
   @PeterBowman @David-Estevez What is your opinion?

PS: I'll be working with 2.3.68 this morning to see if the analogServer is working correctly.

[David-Estevez]

I think we should be working with the gitflow in our minds:

• For the stable software (demos with Teo, and other software that must work always), we should choose the latest stable choice. This requires to freeze in some version we know it will work. This would be the master branch.
• For development (and probably the patches @jgvictores has in mind), we should aim to improve the software not only by adding our own features, but also adding upstream improvements. Therefore, when working on the develop branch it would be ok to use a more updated version, always specifying the version in the requirements/dependencies file.

This, of course, has the problem of maintaining two software versions with their dependencies in the robot. There are many choices to solve this problem, although all of them might not be suitable for Teo:

• Maintaining two different partitions/OS installed in Teo, one formaster (provides stability) and other for develop (provides up-to-date software and features).
• Docker?
• VMs

[PeterBowman]

I'm with option 3, too. @David-Estevez please keep in mind that the ASIBOT's PC is actually a Roboard (RB-110, https://www.robot-r-us.com/vmchk/microcontroller/roboard-rb-110-kit.html). We are nearly out of space ATM for installing additional software (Docker/WMs weigh too much), and I'm not sure yet we can go past the current 8 GB memory size for the Micro-SD card.

[David-Estevez]

@PeterBowman What about two Micro-SD cards, one for develop and other for master? Is that feasible or too crazy?

[jgvictores]

@David-Estevez We've tried to implement this on TEO (see roboticslab-uc3m/yarp-devices#43) but haven't been doing this, it just turns out not to be very practical.

PS: Regarding YARP version on teo-body, I've started roboticslab-uc3m/yarp-devices#89.

[PeterBowman]

@David-Estevez there is only one slot, and the USB is already used by the WiFi adapter. I also forgot to mention that ASIBOT's casing is quite hard to remove, that means that any hardware change (even as simple as changing the SD) would require some work and attention to:

1. undock the robot arm
2. unscrew the casing, do whatever we have to do
3. take the utmost care for not leaving any loose wire connections

[David-Estevez]

I see, it seems that working with robots is a little bit more difficult than working with PCs 😅

[jgvictores]

A quite complex repo in terms of installation and versions that has been updated is the current teo-body.

Can we conclude this is the accepted way to document dependencies and their versions, and can close the issue once the following two steps are done?

• Decide if installation guides are mandatory even if they are very complex and there are are automated tools. (I'd say they shouldn't be mandatory in presence of decent automated installation tools).
• Answer on How should I document my project's dependencies? #14
• Document on https://github.com/roboticslab-uc3m/best-practices

[David-Estevez]

@jgvictores automated installation tools means apt-get , pip , or similar, right?

[jgvictores]

@David-Estevez Yes, I meant apt-get etc

[David-Estevez]

So, if I have understood correctly:

• The installation-guides are a fallback for cases where a dependency cannot be installed/configured with a simple apt-get install or pip install.
• Each repo can contain the dependencies expressed as dependency_name=version (with = or >=), a command to install them all in case an automated tool is available and some links to the guides in the case the installation process is more complex than that.

[jgvictores]

The installation-guides are a fallback for cases where a dependency cannot be installed/configured with a simple apt-get install or pip install.

Please note there was a misunderstanding here. I was referring to the case where the entire repository or all its dependencies could be simultaneously compiled/installed with a single line (via some automated script, PPA or similar).

However, having an automated script should not exclude a best practice of documenting how to install dependencies and compile/install from source.

For dependencies, I still support creating a new page on roboticslab-uc3m/installation-guides. There are several reasons for this:

1. Even looking at different versions of a specific distro, a package can disappear or its name can change.
2. We can easily end up recommending different ways to install the same package, ending up in multiple possibly conflicting installations. Giving visibility to what each project uses on a shared page can really help avoid this bad situation.

Each repo can contain the dependencies expressed as dependency_name=version (with = or >=), a command to install them all in case an automated tool is available and some links to the guides in the case the installation process is more complex than that.

1. We were looking a slight different syntax: CMake v2.8.12+ (Debian 6.0.10)
2. A >= is dangerous, given that maintainers too often introduce changes that break bw-compat. I'd support the assumption that it's a use at your own risk if using a != version.

Other than being tedious (and the trapped in a tunnel, OMG I forgot to download uc3m/installation-guides scenario in which case you can't install the dependencies anyway), do you think this is reasonable?

[PeterBowman]

@David-Estevez I was told that the textiles project is particularly hard to document because of a huge number of dependencies. I wonder if said documenting process could be, at least partially, supplanted by the requirements.txt file. I stumbled upon a more complex case at the pywikibot framework which might interest you, especially concerning dependency version ranges and python 2.x/3.x distinction:

flickrapi>=1.4.5,<2 ; python_version < '2.7'

BTW another important change in YARP, I wonder if it could break anything on our end: robotology/community#187.

[David-Estevez]

@PeterBowman That is something I am still considering. But there are some dependencies that require certain development version or compiling one specific commit of the required library to have some cutting-edge functionality. The project also involves C++, so CMake is required at some point to compile the C++ parts, so this has to be taken into account.

We are in a bit of a hurry now since we have an approaching deadline, so probably I won't be dealing with textiles dependencies / installation until next month. Then I will deal with the Spring cleaning project in textiles.

[PeterBowman]

@PeterBowman That is something I am still considering. But there are some dependencies that require certain development version or compiling one specific commit of the required library to have some cutting-edge functionality. [...]

See https://pip.pypa.io/en/stable/reference/pip_install/#git.

[David-Estevez]

Unfortunately, in my case it is a CMake based repository split into two different repositories (OpenCV 3 with contrib, Python 3 support and fix for SVM_load, see https://github.com/roboticslab-uc3m/installation-guides/blob/develop/install-opencv.md).

Nevertheless that reference page is an excellent resource for all my incoming work on simplifying installation, thanks!

[PeterBowman]

As a side note: @jgvictores and I had concluded that all CMake list files may bump the required minimum version to 2.8.11 (now using 2.8.9 due to YCM or even 2.6 due to old project generators). This is guaranteed to work on Trusty onwards, even Precise systems may find appropriate packages in the kalakris PPA (sudo add-apt-repository ppa:kalakris/cmake).

For reference, see also: https://cmake.org/Wiki/CMake_Version_Compatibility_Matrix.

[jgvictores]

Bump https://packages.ubuntu.com/trusty/cmake3 ?

[PeterBowman]

Idea: set 3.5 (3.5.1) as the minimum CMake version everywhere. Thanks to:

• https://packages.ubuntu.com/trusty/cmake3
• Try hc-driver-4.9 yarp-devices#118

[PeterBowman]

YARP is going to require CMake 3.5, soon (robotology/community#280).

[PeterBowman]

Check this out (kinematics-dynamics-install.md):

First install the dependencies:

• Install CMake 3.5+
• Install YCM 0.8+
• Install YARP 2.3.70+
• Install KDL 1.4+
• Install color-debug

At the time of writing, the first three requirements apply now to several repos in our org.

[PeterBowman]

@jgvictores are you happy with #7 (comment)? Can we finally close this ticket?

[jgvictores]

Perfect, yes!