Proposal: split all (robot) drivers into separate repositories

[gavanderhoorn]

The suggestion is simple: instead of hosting the various (robot) drivers in the same repository as the robot support, Gazebo and MoveIt packages, split them out into their own repositories.

Rationale:

1. easier to support multiple driver pkgs: as the project becomes more mature, we're seeing multiple drivers for the same platform/controller/robot, and even per-controller-generation drivers or drivers that make use of alternative interfaces that depend on software options on controllers. This gets even more complex with manufacturer/OEM supported components and community backed ones. The package that happens to have been written first typically becomes part of the main repository, leading to launch files hardcoding dependencies on that pkg, and users assuming that driver is the driver for a particular series of devices/controllers.
2. easier management of federated maintainership / easier transfer of maintainership (possibly even to the mfg if/when they get involved): instead of assuming responsibility for all packages, it's just one or two, which is much more manageable.
3. easier integration/hosting of contributions (almost reverse 'transfer of maintainership'): separate driver repositories make accepting contributions easier: instead of having to (somehow) merge an external repository, it would come down to a transfer of a repository.
4. increased re-usability: repositories with fewer packages are easier to manage and the pkgs hosted there are easier to re-use. This is especially true in development environments (ie: less pkgs in a workspace, no CATKIN_IGNORE or maintaining whitelists to build just the pkg you need, instead of the 25+ support pkgs for robots you don't want).
5. decouple drivers from support, Gazebo and MoveIt pkgs: these are typically orthogonal to the specific driver used (as long as the driver conforms to expected/dictated interfaces) and they typically have a different development pace from drivers, where the former are typically rather static, and the latter can see quite some development (keeping them compatible with newer robots, controllers, software versions, etc).
6. driver re-use is more complex than re-use of other pkgs: this is especially true when taking things like special controller software options and application-specific configurations and requirements into account. Management of driver variants is easier if they're not part of a repository with all sorts of other pkgs (avoiding branching and tracking an upstream fi). It also facilitates dealing with the increase in custom developed or in-house maintained drivers (as opposed to upstream contributions).

Some possible disadvantages:

1. fragmentation / confusion: without proper indexing/documentation, it will be less obvious for users/newcomers to grasp the set of pkgs that would be needed for a particular hardware setup (ie: "where is the driver?"). The ROS wiki mitigates this as driver repositories can be indexed similar to how it is done now. Metapackages can be exploited to tie drivers and other pkgs together in a similar way. Repository-level readmes should be updated to explain this as well (list of compatible drivers, etc).
2. increased maintenance / more complex inter-repository management: some projects have actually migrated to a single-repository hosting all pkgs, fi because this facilitated cross-package development (MoveIt comes to mind). This would seem to be less of an issue with the (robot) drivers, as there are typically no interdependencies between drivers and other pkgs in their repositories.
3. more repositories to release: while the total nr of pkgs doesn't increase, having drivers in their own repositories will lead to more repositories to "Bloom". However, as mentioned earlier, drivers follow a different development pattern when compared with other packages, so a (re-)release of a driver repository will typically not require a release of a repository containing support, Gazebo and MoveIt pkgs. On average the increased release-maintenance is expected to not be prohibitive.

[gavanderhoorn]

Some examples for rationale 1: the various ABB drivers (EGM, plain rapid, ros_control based ones, industrial_robot_client ones), the various UR drivers (the original one (ur_driver), ur_modern_driver, proposed upgrade of ur_modern_driver, a now proposed customised version with no ros_control support (but better performance in other situations)) and multiple Fanuc drivers.

[gavanderhoorn]

@shaun-edwards, @Levi-Armstrong, @jrgnicho, @JeremyZoss, @Jmeyer1292, @minlingc, @ipa-mirb, @robinsonmm, @ThomasTimm: comments, ideas?

[gavanderhoorn]

We had a similar discussion in ros-industrial/motoman#187 with input from @ted-miller.

@ted-miller was not in favour of splitting out the driver, arguing that it might lead users to believe that the driver is something that can be used without ROS/the other packages.

I believe we can mitigate that with proper documentation, but at least in ros-industrial/motoman#187 the conclusion was to not split out the driver for now.

[ipa-mirb]

easier management of federated maintainership / easier transfer of maintainership (possibly even to the mfg if/when they get involved): instead of assuming responsibility for all packages, it's just one or two, which is much more manageable

This is the point for which I would be mostly in favor of doing this. It also looks like companies starting to commit more tech development in-house to then possibly open-source it when it's ready (e.g., http://rosindustrial.org/news/2018/1/8/announcing-ros) would rather keep pkgs on their own repos.

[BrettHemes]

👍 I support this fully, especially in regard to advantages 1 and 2. Considering the disadvantages, none of them are deal-breakers in my mind and they all pale in comparison to the identified advantages.

The current state of the UR driver(s) alone should be enough to warrant some form of action.

[130s]

I'm not maintaining any driver package, and I'm usually in favor of aggregating packages into fewer repositories (e.g. I just consolidated a few repos), but the rationale provided here makes sense. I just can't stress enough that maintenance effort could blow out so maybe start from smaller sets and see (e.g. split a single manufacturer's repo into each driver's repo)? Just an idea.

[gavanderhoorn]

@130s: I'm not really a fan of merging repositories (I was not a fan of MoveIt doing it, but that was a rather extreme case), but I can see why in some cases it might make sense.

I believe however that in this case the additional work will be manageable -- if there is any at all -- as most of the repositories that would be split see very few releases. And the releases that are done, are typically because of the driver(s). So repositories that -- after the split -- contain mostly MoveIt, Gazebo and description packages will probably not need that many releases, leading to on average the same maintenance load (or at least, wrt releases).

[ipa-nhg]

Great idea! For me, it makes total sense! 👍

If I want to launch a robot I prefer that my workspace only contains the configuration of my robot setup (URDF, yaml files) and the launch files to start my application. A stable released version of the driver is likely installed into /opt/ros… using rosdep.

From the point of view of the maintainer: the release strategy of a driver repository (only updated when a bug is found) had nothing to do with the release of configuration or bringup packages (usually not released because have to be updated constantly).
But also from the point of view of an user is more convenient because do not have to build the source code of the driver and take care of the installation of the required version of dependencies (in case that, for example, a change of the upstream code of the driver requires an update of one of its dependencies).

[gavanderhoorn]

I've done some work on this and that has resulted in the following (test) repositories:

• abb_driver
• fanuc_driver
• motoman_driver
• staubli_val3_driver

Not all of them were created using the same process, but that was on purpose. The one I like best so far is abb_driver: it has all the commits, tags and branches and keeps the readme and travis config of the main repository. This was done using various variants of git filter-branch.

An entirely different approach would be to simply copy a repository to a new name (ie: abb to abb_driver) and then remove everything apart from the abb_driver package in a single commit. This would retain the complete commit history, similar to the git filter-branch approach, but with the added benefit of not rewriting any history (so all references to commits, tags and PRs will remain in tact). Something like this was done for the migration of rosconsole to a separate repository (see ros/rosconsole).

A disadvantage of this last approach is that the history of the new repository will contain everything that was there before removal of the other pkgs, so that could be confusing (go back 'far enough' and all these unrelated pkgs start to appear). It also bloats the repository.

[mathias-luedtke]

I would prefer the git filter-branch approach for the target repository (in source repo, just git rm).
There is not much benefit in keeping the commit hashes,
If we want to try really hard, the mentioned commits could be rewritten in the commit messages:
https://github.com/Blaisorblade/git-filter-branch-msgs (first google hit)

[ipa-nhg]

+1 for git filter-branch. That the new *_driver contains the history of the other packages, that still exist but in a different repo, is weird...

About the variants, I would also preffer the abb_driver approach. @gavanderhoorn please correct me if I am wrong, but it is the only one that keeps the branches. For me it is also important that we keep the travis configuration and the LICENSE file.

The Readmes should be updated, we can create a template for the readmes of the *_driver repos pointing to the repository that contains the configuration packages, but for a first commit I have no problem about keep the old Readme files...

[gavanderhoorn]

abb_driver is the only one that currently has everything 'migrated'. I didn't bother to do that for all the others.

[gavanderhoorn]

I've gone ahead with this and extracted staubli_val3_driver from staubli_experimental. New location: ros-industrial/staubli_val3_driver.

Reason I started with this driver first is some incoming PRs in the near future and as there has been relatively little development of this package (and thus no complicated history), extracting it was relatively simple.

[gavanderhoorn]

Started with ros-industrial/abb in ros-industrial/abb#179.

[gavanderhoorn]

abb_driver has been migrated: ros-industrial/abb#180.