CMake package conventions and conformity

[mxgrey]

For the sake of pull request #461 I started digging through some of CMake's packaging documentation and also reviewing DART's current cmake setup. I thought I would discuss some of my findings and my thoughts on how we might improve DART's packaging. Some of this also comes from discussions with @mkoval.

• I've noticed that according to CMake documentation, the libraries that are returned by find_package are supposed to be absolute paths, but DART currently just returns library names. It may be a good idea to follow the convention that's stated by the doc.
• The split between dart-core and dart is understandable, but I think it could be handled more gracefully. What I would propose is renaming dart-core to just be dart, and then the extra features that are currently in dart should each be split out into its own COMPONENT based on their dependencies. For example, the contents of dart/gui which depend on OpenGL should go into a dart-gui library. The contents of dart/planning which depend on FLANN should go into a dart-planning library.

It gets a bit trickier when it comes to components that are nested within namespaces. For instance, dart/optimizer/nlopt could go into a library named dart-nlopt or it could go into dart-optimizer-nlopt. The first option offers a shorter name, which is nice, but the second option is more precise and consistent. The second option has the added benefit of avoiding possible future collisions, for example if we end up with a dart/planner/nlopt extension which is independent of the dart/optimizer/nlopt. After giving it some thought, I think I would prefer using the full name, but I'm not necessarily opposed to the shortened name.

• We would use the find_package(DART COMPONENTS xxx) feature of CMake to allow the user to decide what they want to include in the ${DART_LIBRARIES} variable. For example, find_package(DART COMPONENTS gui planning utils-sdf) would set ${DART_LIBRARIES} to include dart along with the GUI library, planning library, and SDF parsing library.
• Each of the libraries would be available on the apt repo independently of each other, giving the user more control over which dependencies they want to deal with.

I think this overall approach would conform better to how CMake packages are usually handled. It may also make DART easier for end users to install, because they'll have a finer level of control over the dependencies.

[mxgrey]

It occurs to me that, while this may be easy from the cmake side of things, it could make the debian packaging significantly more complex (and maybe Homebrew as well, but I've never touched Homebrew before, so I don't know). I wonder if it would be possible/useful to have cmake auto-generate the debian packaging files using the configure_file command.

Edit: On second thought, this would probably be more trouble than it's worth.

[psigen]

One minor thought: If you did actually leave dart-core as dart_core (e.g. libdart_core.so), that might help eliminate the ambiguity against the existing DART CMake implementation that @mkoval experience recently.

Or really any naming convention that makes CMake use a name other than just DART when resolving this. There are just so many projects named DART these days. :)

[mxgrey]

This hasn't been mentioned here yet, but we're in the process of deciding upon an entirely new name for DART, so that conflict won't be an issue in the future. There should be an official announcement about this in the not-too-distant future. So everywhere that I used "dart" in the earlier posts, just imagine that's a placeholder for whatever the new library name will be.

[scpeters]

The components of dart_core used to be called rtql8, which I always thought was a clever, unique name.

[jslee02]

rtql8 is one of the candidates. One possible issue with rtql8 is it includes number at the end so that it causes confusion with version number.

[scpeters]

One random example of software with number in name: x264 video encoder

[psigen]

x264 makes sense, since the x264 codec is well-known specification. Maybe rtqlate would be a reasonable compromise?

My two cents are that while the name is pretty cute, it's also nice to have something plain and readable, especially if it needs to get used in literature references or international codebases where the onomatopoeia isn't going to make any sense.

Consider cases like libiberty, which is named to get the cute linker flag -liberty, but then gets compiled and referred to as the typo-friendly libiberty.a.

Anyway, it sounds like @jslee02 has a list going already.

[mxgrey]

We should probably hold off on discussing the merits of potential names until the official thread is posted (we'll be posting it as a Github issue). It'll be better to have all that discussion consolidated in one place.

[jslee02]

Beside on the renaming of DART, I like the idea of reorganizing DART as proposed. This would allow us to install dart more systematically with the package manage systems. For example, we cannot use BulletCollisionDetector when we install dart-core or dart from apt-get. In order to use it, we should build DART from the source on a system libbullet-dev is installed. However, we would be able to use it by installing dart-collision-bullet (it would of course require libbullet-dev to be installed).

If anyone have a plan on implementing this I would like to do.

Here is tentative list of libraries I'm thinking of:

New lib	Old lib	Note
dart-all	dart-core + dart + [alpha]
dart	dart-core
dart-collision-bullet
dart-optimization-nlopt
dart-optimization-ipopt
dart-optimization-snopt		not included in dart-all
dart-planning
dart-gui
dart-utils
osgDart

Please let me know if there is missing.

alpha would contain all the optional components of dart whose dependency libraries are in the each package manager (apt-get and homebrew).

[psigen]

@mkoval and I were also discussing the merits of reorganizing into modules as far as contributing new functionality. Packages can be built and tested in separate repositories, and later be merged as components (as they become mature) without heavy restructuring.

[mxgrey]

If you're referring to git submodules, I've heard both very good and very bad things about using them. I haven't attempted to use them myself, so I don't know exactly where to stand on them. If anyone is experienced with using them and has good reference material for them, I'd be interested in learning more.

[jslee02]

I also haven't used the functionality yet. It would be a good chance to learn about it to me.

[psigen]

I wasn't actually referring to git submodules... they have some serious tradeoffs in complexity and semantics:
http://slopjong.de/2013/06/04/git-why-submodules-are-evil/
https://news.ycombinator.com/item?id=3904932

One alternative you might want to track if you are interested in that approach is git subtree.

I just meant that we could easily move code into the mainline repo without having to merge files into the source directories and refactor the build system to accommodate.

[jslee02]

I don't have a strong objection but am not sure if it is worth to have multiple repositories for each module when the modules are quite small. It would be possible to resolve the package.xml issue for different packages discussed in #523 though.

It would be good if someone can let me know the advantages of doing so.

[psigen]

In general, I would advise against many small repositories:

• It is much harder to version than one repository, mutiple repositories need to be forked together if there are API changes.
• It means that getting useful source may require many clones, each of which can have spurious failures and makes it necessary to manage checkouts very carefully, using tools like wstool or myrepo.
• It is harder to host, as the repos will tend to clutter web interfaces and split up documentation and issue tracking.

It does however help with the following:

• It is easier to handle components that have large binary files, such as mesh models, etc.
• It is trivial to add/remove components from the mainline repository, as they are just added symbolically.
• It may be easier to deal with components that are proprietary and/or have conflicting licenses.

As far as I can tell, no one here, including me, is strongly in favor of splitting into multiple repositories. We have been discussing just organizing the directories of the main repo into a set of modules that build separately.

[mkoval]

I agree with @psigen. I think that it is a great idea to split DART into multiple libraries to help manage the dependency explosion. However, I very much do not want DART split into multiple repositories for the reasons that he outlined.

Even though all of the modules are in one repository, I think each module should:

• have defined inter-module dependencies
• have defined external dependencies
• correspond to a include directory with the same name
• be exposed as a COMPONENT in CMake

My biggest confusion with multiple-component projects is the confusion about what you need to depend on. This is a constant frustration with Boost: the only way I know if I need to depend on boost-system is to run the build and see if linking fails. I don't want to have a repeat experience in DART. 😄

[jslee02]

I believe this is addressed by #587. Please reopen if there are more things need to be handled.