utilrb does not build on various branches

[psoetens]

I'm investigating why utilrb building with 'rake default' fails on my 12.04 system. A git bisect reveals that the commit c38a197 breaks utilrb building, which appeared first on the toolchain-2.6 branch. The bootstrapping was done using 👍

wget http://rock-robotics.org/autoproj_bootstrap
ruby autoproj_bootstrap git $BUILDCONF_GIT branch=toolchain-2.6

The error message is:

running rake default
INFO: cannot load the Hoe gem, or Hoe fails. Distribution is disabled
INFO: error message is: ERROR: 'rake/rdoctask' is obsolete and no longer supported. Use 'rdoc/task' (available in RDoc 2.4.2+) instead.
rake aborted!
undefined method <<' for LL():Rake::Scope::EmptyScope ./lib/utilrb/doc/rake.rb:45:indoc'

I've tried the stable branch, master branch, all not working. For stable for example I get:

$ rake default --trace
'utilrb' cannot be build -- loading gem failed: no such file to load -- hoe
rake aborted!
Don't know how to build task 'default'
/home/kaltan/vbox/hudson/jobs/bootstrap-toolchain/workspace/CORBA_IMPL/OMNIORB/TARGET/gnulinux/arch/amd64/branch/toolchain-2.6/compiler/gcc/orocos-toolchain/.gems/gems/rake-10.1.0/lib/rake/task_manager.rb:49:in []' /home/kaltan/vbox/hudson/jobs/bootstrap-toolchain/workspace/CORBA_IMPL/OMNIORB/TARGET/gnulinux/arch/amd64/branch/toolchain-2.6/compiler/gcc/orocos-toolchain/.gems/gems/rake-10.1.0/lib/rake/application.rb:148:ininvoke_task'
/home/kaltan/vbox/hudson/jobs/bootstrap-toolchain/workspace/CORBA_IMPL/OMNIORB/TARGET/gnulinux/arch/amd64/branch/toolchain-2.6/compiler/gcc/orocos-toolchain/.gems/gems/rake-10.1.0/lib/rake/application.rb:106:in `top_level'
/home/kaltan/vbox/hudson/jobs/bootstrap-

etc..

$ ruby --version
ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]

Any clues ?

[doudou]

• for bootstrapping toolchain-2.6, the problem is caused by a
  backward-incompatible change in rake. utilrb got fixed later on in
  6c26668
  and 93215b2. I'm thinking more and more
  about using bundler to install gems, which would allow us to pin gem
  versions to avoid these kind of problems in the future.

• these days, utilrb requires hoe and rake-compiler. It looks like you
  manually switched branches but forgot to run autoproj osdeps to get the new
  osdeps. Moreover, support for ruby 1.8 has been removed which means that
  you will have to use ruby 1.9 to bootstrap:

  ruby1.9.1 autoproj_bootstrap git $BUILDCONF_GIT branch=toolchain-2.6

2014-06-24 14:56 GMT+02:00 Peter Soetens notifications@github.com:

I'm investigating why utilrb building with 'rake default' fails on my
12.04 system. A git bisect reveals that the commit c38a197
c38a197 breaks utilrb
building, which appeared first on the toolchain-2.6 branch. The
bootstrapping was done using [image: 👍]

wget http://rock-robotics.org/autoproj_bootstrap
ruby autoproj_bootstrap git $BUILDCONF_GIT branch=toolchain-2.6

The error message is:

running rake default
INFO: cannot load the Hoe gem, or Hoe fails. Distribution is disabled
INFO: error message is: ERROR: 'rake/rdoctask' is obsolete and no longer
supported. Use 'rdoc/task' (available in RDoc 2.4.2+) instead.
rake aborted!
undefined method <<' for LL():Rake::Scope::EmptyScope
./lib/utilrb/doc/rake.rb:45:indoc'

I've tried the stable branch, master branch, all not working. For stable
for example I get:

$ rake default --trace
'utilrb' cannot be build -- loading gem failed: no such file to load -- hoe
rake aborted!
Don't know how to build task 'default'
/home/kaltan/vbox/hudson/jobs/bootstrap-toolchain/workspace/CORBA_IMPL/OMNIORB/TARGET/gnulinux/arch/amd64/branch/toolchain-2.6/compiler/gcc/orocos-toolchain/.gems/gems/rake-10.1.0/lib/rake/task_manager.rb:49:in
[]'

/home/kaltan/vbox/hudson/jobs/bootstrap-toolchain/workspace/CORBA_IMPL/OMNIORB/TARGET/gnulinux/arch/amd64/branch/toolchain-2.6/compiler/gcc/orocos-toolchain/.gems/gems/rake-10.1.0/lib/rake/application.rb:148:in
invoke_task'
/home/kaltan/vbox/hudson/jobs/bootstrap-toolchain/workspace/CORBA_IMPL/OMNIORB/TARGET/gnulinux/arch/amd64/branch/toolchain-2.6/compiler/gcc/orocos-toolchain/.gems/gems/rake-10.1.0/lib/rake/application.rb:106:in
`top_level'
/home/kaltan/vbox/hudson/jobs/bootstrap-

etc..

$ ruby --version
ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]

Any clues ?

—
Reply to this email directly or view it on GitHub
#5.

[jbohren]

For running ruby 1.9.1 on Ubuntu 12.04, you can install an older version of rake-compiler:

gem install rake-compiler --version 0.8.3

Related: rake-compiler/rake-compiler#93

[doudou]

• if you can, avoid installing gems with sudo
• the version of rake-compiler that autoproj installs is already proper. You might have to update the 2.6 toolchain osdeps for it to work though (look at what is in master)

[jbohren]

the version of rake-compiler that autoproj installs is already proper. You might have to update the 2.6 toolchain osdeps for it to work though (look at what is in master)

Do you mean this: https://github.com/orocos-toolchain/utilrb/blob/master/Rakefile#L16

[doudou]

No, I mean this https://github.com/orocos-toolchain/autoproj/blob/master/orocos.osdeps#L162. For the ROS-specific stuff, you'll have to adapt (obviously).

We can't unfortunately change the gem dependency constraints in the Rakefile as ruby 2.0 and 2.1 do require rake-compiler > 0.9

[jbohren]

See orocos-toolchain/orogen#14 (comment)

[jbohren]

Sorry, up to you to close, @psoetens

[jbohren]

No, I mean this https://github.com/orocos-toolchain/autoproj/blob/master/orocos.osdeps#L162.

Thanks, I wasn't sure where the deps for this were defined.

We can't unfortunately change the gem dependency constraints in the Rakefile as ruby 2.0 and 2.1 do require rake-compiler > 0.9

That's fine, but for anyone who wants to run this stuff on Ubuntu 12.04, they can see my above suggestion for force-downgrading the gem version. (I updated it to remove the call to sudo)

[meyerj]

I guess we have a more general problem here. Some ROS tools still (e.g. rosdep) prefer manifest.xml over package.xml as the package manifest and the contents of the manifest.xml file has to be "ROS compliant" and consistent with package.xml if both files exists.

Some commits in the current master branch of utilrb added <depend package="..."/> lines to manifest.xml or replaced existing <rosdep name="..."/> lines:

• 37c5c1e#diff-14
• f85da49#diff-0
• 510dd96#diff-0

As these dependencies are not existing ROS packages, rosdep returns an error:

ERROR: the following packages/stacks could not have their rosdep keys resolved
to system dependencies:
utilrb: Missing resource ruby

This would block all releases in ROS and the build server cannot properly resolve the dependencies. Additionally, even if all dependencies would be declared with <rosdep name=...> we have to make sure that the names can be resolved by respective entries in the rosdep database. This is currently not the case for hoe-yard, rake-compiler and readline (which is called libreadline in rosdistro).

I am not an autoproj user, but from the documentation it seems that <rosdep> is the right tag for system dependencies. What is the reason for the transition to <depend package=...>? Is there a simple solution to fix these compatibility problems?

Would it be an option to release orocos_toolchain without utilrb and dependent packages in ROS indigo (only log4cpp, rtt, ocl) and drop ROS support for the rest?

[jbohren]

I guess we have a more general problem here.

Additionally, rosdep install --from-paths won't work with orogen because of the tools/metaruby dependency.

[doudou]

In autoproj, is an alias for and the latter is preferred. For the sake of compatibility, I have no problem with reverting to using for the orocos-toolchain packages. Will have to add a big fat warning to avoid the problem in the future. I assumed that ROS would have preferred package.xml against manifest.xml if both exist ... The current policy makes zero sense to me.

[doudou]

Additionally, rosdep install --from-paths won't work with orogen because of the tools/metaruby dependency.

What is rosdep install --from-paths doing exactly ?

[meyerj]

What is rosdep install --from-paths doing exactly ?

rosdep install --from-path src [--ignore-src] searches for packages in folder src and installs all system dependencies, as if rosdep install <package> would have been invoked for every package. The optional --ignore-src option ignores all dependencies which are existing packages in the src folder or any other folder in the ROS_PACKAGE_PATH.

For the discussion here it is not relevant whether rosdep was invoked with --from-path or with a single package. Both methods return an error if the manifest.xml file is invalid for ROS or contains unknown keys.

This would block all releases in ROS and the build server cannot properly resolve the dependencies.

I am sorry, my statement above was wrong in this point. The ROS release tool bloom does not look at the manifest.xml file, only package.xml. Only rosdep itself when used in source builds to install dependencies is broken, which is a minor problem and would not block a release in ROS.

I assumed that ROS would have preferred package.xml against manifest.xml if both exist ... The current policy makes zero sense to me.

I agree. The current implementation is not consistent. We could submit a pull request to rospkg or rosdep.

Relevant functions:

• https://github.com/ros-infrastructure/rosdep/blob/master/src/rosdep2/rospkg_loader.py#L120
• https://github.com/ros-infrastructure/rospkg/blob/master/src/rospkg/manifest.py#L354

The discussion in ros-infrastructure/rospkg#30 is related, but about getting compile and linker flags with rospack and not about rosdep. rosdep should IMHO prefer package.xml.

[doudou]

At this point, what would be the best course of action for you guys ?

If needed, we can:

• revert to using rosdep for the relevant dependencies
• add an alias "metaruby" for tools/metaruby in the orocos toolchain autoproj config and use it in orogen

[meyerj]

Submitted issue/question to rosdep here: ros-infrastructure/rosdep#353

As a fast solution that does not require any updates in ros tools I would recommend to revert to <rosdep name="..."/> in the manifest.xml for everything that is not another orocos_toolchain package and not use slashes (/) in the names mentioned there, if this has no drawbacks for autoproj. Then we only have to make sure that all keys are in rosdistro's ruby.yaml and resolve to appropriate system dependencies for all supported distros.

For the case the latter is not possible as not all dependencies are available as binary packages in the official repositories, then the only remaining options are to provide them ourselves and ask the OSRF team to add them as 3rd-party system packages to the ROS repo or to release the toolchain without utilrb in indigo.

[meyerj]

To summarize the package.xml vs. manifest.xml issue here:

• The incompatibility does not break building binary packages for ROS and only affects users who want to do a source build of the toolchain and use rosdep to install all dependencies. As such, it is a minor issue and we should not sacrifice advantages of the new additions to manifest.xml for autoproj.
• Updating the manifest.xml files would only solve the rosdep problem if all mentioned package names are
  1. in a format that is accepted by ROS (no slashes)
  2. known to rosdep either as a package released in ROS (if declared with <depend package="..."/>) or a system dependency (if declared with <rosdep name="..."/>)
  3. not have optional dependencies declared with optional="1" (or at least those packages also have to be valid as rosdep simply ignores the unknown attribute)
  4. and if all system dependencies are available as Debian/Ubuntu packages.

As there is probably no good solution I would vote for giving up the rosdep compatibility and stick with the current state of maintaining the two files separately, eventually in different branches.

[doudou]

As there is probably no good solution I would vote for giving up the rosdep compatibility and stick with the current state of maintaining the two files separately, eventually in different branches.

Would also be my preferred option, but since some people use autoproj to install the toolchain, it does not seem to be acceptable.

About the constraints:
i. we can add a 'metaruby' alias for tools/metaruby so that the manifest can use 'metaruby' and make ROS happy
ii. autoproj interprets both formats
iii. the optional dependencies should be turned into depend_optional (a tag I added in autoproj for the sake of ROS compatibility in the orocos toolchain a while back) which is ignored by ROS AFAIK, or kept as-is for these dependencies that we would rather keep. This is obviously a case-by-case basis. flexmock in utilrb should be turned into depend_optional for instance.
iv. every dependency is either packaged or source-based, so I don't see that as a problem.

[meyerj]

About the constraints:
i. we can add a 'metaruby' alias for tools/metaruby so that the manifest can use 'metaruby' and make ROS happy
ii. autoproj interprets both formats
iii. the optional dependencies should be turned into depend_optional (a tag I added in autoproj for the sake of ROS compatibility in the orocos toolchain a while back) which is ignored by ROS AFAIK, or kept > as-is for these dependencies that we would rather keep. This is obviously a case-by-case basis. flexmock in utilrb should be turned into depend_optional for instance.
iv. every dependency is either packaged or source-based, so I don't see that as a problem.

I created patches for utilrb, typelib and orogen with updated manifest.xml and package.xml files according to your suggestions. Those would be compatible with rosdep:

• utilrb: meyerj@6a156fd
• typelib: meyerj/typelib@7d6c839 (only package.xml)
• orogen: meyerj/orogen@9af785e

If those patches would be acceptable for autoproj, they should be merged into master.

[meyerj]

Pushed updated manifest.xml for orogen: meyerj/orogen@8cb82a5

See meyerj/orogen@9af785e#commitcomment-8462812.

[meyerj]

We should close this in favor of #11, orocos-toolchain/typelib#24 and orocos-toolchain/orogen#24. The manifest discussion is not directly related to the original topic of this issue.