Groovy rosmake / rosdep is very slow

[pirobot]

After upgrading from Fuerte to Groovy (Debian packages under Ubuntu 12.04) I have found rosmake runs much slower than it used to. For example, for one of my typical stacks (http://ros-by-example.googlecode.com/svn/trunk/rbx_vol_1) which has 7 packages and mostly Python code, rosmake takes 7.9 seconds under Fuerte and 149 seconds (2 min 29 sec) under Groovy. That's 18x slower.

A posting on answers.ros.org (http://answers.ros.org/question/53925/rospack-became-agonizingly-slow/#53946) suggests that the problem is with rosdep2.

[roehling]

The YAML parser seems to be the culprit: http://pyyaml.org/ticket/70
As Ubuntu ships with the optimized CLoader by default, I modified the cache loader accordingly: fkie-forks/rosdep@89a6848

However, even the "fast" loader introduces a noticeable delay to rospack when compared to Fuerte. A better solution would be to get rid of the repeated YAML parsing entirely and store the cache in a pre-parsed format that is optimized for fast loading. Maybe the source list can be serialized using Pickle or marshal?

[tfoote]

b0aaeb0 in #222 provides a ~6x speedup for an empty package rosmake timing

[tfoote]

The remaining time is mostly os.walk inside catkinpkg

Closing and reopening as ros-infrastructure/catkin_pkg#33

[jbohren]

Sweet.

Fuerte: https://gist.github.com/jbohren/4997805
Groovy: https://gist.github.com/jbohren/4997788
Groovy (post-rosdep-fix): https://gist.github.com/jbohren/4999368

[roehling]

OS detection is time-consuming as well: ros-infrastructure/rospkg#28