Migration to colcon-enabled version (incl. ROS2 support)

[mathias-luedtke]

As already discussed in #333, the migration to colcon is the best way to enable ROS2 support.
(see https://github.com/ros-industrial/industrial_ci/projects/1 as well)
I have prepared a working prototype (https://github.com/ipa-mdl/industrial_ci/tree/colcon), which I#d like to move upstream

This new version introduces some new features, but drops some as well (see below).

To migrate to the new version, I propose the following process:

• clean up the prototype
• update documentation
• add migration guide
• create colcon branch upstream (from master)
• create pull request to discuss the changes
• create legacy branch (same as master for some time)
• make legacy branch default, keep master updated
• Announce on ROS discourse ([1], [2])
• make legacy point the the master branch and vice versa (deprecate legacy branch #397)

@130s, @miguelprada: What do you think?

Features of the new version:

• Support for colcon (melodic and newer) AND catkin_tools (ROS1 only)
• Support for ROS1 and ROS2 (ROS2 support  #333)
• Support for vcstool (vcstools support #357) and flexible confguration (UPSTREAM_WORKSPACE syntax #147)
• Strict separation of upstream, target and downstream packages (Upstream/downstream logic design #106)
• Refactored workspace management
• Uses official ROS docker images per default
• Hook script system (Install additional package before 'wstool update' #272)
• Source folder is writable (Read only file system ? #210)
• Prerelease tests for ROS1 and ROS2 with support for upstream and downstream packages
• ABI check

Dropped featrues:

• Support for Hydro
• Support for devel space
• enforcing of 4 build jobs by defaults
• testing installed *.test files (Remove testing installed test files? #252)

[130s]

Looks great. Thanks so much for taking an initiative on this. No strong opinions from me.

• I admit I have been ignorant about the changes made so far to industrial_ci. That said, is the plan to provide colcon support in addition to what it's been already available in a single branch in the future? If so that's even greater (and I think that's your plan. Just clarifying). And I assume it'll be backward compatible against non-colcon version of industrial_ci.
• Is dropping support for devel space well supported idea (I thought there was a discussion in industrial_ci when install became the default but I can't find it)?
  • I discussed with some senior level ros engineers and they didn't like the idea of always using install on developement or for even deployment in some R&D projects (though I insisted using install).

Lastly apology for ignorance...I assume there isn't a CI setup helper in ros2, something easier to setup like ros buildfarm, industrial_ci?

[mathias-luedtke]

That said, is the plan to provide colcon support in addition to what it's been already available in a single branch in the future?

No. The "colcon" branch contains a heavily refactored version of industriaLci to support catkin_tools, colcon, ROS1 and ROS2
And it should become the default after some reasonable transition time (3-6 months from now?).

And I assume it'll be backward compatible against non-colcon version of industrial_ci.

Not for all, but most users.
That's why there will be the "legacy" version as well.

The new version deafults to catkin_tools for ROS1 and colcon for ROS2.

Is dropping support for devel space well supported idea

colcon does not support the devel space anymore (even for ROS1).

I discussed with some senior level ros engineers and they didn't like the idea of always using install on developement or for even deployment in some R&D projects

They don't have to use it for development, but they should add install tags regardless.

[miguelprada]

Thanks @ipa-mdl for this thorough revision. I have yet to familiarize with the new codebase, but it does look much better organized than the old version.

The migration path you propose seems reasonable to me.

You can find some additional comments I made regarding the specifics of your colcon branch in #333 (comment).

[mathias-luedtke]

I just came to the conclusion that it might be easier to just drop the master branch.
For now, legacy would become the default branch and later we can switch to colcon.
This enable the user to select a version or just use the default, without the need of synchronization, which is quite tedious and error-prone.

If it is okay for you, I will make legay the default branch and delete master once the colcon is ready to be announced.
(or just rename master)

[gavanderhoorn]

Would it be an idea to keep master around for a bit, make it basically identical to legacy, but make the scripts in master print a (rather large and hard to miss) warning / notice that the master branch is going to be removed in the near future and that users should switch to either legacy or the new colcon version.

There is a non-zero chance that some setups clone a specific version and/or branch of industrial_ci and it would be unfortunate to break those setups without giving those users a chance to do something about that.

This would be something similar to a tick-tock release process.

[mathias-luedtke]

There is a non-zero chance that some setups clone a specific version and/or branch of industrial_ci and it would be unfortunate to break those setups without giving those users a chance to do something about that.

Good point!

but make the scripts in master print a (rather large and hard to miss) warning / notice that

Nobody reads warnings, especially if the build passes.
And this would require merges from legacy to update it (instead of pushes).

Would it be an idea to keep master around for a bit, make it basically identical to legacy

As a compromise we could keep master until we switch to the colcon branch as default.
This should get mentioned in the announcement as well.
I have updated the ToDos accordingly.

[mathias-luedtke]

@miguelprada, @130s: I'd like to switch the default branch to legacy now after #369 was merged, any objections?

[miguelprada]

Nothing other than @gavanderhoorn's suggestion of keeping master around for a bit (and possibly printing out a warning), just in case there's people cloning with -b master.

[mathias-luedtke]

I announced the transition to the legacy branch, but did not mention the new branch name.

colcon might be misleading, any better ideas?

1 . We could stick with colcon for now and move it to master later (this might break existing set-ups, requires early adopters to migrate later).
2. We come up with a more generic term (e.g., nextgen) and treat this as a master/stable.
3. Or we establish a develop branch for colcon and further developments (#129) and sync it over to master eventually (collisions are less likely, early adopters might want to migrate to master later)
4. Like 3., but with stable as the master/default branch (no collisions)

[miguelprada]

This search is far from perfect (it possibly searches only default branches), but suggests not many people are cloning master branch explicitly.

Is it so crazy to merge #361 into master and keep legacy as default for a while?

• It shouldn't break most setups (yet, until switching the default)
• It sends a clear signal that master (a.k.a. colcon) should be the go-to branch
• It's consistent with your announcement, which recommends pointing to legacy or default while warning that the latter will be updated

I know I'm contradicting myself here. But most problems are likely to happen when switching the default branch anyways. And the benefits of introducing another named branch may not be worth the added confusion.

[mathias-luedtke]

Is it so crazy to merge #361 into master and keep legacy as default for a while?

No, I agree that it would be the simplest solution - easy to understand for users and easy to maintain.
@130s: Do you agree?

[mathias-luedtke]

@130s: friendly ping

[130s]

1 . We could stick with colcon for now and move it to master later (this might break existing set-ups, requires early adopters to migrate later).
2. We come up with a more generic term (e.g., nextgen) and treat this as a master/stable.
3. Or we establish a develop branch for colcon and further developments (#129) and sync it over to master eventually (collisions are less likely, early adopters might want to migrate to master later)
4. Like 3., but with stable as the master/default branch (no collisions)

I like 3 as it might align the best with popular branching models like git flow and its many variants. That way, we'd maintain develop (bleeding edge contains colcon support) and master (released, more stable).

As we cannot rule out the chance of having users who specifically clone master (*1), I'd like to at least give notice and ample time for migration.

Once the current major development settles down, we might think about merging develop into master and/or we can eventually remove legacy (and announce again). This needs another discussion I think so I won't go further on it.

*1 #358 (comment)

This search is far from perfect (it possibly searches only default branches), but suggests not many people are cloning master branch explicitly.

Interesting! I have to say though that I assume a good number user repos are private (good thing for us :)

[mathias-luedtke]

I like 3 as it might align the best with popular branching models like git flow and its many variants. That way, we'd maintain develop (bleeding edge contains colcon support) and master (released, more stable).

While I would use this for regular software packages, this seems to be not applicable to industrial_ci, because we don't bundle releases, but it gets cloned directly.

Once the current major development settles down, we might think about merging develop into master

We cannot let development settle down, we need people to test this ASAP.
I am already holding back new features and improvements for a month now..

As we cannot rule out the chance of having users who specifically clone master (*1), I'd like to at least give notice and ample time for migration.

It was announced more than 3 weeks ago.
There is a big chance that the new version works out-of-the-box.

we can eventually remove legacy

No, the legacy branch will be kept in the foreseeable future. That was the point of introducing it.

[mathias-luedtke]

I'd like to merge the colcon-ready version into master. I am certain that it is mature enough.
It would be great if we could (crowd-)test it during WRID19 (#390) and at the same time update/polish the migration guide and the new documentation. At the end of the day (US time?), it should get merged.

PS: The legacy branch will stay the default for some time, but it will get updated to point to the new version.

[130s]

TLDR Got it. I missed that the discourse announcement explained branch switch. Comments follow for further discussion but I'm fine with merging #361

[130s]

I like 3 as it might align the best with popular branching models like git flow and its many variants. That way, we'd maintain develop (bleeding edge contains colcon support) and master (released, more stable).

While I would use this for regular software packages, this seems to be applicable to industrial_ci, because we don't bundle releases, but it gets cloned directly.

You mean "not" applicable?

because we don't bundle releases

I'm not sure what this means.

What I meant was using more common branching will hopefully allow people to grasp the purpose of multiple branches we would have without us having to explain.

Once the current major development settles down, we might think about merging develop into master

We cannot let development settle down, we need people to test this ASAP.

I'm a bit confused. Are we planning to keep adding disruptive changes (by "development settle down" I meant no more such changes) like #361?

[mathias-luedtke]

You mean "not" applicapble?

Yep 👍 , updated my post

because we don't bundle releases
I'm not sure what this means.

For other ROS packages you have a stable release version, and the devel version.
So it is easy for the users to choose.
For industrial_ci I would consider the default branch to be our stable version.

What I meant was using more common branching will hopefully allow people to grasp the purpose of multiple branches we would have without us having to explain.

We will will have legacy and master.
I don't see the need for a develop branch right now.
It would just increase the maintenance effort (cherry picking hot-fixes for 3rd party changes).

I'm a bit confused. Are we planning to keep adding disruptive changes (by "development settle down" I meant no more such changes) like #361?

#361 is a disruptive change, but I limited changes to the scope of making colcon work (and maintainable).
I'd like to refactor even more "under the hood".

[130s]

Ok, then I agree with moving forward with #358 (comment)

merge #361 into master and keep legacy as default for a while?