Introduce a release cycle

[salkinium]

xpcc does not have any releases, instead you use the latest changes on the develop branch.
This is not always desirable since develop is sometimes still unstable, despite our best effort.

I would like to discuss some options for introducing a release cycle.

[salkinium]

I'm uncertain what kind of release cycle would be most beneficial to our users.
xpcc has little to no roadmap and all work is done on a voluntary basis.
A release is not an insignificant amount of work: A changelog must be maintained, the commit to be released must be tested more thoroughly on all available hardware, fixes for the release must be backported or marked as known bugs.

One idea is to make one release per year. The release date would be around this time, after the Eurobot finals, but before the rules for the new season come out.
We would push the develop branch into master and add a changelog.
I'm not sure however, if one release per year is actually enough to keep up with the rapid development of xpcc sometimes. Maybe devs will just continue to track develop anyways.

Another way of dealing with such an organic process is to use Semantic Versioning.
However, this doesn't really work for a repository the size of xpcc.
If we do one breaking change in a small part of xpcc (like rewriting a Sensor driver), all of xpcc would require a Major version bump, which is technically correct, but messes with the "semantics" of the versioning.

To be honest, I think not doing a release and instead maintaining a changelog (see past "changelogs") is the best option for us. It is reasonably simple to do, and gives our users a better overview over what has changes.

What requirements and improvements do our contributers and users want to add to this?
cc @ekiwi @dergraaf @cajt @strongly-typed @daniel-k @Sh4rK

[Sh4rK]

I'm okay with not having releases at the moment, because xpcc is (still?) changing rapidly, so it wouldn't really work (unless there are more frequent releases). Most of the time the reason I had to update was because I needed something immediately to be able to continue (or I could maintain my patches locally, but that would be even worse).

One way that I think versioning and releases could work, is by breaking up xpcc into modules. This would solve the problem you stated, and would also make it possible to have a few really stable "core" modules, but also less stable 0.x.y, or even unversioned ones. Of course, this has to be implemented in a way that's easy to use (maybe it's possible to reuse an existing package manager?). The downside is that this takes a lot of developer time.

[ekiwi]

We at the Roboterclub just track upstream xpcc on the develop branch and complain when something breaks. I don't think that releases would help us.

The only thing I would think of that might help is if big changes would be tested for a little bit longer before merging the pull request. For example we disabled the TLSF allocator when it was made default as that was a rather quick decision and apparently we were right in doing to as there was a bug with it (#136)

[cajt]

The current approach is bit tricky for people who aren't actively involved in the development of xpcc, or regulars at the RCA meetings.
For those using xpcc currently results in, either trying to tracking the develop branch, or picking (more or less randomly) the latest revision and freezing it.
Both approaches may result in not so great experiences, and can randomly go bad sooner or later.

I'm not aware of the userbase of xpcc, but i'd expect that not all can be actively involved in its development. What might also be of interest: Does everybody use all the parts and platforms of the system at once?

[salkinium]

Thanks for contributing!
I thought about this and I think xpcc releases are not a good fit with how it is developed.

However, from this discussion I can see three major requirements:

1. Better communication of changes to xpcc.
2. Better testing of changes.
3. Modularization of xpcc.

Let's discuss these in more detail.

1. Better communication of changes to xpcc

I will be adding a detailed Changelog to xpcc. It will group several commits or PRs into a "feature" and describe and grade it's impact on xpcc. The changelog is continuous with the finishing date as "version" and recent changes added on top. Note that small fixes and features are not placed into the changelog.

The idea is to give a good overview over the changes so that developers that are not actively following xpcc development along get an easy understanding of how the changes impact their code.

Example format:

2016-07-20: Add Blue Pill board and examples

Add board support for STM32F103C8T6 Minimum System Development, aka. "Blue Pill", and examples.

PR #154 –> 7ab0132 with low impact.
Tested in hardware by @strongly-typed.

2016-07-09: Initialize display for STM32F469I-DISCO

Adds initialization code and examples for the DSI, display and touchscreen controller found on the board.

PR #155 –> a6519c3 with medium impact on STM32F469I-DISCO targets.
Tested in hardware by @salkinium.

2. Better testing of changes

@strongly-typed is consistently pushing for better testing.
He has kindly written up our testing strategy and introduced several new testing features.
For example, we can now also run the unit tests directly on the device. However, we have no hardware setup for this, so TravisCI only compiles it, but does not run it on the device (doh!).

Furthermore we lack system level tests that assert more complex properties, such as system clock frequency, memory management, IO operations. However, @strongly-typed has built a very interesting proof-of-concept validator using a cheap logic sampler and SigRok analyzer via the hwut concept.

Keep in mind, that the priority of CI in hardware is not very high though, because it is very complicated.

3. Modularization of xpcc

The elephant in the room.

Ignoring our special HAL generation (the stuff in xpcc/architecture/platform/driver and supporting build tools), the modularization of xpcc should be relatively straight forward.
While our SCons build system has some special things (like test folders and some build.cfg files) I think they can be reasonably easily mapped to another build system.

I originally wanted to use Yotta, but it is being re-evaluated and hasn't seen much development recently, so I recommend we look at Conan.
Conan comes with the concept of options, which fits very nicely with some of the customization we need to do in xpcc.

As far as I understand, Conan is not yet ready to deal with microcontroller targets. There has been a discussion on microcontroller targets started by @astralien3000 (who's Aversive++ framework already uses Conan), to which I contributed too.
I'm not on top of the latest Conan development, so there would need to be some research on how to do this correctly.

Most notably, Conan does not have a Software Construct integration, and I'm leaning towards writing the new build system in something a little more standard and more actively developed like CMake. SCons is not very accessible in terms of documentation and community.

The HAL generation part will have to be custom made anyways, and I suggest we use @dergraaf's library builder, which he wrote for this purpose.

HOWEVER, I'm not sure modularization is really a good idea. Sure it allows proper semantic versioning of correctly scoped modules, but it also dramatically increases the work to keep interfaces aligned across different modules. I'm not certain we can spare the effort to modularize xpcc, especially because it is not widely used. Furthermore there are only a few parts of xpcc that are not dependent on our HAL (like math) and so the modularization is not really that useful.

To be clear: I won't be putting any work in the modularization of xpcc. I'm more interested in cleaning up our HAL generation capabilities (using the library builder) and making this technology more accessible to outside contributions (perhaps in a separate repo). It is the distinguishing feature of xpcc after all and is, as far as I know, the only HAL generator of its kind.

[astralien3000]

Hello !

I Just want to answer about the modularization, even if it is not the high priority. Indeed, the official conan is not ready for microcontroller targets, but I have a working version that can handle some MCU as a proof of concept. Actually adding targets to Conan is not difficult, since it let the package handle the build chain. The current blocking point is to be able to add some targets easily with a configuration file. I have not worked on it for 1 month because I am on other projects, but I think it is not so hard.

But you talked about some hardware independent modules (math ?) that could be easily modularized. I may be interested in helping to make some modules. Or even doing it alone ?

[salkinium]

But you talked about some hardware independent modules (math ?) that could be easily modularized. I may be interested in helping to make some modules. Or even doing it alone ?

Hm, math has dependencies on at least these three xpcc core headers:

• xpcc/architecture/detect.hpp (bit operations, endianness)
• xpcc/architecture/utils.hpp (compiler tools)
• xpcc/architecture/driver/accessor.hpp (for reading from flash)

Though I'm not entirely sure how useful a free floating math module is with respect to demonstrating Conan's validity for embedded.

[salkinium]

A changelog has been added and we're doing quarterly releases now.

Regarding modularization: we're hard at work on this over at @modm-io.