Proposal: Channels -- parallel development labels

[murrayh]

Channels are a release pattern where multiple versions of the same software are developed in parallel.

Consider the following use case.

1. I have a config package managed using SemVer.
2. I have an application that uses the config package.
3. My application's output includes the version of the config package used (for traceability).
4. A small subset of users require an incompatible change made to the config package.
5. The config package gets branched, and both branches need to be maintained in parallel (i.e. two release channels).
6. Both config package channels receive the same patches. Changes to the main channel are always trivially merged into the specialized channel. A "release" consists of releasing both channels, which are consistently the same except for the originating incompatible change.
7. The vast majority of users should not know that the branch exists. From their perspective, there is only one true config package that must be used.

At this point we have two options:

1. We need to include another variable in our application's output (that describes which channel the config package came from); or
2. We can use a field in the version to differentiate between the two branches.

I prefer the later option because:

• The application does not need to be modified with a somewhat ugly change (output the channel if the channel is not the "main" one).
• By extension, all application's facing the same problem do not need to be modified. A class of defects is eliminated.
• The channels are strongly related -- it feels more intuitive to differentiate them via versions than package id/names.
• Versions that support channels are flexible -- I have described one particular kind of parallel development, but there are heaps of other kinds.

Note that I agree that applications should not use a channels in the versions (i.e. I agree with the approach taken for the Firefox suite of applications, which uses package id/names to differentiate channels). However, I prefer dependencies (e.g. libraries and services) to use channels in their versions because it is a simpler for the host applications who only care about (i) the API being stable and (ii) recording the minimum set of traceability info.

Channels by their nature are incompatible with each other. So I propose that:

• We add an optional field to the SemVer. MAJOR.MINOR.PATCH.CHANNEL-PRERELEASE+BUILDMETADATA
• CHANNEL can consist of [a-zA-Z0-9]
• SemVer's that have differing CHANNELs cannot be compared (illegal operation)

[FichteFoll]

Highly related: #265.

[ameenross]

Sounds like a useful addition, applicable not merely to the mentioned use-case. Some thought has to go into the positioning though. Right now I'd think prefixing the lot makes more sense, i.e.

• d8.1.3.0
• instead of 1.3.0.d8

[ameenross]

Actually, it might be safer to use another delimiter as well, like a dash (dash is allowed in the PATCH part) an at, resulting in 1.3.0-beta1@d8 or d8@1.3.0-beta1

[ameenross]

Also, there's a difference in flavor between the mentioned use-case and Drupal's case in #265.

• 1.3.0@channel1 and 1.3.0@channel2 are essentially the same version and have feature parity and a compatible API.
• 1.3.0@d8 and 1.3.0@d9 may or may not be essentially the same version, the API of these package versions may or may not be compatible.

Could you ensure that 1.3.x@channel1 and 1.3.x@channel2 will still be the same version by the time 1.3.29 is released? What if, say, a security release is issued that only affects that certain subset of users. Would you create a dummy release for the unaffected channel? I'm asking because I wonder whether it matters at all if the versions in different channels are compatible or not. Those versions cannot be compared in the first place.

[timmillwood]

The issue I have with using the @ symbol relates to how package managers currently use it. Coming from the PHP world, Composer uses the @ symbol to denote stability. For example "1.0.*@dev" can be used to get the dev version or "1.0.*@RC" for the release candidate.

So if we were using @ for the channel too, how would we get the dev release of a channel? 1.0.*@dev@channel1? I don't think that'd work.

I'd rather see support for prepending a "channel" as mentioned in #265. The proposal there is to just use a numeric version rather than [a-zA-Z0-9] as suggested here. This allows for instant compatibility with many package managers but the "channel" would replace the major version, major would replace minor, and minor would replace patch. This would cause some issue, and possible rethinking about usage, but all compatible.

[ameenross]

Quoting @haacked (a main contributor to Semver):

I have to admit, I think the 4 segment version would be confusing because some systems already have 4 segment versions (example, .NET) and the first segment is Major version in all those cases. Perhaps a new delimiter would help?

We should probably stick to a delimiter, though @ may not be the right character to use. There's a whole range of other characters above my number keys that are potential candidates though. Anything package-manager safe in ~!@#$%^&? Below are my findings so far.

• ~ Used in Composer as version constraint.
• ! Not used in Semver, Composer.
• @ Used in Composer as stability flag.
• # Used in Composer to refer to a specific commit hash.
• $ Not used in Semver, Composer.
• % Not used in Semver, Composer.
• ^ Used in Composer as version constraint.
• & Not used in Semver, Composer.

[eelkeblok]

Additionally, I think there would be problems with "channels" that are numeric and dot-separated when adding channels to an existing package (as with the scenario in the issue summary); I don't see the versioning working out in any way that is not at least confusing. Personally, I very much like the semantic nature of channels that are more than just numeric (I would vote for an "alphanumeric identifier", in terms of the spec at https://github.com/mojombo/semver/blob/master/semver.md).

To address the concerns raised by @ameenross in #264 (comment), I would say the channel needs to be prefix. I do believe that in the generic case, there should be no assumptions about how similar the same version identifier in different channels is. Since semver is basically built as going from most significant to least significant, left to right, I'd say the channel needs to be the very first (although optional) part of the semver identifier.

[ilanbiala]

Any progress on this spec? I'm looking for something like this as well to version a package based on the version of another package and this would help.

[rugk]

Reply to @ameenross in #264 (comment)

There's a whole range of other characters above my number keys that are potential candidates though

What about _? Is not that possible too?

[a-teammate]

The issue I have with using the @ symbol relates to how package managers currently use it. Coming from the PHP world, Composer uses the @ symbol to denote stability. For example "1.0.@dev" can be used to get the dev version or "1.0.@rc" for the release candidate.

So if we were using @ for the channel too, how would we get the dev release of a channel? 1.0.*@dev@channel1? I don't think that'd work.

That is not an issue, that is exactly the same usage.
think of it as 1.0.*@devchannel.

imo channels are the most needed thing in semver (if you do not define it, people define it themselves, see composer)

although i understand that versions should be descending in significance, we should take into account how channels are currently already used in most reallife software projects: with a postfix @
Probably they all (another example: the conan package manager) decided this way since you say "at channel x" in natural language

[jwdonahue]

@murrayh, you can encode build branch names in the build metadata or the package name. Also, this seems like a dev-ops tooling issue that should be out of scope for SemVer. Look at it this way, PackageName.1.0.1 is not a SemVer string, but it contains a SemVer substring and that can be easily extracted as a SemVer string and parsed accordingly. The same can be applied to branches or channels. I would leave it up the tool chain devs how to solve this problem.

Unless you have further questions/comments or intend to submit a PR referencing this issue, please close it at your earliest possible convenience.

[murrayh]

I'm happy to write a PR, but would I be wasting my time? I can see that some users want this kind of addition, but do the maintainers?

[cweagans]

I would leave it up the tool chain devs how to solve this problem.

Note that this is the status quo, and it's very annoying. Every tool chain has their own idea about how to solve the problem. We need to get everyone to agree on something so that semver is semver no matter what toolchain or language you're using.

Unless you have further questions/comments or intend to submit a PR referencing this issue, please close it at your earliest possible convenience.

I strongly disagree with this position. The issue is an open todo item that hasn't been fixed. There's a lot of interest in submitting a PR, but nobody wants to go through the trouble if it won't be accepted. We're looking for a preliminary "yes, this is something that would be of interest given an appropriately written standard for it" from someone with decision-making power for semver. That's the only blocker right now.

[jwdonahue]

@cweagans

We need to get everyone to agree on something so that semver is semver no matter what toolchain or language you're using

1. Everyone will never agree to this or any other proposal.
2. Who's "we"?

@murrayh, I can't speak for the maintainers, but I think I have some insights into at least some of the feed (channel) implementers that also cover SemVer. What about other versioning schemes? If SemVer is going to force my hand in the feed design space such that other popular versioning schemes are more difficult to cover, why should I bother with SemVer? What about all my feeds that have nothing at all to do with versioned content? What if I have a million subscribers who never heard of SemVer and just a few hundred ask me to support it? Your proposal implies that I should only support it a certain way. I would personally oppose such an initiative.

Sure, you can propose a PR, but if you are going to wait another two years to champion the idea, then you will probably never see it accepted anyway. In the mean time, we currently have over 100 open issues and only a few active contributors willing to work on SemVer and they need to prioritize. The best way to do that at this point is to punt/close all issues that have little chance of success in the next few years and then focus on what remains.

[a-teammate]

@jeme:
It's about tool support.
There exists an ecosystem around semversion which implement some common operations based on the clearly specified input.

You are right: the current state is that semversion only works linearly.
The question is whether the parallel aspect of development should get covered by a clear specification or not.
The need is (undoubtedly) very big.
Any more or less agile dev team actually with different branches could use it or uses a similar self-made solution.

So why should this feature get covered by a completely separate spec (like "concurrent versioning" or sth) if that spec would be a superset of semversion?
And if that feature is widely used but everyone does it their own way?

The only answer I can think of currently is that the needs of all the people who do invent that are
unique in some regards. So different means could be appropriate.
But are they?

[FichteFoll]

A spec does not need to be a superset of semver if it just references it. You can build specs based on other specs and say "this is the channel declaration, now follows a semver as per https://semver.org/ and finally you can choose between dogs and chickens".

While yes, it does indeed include the semver spec, it does so in a modular manner, which you should be used to from programming in general. Keeping things modular allows for better separation of concern as opposed to doing everything in a single spec.

[cweagans]

Everyone will never agree to this or any other proposal.

That's a bit nihilistic. We should just do nothing because there will always be someone that disagrees? No thanks.

Who's "we"?

Us. The people here in this issue. The people that evidently care about this functionality. I thought that was pretty clear.

If SemVer is going to force my hand in the feed design space such that other popular versioning schemes are more difficult to cover, why should I bother with SemVer?

My understanding is that channels would be opt-in. If you want to continue using semver as you are today, you should be able to. Channels are for the people/projects/processes where channels would actually solve a problem.

[jwdonahue]

Channels are for the people/projects/processes where channels would actually solve a problem

Who's channels? Visual Studio Online or somebody else's?

Us. The people here in this issue.

Wonderful, gather your contributors, write and publish a spec that references SemVer. It might even turn out worth cross-referencing back from SemVer to whatever you write.

@murrayh, it's been more than two years and no progress has been made on this issue, nor does it seem likely. Please close this issue at your earliest possible convenience.

[ameenross]

@jwdonahue You're not very convincing in your reasoning. You just seem to discount the arguments in favor with some rhetoric and fallacies. And that wouldn't necessarily be a problem, but you don't appear to stand out from the crowd; you aren't a core contributor to Semver, you don't even have any meaningful reputation to speak of on Github. You're (at best) a peer of the rest of us. So please come up with proper arguments.

I don't disagree that parallel development might not be a great fit for Semver. It's just that, the consequence of not covering it are that developers can't have one repository for their project anymore: they need one per parallel release train.

Whether parallel development is covered in Semver or elsewhere is not necessarily relevant. But I think most of us think it's most convenient if it's in Semver. Take for example the fact that Semver reserves some characters to have some special meaning in a version string, and dependency managers (like Composer) introduce their own reserved characters. Each dependency manager re-invents the wheel in that regard, and various standards would have the exact same effect and (probably) not be interoperable. Obligatory xkcd: https://xkcd.com/927/

[jwdonahue]

@ameenross

In practice, I have seen parallel development tracks handled using various combinations of build metadata, prerelease tags and package name prefixes/postfixes. None of which required any changes to the SemVer spec. The fact is that different organizations use different development/release workflows and those workflow choices are often reflected in their naming and versioning schemes. I have personally applied multiple patterns over the years that were adapted to specific needs, some of which had embedded SemVer strings in them.

The current proposal includes a breaking change that would require a major version bump of the SemVer spec. The proposed usage pattern can be implemented without any need to change SemVer, just prefix/postfix whatever identifiers you have need of, to the SemVer string. Given that there are viable workarounds and this issue has languished for two years, I think it should be closed.

I can find no evidence that the maintainers or a majority of the interested parties here have much interest in making another major version bump of the Spec, but I've only reviewed about 150 open issues (some have since been closed) over the past month and I have not yet made the deep dive into the closed issues.

[murrayh]

Proposal not accepted, closed as requested. Thanks everybody for their consideration and contributions.

[a-teammate]

I have personally applied multiple patterns over the years that were adapted to specific needs, some of which had embedded SemVer strings in them.

Out of interest: could you maybe list some here if you remember them?

[jwdonahue]

A pattern that is easily adapted to file systems and online feeds follows something like:

PackageName.BranchName.X.Y.Z-prerelase+meta
PackageName.BranchName.X.Y.Z+meta
PackageName/BranchName/X/Y/Z/-prerelease.meta
PackageName/BranchName/X/Y/Z/+meta
PackageName\BranchName\X\Y\Z-prerelease+meta
PackageName\BranchName\X\Y\Z+meta

In some environments, the branch name is effectively a team/org name and it sometimes makes sense to list it first. Also it can be difficult to use the plus symbol in some service URI's so it is common to translate those to some other symbol that feeds through and does not conflict with anything that is already in use. In real directory structures such as shared file systems, it occasionally makes sense to convert the dotted fields in the prerelease and meta tags to directory separators as well. Note that I have found it's best to always use some kind of distinguishing mark for the prerelease and meta tags.

Of course, PackageName.X.Y.Z... is reserved for "official" release channels.

I have also seen the following:

PackageName.X.Y.Z-feature_branch+meta
PackageName.X.Y.Z+meta
PackageName.X.Y.Z-prerelease+branch.meta
PackageName.X.Y.Z+branch.meta

Which patterns to use, depends a great deal on which dimension you intend to search for product, the search capabilities of any underlying package store and the query channel. Do you find your packages with an online search URI, ls/dir, SQL? Some other tool?