Allow infinite (n) optional version segments

[JamesMGreene]

The Problem

Among the other reasons mentioned in various issues both opened and closed in this repo (#213, #241, #200, etc.), being confined to only a 3-segment version number for public releases is especially bothersome for downstream wrappers (e.g. bootstrap-sass, the Node.js module wrapper for PhantomJS, the Node.js wrapper for Adobe/Apache Flex SDK, etc.) that would benefit greatly from being able to maintain some semblance of being aligned with the upstream library's version number.

For example, with JamesMGreene/node-flex-sdk, I've wrapped the Adobe/Apache Flex SDK for consumption (i.e. build automation) in Node.js. The actual Adobe/Apache Flex SDK releases have a 3-segment version number, e.g. 3.0.1. Ideally, my corresponding version numbers would be identical, plus a wrapper-specific post-build number, so that consumers don't need to learn much about the wrapper itself in order to install the correctly desired version of the upstream library. I was excited when SemVer 2.0.0-rc1 was posted because it allowed for this exact desire using + to indicate the start of a post-build number... but then SemVer 2.0.0-rc2/2.0.0 changed + to indicate the start of "build metadata" and suggested that it "should" be ignored when determining version precedence. So... back to square one! ⬛

Today, I am choosing to use the pre-release version part for tracking my wrapper's post-build number but, alas, that is incredibly _un_semantic, as it would suggest that my wrapper is not on par with the associated upstream version. The result is that someone trying to install 4.6.0 of my Node.js module will end up not getting anything installed at all when they run something like npm install flex-sdk@4.6.0. They must instead run something less intuitive like npm install flex-sdk@^4.6.0-0.

And please don't just shout "that's a tooling problem!" at this point... NPM is just trying its best to actually adhere to supporting the SemVer 2.0.0 spec correctly for its package ecosystem.

Proposed Solution

After looking at #213, which proposes adding a 4th segment to the "core version", my big fear is that we might add a 4th segment in the next major version of the spec (which I am mostly in favor of) but then that would become the new standard for some upstream libraries, and subsequent the downstream wrappers would need a 5th segment, etc. etc. (or a proper post-build versioning scheme that MUST be considered when determining version precedence).

In light of that fear, my more generic proposal is to allow an infinite (n) optional .{number} segments [after the required 3-segment "core" version] for additional version comparison. Associated tools would just need to iterate through them, which they are already required to do in order to support pre-release version segments correctly. The only big difference is that the "core" version requires (a) [a minimum of] 3 segments, and (b) that all of its segments be strictly numerical and without any leading zeroes.

Supporting this idea in the spec would require only a few changes, though I'd gladly welcome help on coming up with the correct wording (changes in bold):

• Update to the final sentence of the "Summary" section:
  • "Additional labels for complex sub-versioning, pre-release**,** and build metadata are available as extensions to the MAJOR.MINOR.PATCH format."
• To be inserted before the current "9" (regarding pre-release labels):
  • Complex sub-versioning MAY be denoted by appending a dot and a series of dot separated identifiers immediately following the patch version. Each additional version identifier MUST be non-negative integers, and MUST NOT contain leading zeroes. Each element MUST increase numerically. Complex sub-versioning identifiers MUST all be reset to 0 OR DROPPED when any of the major, minor, and patch version segments are incremented.
• Amendments to the current "11"/future "12" (regarding determining version precedence):
  • Precedence MUST be calculated by separating the version into major, minor, patch, complex sub-versioning, and pre-release identifiers in that order (Build metadata does not figure into precedence). Precedence is determined by the first difference when comparing each of these identifiers from left to right as follows: Major, minor, and patch versions identifiers are always compared numerically.
  • When major, minor, and patch identifiers are equal, any complex sub-versioning identifiers present MUST be considered. Any complex sub-versioning identifiers present are always compared numerically; if one version includes more sub-complex versioning identifiers than the other version, the corresponding missing identifier MUST be treated as 0. Examples: 1.0.0 == 1.0.0.0 == 1.0.0.0.0 < 1.0.0.0.1 < 1.0.0.1 == 1.0.0.1.0 < 1.0.1 == 1.0.1.0.0 < 1.1.0 < 2.0.0, etc.
  • When major, minor**, patch, and any optional complex sub-versioning identifiers** are equal, a pre-release version has lower precedence than a normal version. Examples: 1.0.0-alpha < 1.0.0**, 1.0.0.0-alpha < 1.0.0.0, 1.0.0.0-alpha < 1.0.0, 1.0.0-alpha < 1.0.0.0, etc.**
  • A larger set of pre-release fields identifiers has a higher precedence than a smaller set, if all of the preceding identifiers are equal UNLESS all of the following identifiers are specifically equal to 0. If both versions have pre-release sets BUT one version's pre-release set includes more identifiers than the other version's pre-release set, then any corresponding missing identifiers MUST be treated as 0. Example: 1.0.0-alpha == 1.0.0-alpha.0 < 1.0.0-alpha.1 < 1.0.0-alpha.beta < 1.0.0-beta < 1.0.0-beta.2 < 1.0.0-beta.11 < 1.0.0-rc.1** == 1.0.0-rc1.0** < 1.0.0.

Example Partial Implementation

I wrote an example partial implementation of this idea in JavaScript (modeled as a subset of the npm/node-semver API) with a fair number of particularly pertinent version precedence unit tests, you can take a peek at it here:
    http://jsbin.com/sekuxo/5/edit

It's pretty simple, IMHO.

Feedback

Feedback, please!

[JamesMGreene]

@isaacs: Would love your thoughts and feedback, too.

[FichteFoll]

Personally, I'm much more a fan of using a different syntax for these cases so that we wouldn't end up in a world with 1.0.0 1.0.0.0 and 1.0.0.0.0.0.0 and a soft break after three numbers.

Instead of just appending numbers I'd rather suggest to introduce a post-release section that hi-jacks the current + sign to denote a version higher than the base and behaves similar to the --denoted pre-releases, except for the other direction. Metadata, which is currently defined with the + sign, could then be defined with ~, which I find more fitting thematically. What exactly you'd do with the post-release should then be in your own hands since it serves as a general purpose thing. It should be avoided to postpone not increasing the patch for bugfix releases or similar though because that would be an abuse.

The use cases I can imagine:

1. Versioning of regular debug/dev builds or releases prior to knowing which the next version will be (alas which identifier should be increased, patch, minor or major) (Nightly builds not supported #200)
2. For when you base your version on an external project and need another version identifier for your own release. This one has problems however since you might change the "public API" of your library and just increase a post-release identifier instead of the major one since it is based on the external lib.(Dealing with forks #17)

Because there are many things to consider with the second problem (e.g. integrate a lib that depends on another lib already and bases its version on that) I have not yet been able to find a good solution for this problem as post-releases will remain abusable in that regard. The same thing applies to your suggestion however.
In an ideal world, people would use metadata for the version they based their lib on and a package manager would be able to resolve a query that only finds releases with that metadata (see #205)

Examples:
0.9.0 < 0.9.1 < 0.9.1+dev.234 < 1.0.0-alpha.1 < 1.0.0-alpha.1+dev.456 < 1.0.0-alpha.1+dev.457 < 1.0.0~x86 = 1.0.0~x64 < 1.0.0+1.2.1
1.2.1~3.2.3 < 1.2.2~3.2.3 (3.2.3 is the version of the dependency), and parallel to that 1.2.1~3.4.0 < 1.3.0~3.4.0

Note that this is a breaking change and would require SemVer 3.0.0.

[isaacs]

I struggle to express just how strongly opposed to this I am.

It is completely unclear whether 1.0.0 is greater or less than or equal to 1.0.0.0. This change will make everything more challenging, both for humans and for machines.

What you are describing is not SemVer, it's a completely different thing. Use the prerelease and/or build tagging features. If the dependency version isn't something that you're going to select on, then use build metadata, because that's what that is for, and does not require Yet Another magic character. Furthermore, how would one express, as a dependency, that you require version 1.2.x of the "public API", but version 2.3.x of the "internal dependency"? This is the inappropriate intimacy antipattern all over the place.

Let's back up several steps here and identify the problem that we're trying to solve. Then, let's evaluate whether we actually need to solve that problem, or if in fact we need to: (a) evangelize the use of SemVer, (b) factor dependencies in a more modular fashion, (c) tie public APIs less tightly to internal dependencies, (d) become more comfortable with the idea that SemVer only expresses semantics about public API and not about internal implementation details, or (e) do something else entirely.

[EddieGarmon]

@isaacs exactly.

---

From: isaacsmailto:notifications@github.com
Sent: ‎1/‎21/‎2015 6:30 PM
To: mojombo/semvermailto:semver@noreply.github.com
Subject: Re: [semver] Allow infinite (n) optional version segments (#242)

I struggle to express just how strongly opposed to this I am.

It is completely unclear whether 1.0.0 is greater or less than or equal to 1.0.0.0. This change will make everything more challenging, both for humans and for machines.

What you are describing is not SemVer, it's a completely different thing. Use the prerelease and/or build tagging features. If the dependency version isn't something that you're going to select on, then use build metadata, because that's what that is for, and does not require Yet Another magic character. Furthermore, how would one express, as a dependency, that you require version 1.2.x of the "public API", but version 2.3.x of the "internal dependency"? This is the inappropriate intimacy antipattern all over the place.

Let's back up several steps here and identify the problem that we're trying to solve. Then, let's evaluate whether we actually need to solve that problem, or if in fact we need to: (a) evangelize the use of SemVer, (b) factor dependencies in a more modular fashion, (c) tie public APIs less tightly to internal dependencies, (d) become more comfortable with the idea that SemVer only expresses semantics about public API and not about internal implementation details, or (e) do something else entirely.

---

Reply to this email directly or view it on GitHub:
#242 (comment)

[JamesMGreene]

Some great points among those (though I disagree with a few), guys. Thanks!

To clarify my particular use case:

• My Node.js module that provides a thin wrapper around an external library (tool chain, really)... not even a wrapper around the API, just a wrapper around getting the dependency installed (with some normalization during prepublish) and a tiny little API to basically just provide the OS-dependent file paths when it (the module) is instantiated (required for the first time).
• It is not really accurate to call that external library an "internal dependency", though, as its various versions offer distinct capabilities and compilation targets that cannot always be achieved by using newer versions.
• Consumers will want to install a version of it based on the corresponding external library's version number, not based on an arbitrary version number supplied for my wrapper package.
• Likewise, my wrapper needs to support multiple versions of the upstream library simultaneously.

So, thinking through this my needs further, I think I must conclude that SemVer--in pretty much any theoretical future state--is not really capable of supporting this situation because I essentially need to represent 2 public API versions simultaneously.

As such, I believe my only remaining option as far as SemVer + the NPM tooling goes is to publish a different module for each external version (e.g. flex-sdk-4.6.0) and retain use of the actual wrapper's version number for proper SemVer usage. So, rather than running npm install flex-sdk@^4.6.0-0, they would instead run npm install flex-sdk-4.6.0@*.

Disappointing... but I feel like I have a better understanding of the situation's depth after talking through it, so thanks for the outlet.

[isaacs]

@JamesMGreene Certainly putting the version number in the name is an option. That's why we have, for example, https://www.npmjs.com/package/unicode-3.0.0 which is version 0.1.5.

On the other hand, you could just make your wrapper even lighter, and just go ahead and tie your version number to the flex-sdk's version. But it means that you can never update the wrapper code.

You can also put maybe just the major version or major.minor in the package name, and say that you accept patch updates to the internal dep. Since flex is likely to update only very rarely, maybe that's fine.

There's a lot of ways around this. But SemVer on your module version should refer to its external API surface, and nothing else.

[JamesMGreene]

On the other hand, you could just make your wrapper even lighter, and just go ahead and tie your version number to the flex-sdk's version. But it means that you can never update the wrapper code.

That's exactly how I started... and then I needed to update my wrapper code to normalize inconsistent line endings that snuck into a few versions (apparently Adobe sometimes packaged it on Windows, and sometimes on Linux/Mac (or always on Windows and forgot to normalize it sometimes)). And so, I was quickly put back a few steps, especially after discovering that NPM no longer allow republishing anymore (not exactly fortuitous timing 😢), and then ended up resorting to the pre-release versioning strategy.

That worked alright until NPM moved to its node-semver version 4 range semantics that handicapped consumers of modules using a pre-release versioning strategy like mine. I understand the rationale behind the change but that was a major bummer for me.

[JamesMGreene]

Instead of just appending numbers I'd rather suggest to introduce a post-release section that hi-jacks the current + sign to denote a version higher than the base and behaves similar to the --denoted pre-releases, except for the other direction.

I would have rather seen that in SemVer 2.0.0 as well, and I was very excited to see that functionality in SemVer 2.0.0-rc1... and so I was really, really sad when the + sign was excluded from determining version precedence in SemVer 2.0.0-rc2 and 2.0.0.

At this point, I would worry that changing the meaning of the + sign would result in major backward compatibility issues for package managers like NPM et al, (right, @isaacs / @npm? or am I off-base there?) so even though that is the appropriate symbol in my mind, I would probably encourage the use of a different symbol to not break the hell out of the world of dependency management ecosystems. 😢

[JamesMGreene]

And, even if we did change the meaning of the + sign, I would question its semantic correctness a bit.

So, let's say I have versions A.B.C+x.y.z where A.B.C is my module's public API version and x.y.z is the external dependency's public API version.

Now, going a little outside of the scope of SemVer proper and into the world of querying/selecting/matching/etc. version ranges, I would then expect my consumers would want to install such a versioned module by running npm install flex-sdk@^1.0.0+4.6.0.

Similarly outside of the scope of SemVer proper, the NPM registry would need to start accepting modules with + trailers and either (a) comparing their values (<, <=, >, >=, =, !=, etc.), or (b) at least comparing their values for strict equality (=, !=). This would be critical for being able to publish and consume versions where only the external dependency is different, e.g. 1.0.0+4.6.0, 1.0.0+4.6.1, etc.

[FichteFoll]

@JamesMGreene For your use case I actually intended the examples 1.2.1~3.2.3 < 1.2.2~3.2.3, where 3.2.3 is the version of the dependency. Since ~ would denote the metadata thing, it would potentially be searchabale using metadata selectors that I abstracted in #205 (using the current layout with + for metadata). Dependency would then be denoted with 1.x.x ~4.6.0, which finds the highest release with major 1 and metadata 4.6.0 (so, matching the 4.6.0 dependency). Using the current + this should already be possible right now since the symbol changes would only be necessary for the post-release scenario that I described earlier in my first point. However, I can see how npm would have issues with the space in that version selector, so there may be technical issues with the exact implementation.

This is probably a bit confusing though so if you hit any problems, feel free to ask.

[EddieGarmon]

SemVer is there to express the version of your code, and your code alone. It is the job of the package managers to manage dependent packages. This is the use case for build metadata, so that you can easily and fully express with words these ideas, without making the version precedence rules insane.

You could easily introduce a breaking change in your wrapper API, and as a consumer I want to know this. Also, I'm not going to choose a wrapper because I am already using some specific framework; for me it works the other way around.

[JamesMGreene]

@FichteFoll: OK, interesting, that gives me a better feel for what you're talking about. Using ~ in particular as the metadata/post-release symbol indicator would definitely be problematic with your metadata selectors (which are pretty well aligned with NPM, yay) as ~ already has an important role as a selector. Obviously, though, we could pick from many alternate symbols (e.g. %, :, ;, ', \, /, _, &, etc.)... or keep using + to preserve backward compatibility despite its inheritantly unsemantic nature. 😢

After that, though, NPM's client and registry would also need to be updated to stop stripped build metadata and allow multiple versions of the same package version to be published with different build metadata. Would also need to add the appropriate corresponding selectors into node-semver, of course, but I'm personally less worried about happening successfully than the former concern with publishing.

[EddieGarmon]

Build metadata is a human friendly note, and nothing more. Two packages with the same version and different metadata (and differing content) would be 😿 all over the place.

[JamesMGreene]

Build metadata is a human friendly note, and nothing more.

I think it can serve a much better role than it does—at least in NPM—today.

SemVer §10 states:

• "Build metadata SHOULD be ignored when determining version precedence. Thus two versions that differ only in the build metadata, have the same precedence."

This means that build metadata differences do NOT alter the version "level". For example, the versions 1.0.0+win, 1.0.0+mac, and 1.0.0+linux all share the same version "level": 1.0.0. So, particularly for determining precedence ordering, they are congruent versions that can exist in parallel.

It does NOT mean that versions with the same version level but differing build metadata are identical/equal versions/content. NPM makes this faulty interpretation today by always stripping the build metadata off the version number during prepublish/publish, which thus disallows publishing another "parallel" version with different build metadata.

For example, let me describe a sequence of events:

1. I publish a new version of my module as version 1.0.0+3.0.0 (or 1.0.0+win)
2. A consumer requires that version as a dependency and installs that version via a selector like 1.x+3.x (or 1.x+win)
3. I get around to publishing the parallel implementation with a differing build factor as version 1.0.0+3.5.0 (or 1.0.0+mac)
4. The same consumer checks for updates to his/her dependencies. The check's results should (MUST) be that there are no newer/later versions available because 1.0.0+3.0.0 and 1.0.0+3.5.0 (or 1.0.0+win and 1.0.0+mac) have equal version precedence, despite not being equal versions (equal version "levels").

Two packages with the same version and different metadata (and differing content) would be all over the place.

Yes... but that is not inherently incorrect unless the publisher makes it such. If the APIs (and [known] bug statuses) are in sync, then this is actually perfectly correct usage.

I also think that the package manager can make the call on the default behavior of publishing versions with build metadata. For example, I think it is fine if NPM continues to exclude the build metadata from the version by default so long as it provides a mechanism to include the build metadata in the version (e.g. adding a "publishWithMetadata": true pair to the "package.json" manifest file, adding a --includeMetadata flag when running the npm publish command, etc.).

[FichteFoll]

Yes, that's exactly what I'm thinking of build metadata. However:

The current semver spec doesn't exactly allow this use with multiple parallel releases that share the version number but have differing metadata, but this is pretty much the only viable use case for metadata that I have found so far. On the flip side, this would probably be better implemented as a prefix since semantically the metadata is more important than the major number and would separate channels respectively (so win_2.1.0 instead of 2.1.0+win). In that regard, it probably makes more sense right now to rename the project and provide the same code base built differently in packages proj-win, proj-osx etc. and not abuse metadata because of that.

For the dependency example, that would mean include the dependency version in your name and then provide your own version number after that. npm install flex-sdk-4.6.0@*, as you mentioned in your OP already. Compare that to npm install flex-sdk@4.6.0_*, which is not much different and more prone to causing confusion.

Going back to @isaacs's point: I'm not actually sure if semver needs to (or even can) solve this use case since it is likely to behave different for different platforms. The closest proposal I can think of right now would be the <prefix>_ addition that would take over metadata's only viable use case I can think of, so it can remain to be polluted with useless info/stuff or maybe be used as the post-release identifier that would be a lot more useful. Since the prefix technically denotes a separate project however (version comes after it), it doesn't need to be specified by semver.

[JamesMGreene]

Sorry, apparently I am back-tracking to respond to @EddieGarmon's comments in reverse order:

You could easily introduce a breaking change in your wrapper API, and as a consumer I want to know this.

Agreed.

Also, I'm not going to choose a wrapper because I am already using some specific framework; for me it works the other way around.

At least for my particular use case, that is definitely not the norm. Consumers come seeking the specific version [ranges] of the external library to ensure they can compile to their desired target level... they don't really care about my wrapper beyond knowing the basic API hasn't been broken (major version upgrade) as it truly as paper thin.

[JamesMGreene]

@FichteFoll:

The current semver spec doesn't exactly allow this use with multiple parallel releases that share the version number but have differing metadata, ...

I would disagree. As I mentioned earlier, SemVer states that build metadata should be ignored for the sake of version precedence (which only means for sorting order), not that it should be ignored altogether for version equality checking.

...but this is pretty much the only viable use case for metadata that I have found so far.

Agreed. I see no point to having the metadata extension even be specified if you can't publish a version that includes it... but perhaps that is a problem limited to NPM's interpretation of the SemVer spec? Not sure if other package managers suffer from the same interpretation.

Since the prefix technically denotes a separate project however (version comes after it), it doesn't need to be specified by semver.

If it's not part of SemVer, then it will not end up well-supported... in which case, changing the package name rather than the version still makes more sense to achieve better cross-cutting support.

[JamesMGreene]

The troubling part is that now this has me thinking through how to basically abuse NPM package names and dependencies in order to fake double-versioned SemVer....

For example:

Top Level:

• flexsdk@3.0.0
  • flex-sdk-3.0.0@*
• flexsdk@3.0.1
  • flex-sdk-3.0.1@*
• flexsdk@3.5.0
  • flex-sdk-3.5.0@*

Low Level:

• flex-sdk-3.5.0@1.0.2
• flex-sdk-3.0.1@1.2.3
• flex-sdk-3.0.0@1.3.4

Top Level Convenience Aliases:

• flex-sdk-3.0@1.4.5
  • flex-sdk-3.0.1@*
• flex-sdk-3@1.6.1
  • flex-sdk-3.5.0@*

[FichteFoll]

"Top Level Convenience Aliases" is where things start getting dirty.

[JamesMGreene]

"Top Level Convenience Aliases" is where things start getting dirty.

Agreed. They're also really not necessary since they are achievable via actual SemVer-based selectors:

• npm install flexsdk@3.0.x ~== npm install flex-sdk-3.0
• npm install flexsdk@3.x ~== npm install flex-sdk-3

So I should [happily] just drop those so long as I continue having a faux-SemVer top level package like flexsdk. Without that (or if I need to rev the major versions of the underlying flex-sdk-x.y.z packages), I would feel obligated to provide the aliased package names.

[timdp]

Yet another option would be to create a package that is able to install any version of the Flex SDK by specifying that version in the code rather than in the package version. For example:

var flexSdk = require('flex-sdk-wrapper')('4.6.0');

That would give you the flexibility to release (wrapper) package updates as you see fit, and support newer Flex SDKs in newer versions of your package. In a way, that makes a lot more sense to me. And you could still support semver in the argument that you pass to the function.

Of course, the downside would be that you replicate some of npm's logic in your package's source. Also, your Flex SDK version number would no longer be in package.json, although you could easily have the package extract a key from that file to find out which version of the SDK to install.

[JamesMGreene]

@timdp: Valid idea but not something I would want to provide or use, personally. Additionally, that means that all installation variations necessary must be handled within a single version of the module, rather than being able to handle those on a per-version basis.

[timdp]

@JamesMGreene Is the installation logic that complicated? I'd be willing to have a stab at creating such a package, but if you say that you had a lot of issues with yours, I'll probably bail.

Sorry to kinda go off topic here.

[timdp]

For what it's worth, I built:

• flex-sdk-provider: downloads and unzips Flex SDK builds, and gives you the root path to each build.
• grunt-mxmlc-lite: uses flex-sdk-provider to run mxmlc with a flex-config.xml that contains all of the configuration.

James, it's nowhere near as versatile as your approach, but all the configuration logic could theoretically be backported if need be. Since it covers my scenario, I'm not going to bother with that for now though.

I'll shut up about the Flex SDK-specific case now.

[JamesMGreene]

@timdp: More power to you. 👍

Unfortunately for me, I haven't any free time to really work on my open source projects since like Christmas time, so everything has been a it stale/neglected, and this particular issue is not at the top of my list. 😢