Consider ^ over ~ for default --save option

[guybedford]

If npm install underscore --save automatically used ^ instead of ~ in the dependency, surely this would be in the interest of growing semver compatibility?

Additionally approaches like Browserify would benefit from better pruning from semver compatibility being both used and maintained.

I understand it's not a trivial change, so happy to discuss further.

[jakobmattsson]

+1

[tj]

+1 for sure. People probably wouldn't be happy if we changed ~ semantics to match ^ but I would be ok with that personally, ~ looks nicer haha

[mikolalysenko]

+1 I brought this up in IRC the other day after the pervasive use of "~" bit me during a routine upgrade of one of my modules.

I also opened an issue over on npm/init-package-json#10 (and maybe I was a little harsh).

If the automatic upgrading via full semantic versioning is too offensive, I think it is reasonable to completely lock to a specific version. But I stand by the assertion that the current situation with "~" is crazy.

[balupton]

This has landed with v1.4.3 with pull request #4589

Honestly, for me and the DocPad community this is a terrible change. I expressed this in the tweets here a while ago: https://twitter.com/balupton/status/430248212117454849

We use to allow minor version updates, but in node land, minors mean things could break for some people, so we started using ~ instead, as revision changes ensure that b/c is never broken.

The reality is that minor changes introduce new functionality, even if they are not meant to include b/c breaks, this new functionality could break something unintentionally, and the as more minor and revisions come out, b/c breaks compared to the original release it was for.

The only definition of the versions that I have found to be safe, is that included in the tweet I mentioned.

So why would anyone want npm --save to ever include something that could risk breaking?

[timoxley]

@balupton I think you accidentally the same link twice?

[balupton]

@timoxley intentional, but I removed the second link and update the wording instead based on your feedback :)

[balupton]

Do those config options already work?

I just pushed up docpad/docpad@1656771 which does a standard find and replace to work around this.

[timoxley]

Allowing it to be configurable + passed on the command line would also probably get more developers making a conscious choice about their semver ranges on a per-package basis.

Currently, it seems if you don't want to use the default semver, you're effectively punished by npm as you can no longer use any of the --save conveniences that it provides.

Most will pick the default semver range and not give it another thought simply because it's convenient. Maybe that's a good thing though, as ~ is probably better than everyone getting paranoid and using fixed ranges all the time.

[guybedford]

@balupton thanks for voicing your concerns. I think you're looking a little too closely at this in comparison to how things work today, instead of seeing this as an important cultural change.

Consider the breaking scenario:

• I install a dependency with save, which now defaults to ^x.y.z
• When updating that dependency, a minor increment may bring in new functionality which breaks the old functionality.

Now lets consider a world in which most users now have the dependency range set to a semver (^x.y.z).

The update will have broken for many users. Lots of angry responses get sent the author. They release a patch, and the breaking change is fixed. In future the author tests their releases more thoroughly with beta users, now bearing in mind backwards compatibility and understanding the importance of semver.

This is a very healthy correction process.

Consider also that modules that do break with minor revision changes are typically versions 0.y.z. The nature of semver means that these minor revisons don't automatically update, so you still get the existing functionality, only for 0.y.z versions.

The key use case to bear in mind for this as well is for using Browserify. The only way npm modules can properly scale to work in the browser is by having this cultural change to using semver properly. Knowing backwards compatibility is supported allows modules to share dependencies, which is the only scalable way we can make modules work in the browser without heavy duplication.

[balupton]

Thanks for the detailed response. For me, it seems optimistic, let me try to explain:

1. With DocPad, we have say docpad-plugin-less, that includes the less css dependency. Now, when less updates minor versions, no api change has occurred, but the new functionality causes the result output to change, which can then affect our tests as we compare the result output. Our tests would now fail if we used ^ instead of ~. This is just one simple example that has previously happened, but is equally applicable not just to tests but app logic as well.
2. We then follow the advice of getting angry with the less author, but they've already released a whole bunch of new versions which others now depend on. So if they fix the problem we're experiencing as a new minor release it would now break all the more recent minor releases. That's the problem, fixing a b/c break, is a b/c break in itself.

Semver is great in concept, but if fails realistically, as one can only know about the b/c breaks that they know about, and too often than not we introduce ones we don't know about, and life goes on, and by the time it's discovered, it's too late to fix it, and anything that was using the old way is now broken, and if that is in a nested dependency that is no longer maintained, that's a big PIA.

The reality of this brings the ideal of semver to work in the way myself and Rod Vagg describe in the twitter conversation, and I can't see a way this is unavoidable. Ideal says one way, reality says another. It's noble to chase an ideal, but when the recommendation of pursuing this ideal is to have things break, and users cry out, in the hopes that the maintainers will and CAN STILL fix the issue, is another thing.

---

The only way npm modules can properly scale to work in the browser is by having this cultural change to using semver properly.

Additionally approaches like Browserify would benefit from better pruning from semver compatibility being both used and maintained.

I don't understand these quotes... the solution to maintaing packages isn't adding more risky version ranges, but by getting more maintainers. For instance, for DocPad we now have a team of about 20 people helping maintain our plugins.

We also keep our plugins minor versions in sync with their dependency minor versions. E.g. if we have docpad-plugin-less@2.2.0 which uses less@2.4.0, and less bumps to 2.5.0, we then include in the plugin via ~2.4.0 and bump the plugin minor version to 2.3.0.

This dual-approach has worked really well for us and our users so far.

---

Rather than trying to change culture's obedience to a standard they've already read, this time by beating them with a stick, wouldn't it be better to just update the standard with what reality actually defines it as? This is the same thing with language, no one uses language as it was set out to be thousands of years ago, it evolves despite the language police best efforts.

For me, as stated in the tweets, the best solution I've found is simply to use ~ and define semver as:

revisions = no b/c breaks
minors = b/c breaks for some
majors = b/c breaks for everyone

An example of where this would work would be less. Less introduces an improvement that won't break anything, that's a revision release. Less introduces optimisations to their output rendering or changes some obscure API that only a few people use, that's a minor release. Less changes a part of their API that everyone uses or introduces dangerous optimisations for the output by default, that's a major release.

The original specification of semver just doesn't seem to catch these use cases well, or maybe it does but then we'd be on less version 1000, even when only output changes, or obscure APIs that no one should have been using in the first place but maybe someone was.

[rlidwka]

revisions = no b/c breaks
minors = b/c breaks for some
majors = b/c breaks for everyone

+1

[guybedford]

There is so much benefit to be gained by maintaining a single backwards-compatible version. I completely agree we need to ensure this is realistic, but I am still not convinced why it can't be, and until then would rather try to work towards the ideal.

If there was a bad maintainer that was causing issues, I can always switch back to using ~ (perhaps in your LESS scenario, since you have specially accounted for it in your process anyway).

Also, in most cases bug fixes typically only get applied to the latest minor version only. So a user can be stuck on a previous minor, with a bug, that can only be resolved by making a breaking upgrade (consider that the module author has refactored, and can't afford the effort to solve the bug on the old code base).

In terms of browser use, what I am describing is that with the current system we can often end up in situations where the same dependency is being used in different minor versions within the same overall application. This is a duplication of code sent to the browser, making large browser apps impractical.

Semver thus properly enables npm for use in the browser, by ensuring backwards compatibility allowing for duplication to be avoided. It also ensures bug fixes are shared more widely. In the browser scenario I really do think it is more of a necessity than an ideal.

I definitely agree though that we do need to consider these edge cases as you describe and ensure that we are catering for minimal risk in the process.

Considering your example - I'm not sure I fully follow this from the perspective of whether it is an internal API change or not. This is my best interpretation:

• LESS makes a change that altered a previous output (into CSS), but retained it's outward functionality (in terms of how the CSS is interpreted by the browser), so this is not considered breaking and makes sense as a minor release.
• Your tests break though because of the nature of the output has changed.
• So you update your tests to accept the new output and release a patch, with the LESS dependency now set to this new minor as the semver minimum.

Where in this process is the breaking change? It would be good to understand this further if you don't mind sharing.

Thanks for the discussion - it's really good to hear the arguments on this.

[guybedford]

In terms of unknown backwards compatibility changes, the point is that these get discovered in the process and resolved. Early upgraders would notice the issues first, and patches get properly applied for subsequent installs. By forcing this process outward, that is how to create an ecosystem of backwards compatibility.

Note also that most npm packages are less than the 1.0 release, so most packages will continue to behave the same though.

Packages over a 1.0 release should at least be able to commit to this effort, as that is the nature of maintaining a project for users.

In terms of risk, I wonder if we really are increasing risk or decreasing risk in fact overall. The first to upgrade does take on a slightly higher risk, but the early adopters then solve the issues for the rest. The issues get put together into one place, and solved quickly, instead of distributing the risk over lots of minor changes.

[balupton]

Thanks for the reply. Here's some replies to the parts that stood out for me.

Also, in most cases bug fixes typically only get applied to the latest minor version only. So a user can be stuck on a previous minor, with a bug, that can only be resolved by making a breaking upgrade (consider that the module author has refactored, and can't afford the effort to solve the bug on the old code base).

For us, we release each change independently (or at least try to). So we always try to ensure each minor version range is completely stable. If a bug is found, we release a revision level fix for it — unless the fix requires b/c breaks, in which if the b/c break is for some (it is released as a minor), if the b/c break is for everyone (it is released as a major).

Semver thus properly enables npm for use in the browser, by ensuring backwards compatibility allowing for duplication to be avoided.

The problem is that semver only talks about the public API, not result output, or in the case for the browser, stylesheets. For instance, if I depended on a browserify component that had the default background color black, but then changed it to white. By semver's definition, stylsheets are excluded as they are not a public API. This leads me onto my next point.

In terms of risk, I wonder if we really are increasing risk or decreasing risk in fact overall. The first to upgrade does take on a slightly higher risk, but the early adopters then solve the issues for the rest. The issues get put together into one place, and solved quickly, instead of distributing the risk over lots of minor changes.

For websites, it does increase risk, as say we are developing locally on our machine, we do a deploy, or in 6 months our website goes down and is then rebooted with new deps. Our website is broken, or doesn't display correctly, something's background color has changed, or our stylesheets are messed up due to different optimisations. Etc.

The expectation that the default would be for us to complain to the author here is really weird. Yes, we can use npm shrinkwrap if we really really care, but then we lose out on all the benefits of getting non-breaking bugfixes applied, my definition of semver's revision releases, which the ~ default option on npm --save gives us.

Changing the default to ^, or requiring us to use npm shrinkwrap, just makes it harder for those not wanting things to break. It also makes my life a lot harder as someone maintaing a content management system, as if say we use ^ and one of our plugin dependencies introduce a b/c break in a minor version, as they do. Then the user's of my content management system don't blame the dependency author, they blame me and my team, as they should, as any problem they encounter is my responsibility to fix.

This is why we've done docpad/docpad@1656771 to revert this change when using DocPad, but it is dangerous, as most people using DocPad are new to node, new to npm, new to semver, and they come to us when things break, and things should never break. But unfortunately, they do break with minor versions, and are way harder to fix because the b/c breaks are coupled with new functionality that sometimes just can't be reverted without breaking things for other users. Which is why keeping the ~ range of a plugin always stable wherever possible works so well.

[balupton]

• So you update your tests to accept the new output and release a patch, with the LESS dependency now set to this new minor as the semver minimum.

Where in this process is the breaking change? It would be good to understand this further if you don't mind sharing.

The breaking change is the output changed, which causes our tests to break. Which makes it a breaking change, as we now have to fix something, be it tests or code, or whatever, to get things working again. More unnecessary man effort.

But this is one example applied to tests, it also applies to other things. Here's another example, same structure, different content:

1. Less releases a minor release that has the same API, but changes the way it parses the syntax of less content. This is fine by both the traditional and my definition of semver.
2. People installing a previously working docpad site that uses ^, now breaks. As less has parsed the css files differently.
3. People get upset at DocPad. DocPad is so unstable. It never works. Always breaking.
4. I report this to less, but as they have partnered this with a bunch of other changes that now depend on the new parsing engine, and now there are others now depending on the new parsing engine, there is no forward way of fixing this, the only option is for me to tell the users, tough.
5. Or of course, we just use ~ everywhere and avoid such problems.

Some other big projects in our experience that follows majors break for everyone, minors could break for some, revisions won't break for anyone, are coffeescript and backbone. These are also projects that could never fix one of their b/c breaks, as because of their size, as soon as they are out, people are already using and implementing them with the different functionality, so any b/c fix is a b/c break for some too.

[guybedford]

It's great that you update bugs for each minor, but most maintainers wouldn't.

Exactly as you say, using this stuff in websites does require version lock for production, exactly with npm shrinkwrap. This certainly should be part of the standard process for putting an npm-managed browser application into production, I don't think it's too much to ask, and there may be better automation options.

Style Changes

This is an interesting point you raise, in terms of how to draw a line between a style change being part of the public API or internal API.

A change in the look of a visual component is a public API change.

Ideally though, a new style would be a new theme. The default theme would remain the same, retaining the public API. If the user wants to use one of the new themes, they can load from one of those instead, as a new API opt-in.

Small alignment changes or CSS browser compatibility fixes would be seen as patch or revision changes.

Small visual adjustments do sit in a grey area, that very much depends on the context of the component. I think authors could work this out sensibly.

LESS and Parsers

I still think we need to clarify this LESS example - I wouldn't say that your tests breaking is seen as a backwards-incompatibility here. It continues to work fine for users, so your tests being updated is just a patch. Effectively your tests are running against the internal API (the CSS), not the external one (the way the CSS is displayed).

For parsers, yes there are scenarios where backwards compatibility is not possible. But mostly these are edge cases and most users would upgrade without issues.

If I want to provide a project to the public and am worried about a parser introducing a breaking change that will get blamed on me, I can always revert to ~. The point here is simply that the default is the semver.

Since many of your plugins are parsers, I can certainly see your incentive to stick with ~ due to potential breaking parser changes introduced when your users install them.

But I don't think your situation is indicative of the default install scenario so I don't see how it is worth overlooking all the benefits for this reason.

Perhaps an alternative fix in your situation could be accommodated - since you are wrapping npm install anyway, perhaps an API hook to alter this default is all you need?

[balupton]

As the twitter conversation has continued, for clarity, here are my issues with traditional semver:

revision = new b/c public api changes
minor = new b/c public api introductions
major = b/c breaking public api changes or introductions
use ^ for ranges

The issue with this I've found, is that there are more things that can break rather than just APIs. For instance, you can keep the same API, but the output could change, which could consequently cause breaks. In this case, developers I've found use the following unnamed semantic versioning standard instead. Let's call it realsemver:

revision = no b/c breaks
minor = possible b/c breaks for some
major = b/c breaks for everyone
use ~ for ranges

Realsemver is what I see in use by pre-processors like CoffeeScript and LESS, and libraries like jQuery and Backbone.

I find this definition to be more realistic, as it seems semver's is shortsighted to only care about only API changes, rather than a vast array of other things that could possibly affect b/c compat.

For instance, let's take Backbone's v1.0.0 release, that would break for everyone. Backbone's v1.1.0 release, that would break for some people.

This is what semver says about this:

Use ^ and as a responsible developer you will, of course, want to verify that any package upgrades function as advertised. The real world is a messy place; there's nothing we can do about that but be vigilant. What you can do is let Semantic Versioning provide you with a sane way to release and upgrade packages without having to roll new versions of dependent packages, saving you time and hassle.

What do I do if I accidentally release a backwards incompatible change as a minor version?

As soon as you realize that you've broken the Semantic Versioning spec, fix the problem and release a new minor version that corrects the problem and restores backwards compatibility. Even under this circumstance, it is unacceptable to modify versioned releases. If it's appropriate, document the offending version and inform your users of the problem so that they are aware of the offending version.

What should I do if I update my own dependencies without changing the public API?

That would be considered compatible since it does not affect the public API. Software that explicitly depends on the same dependencies as your package should have their own dependency specifications and the author will notice any conflicts. Determining whether the change is a patch level or minor level modification depends on whether you updated your dependencies in order to fix a bug or introduce new functionality. I would usually expect additional code for the latter instance, in which case it's obviously a minor level increment.

Which translates to:

• Use ^ for the convenience
• In real life minors break things for whatever reason (either non public api breaks, or not following semver, but still following realsemver)
• But as you use ^ things will break for you without you doing anything
• If you run a content management system, people will blame you, rather than the broken dependencies
• You then have to yell at the dependency authors, but fixing the b/c break will now introduce a new b/c breaks for those dependent on the new/changed functionality that broke things for you, so nothing they can do about it, and nothing you can do about it either

---

To be honest, I feel I have said all I can say on this. Semver reality is minors often break things, people get upset, and you can't simply fix the b/c break now as the b/c break fix would be a b/c break. Using ^ as the semver standard says, will mean things will break, but their ideal is that things can still be fixed under minors, that that is not the case, you can't fix a b/c break with a b/c break. It's flawed — who actually cares about api changes or additions, it doesn't even matter! it's irrelevant, the only thing we care or should care about with versioning, is the likelihood of b/c breaks not whether or not yay it has new api for me, or yay it changes some api — who gives a hoot, we can read the changelog for that — we care about if this is going to break stuff, and what the likelihood of that is.

To combat this oversight of semver with the flaw that minors will break, people rely on ~ and use realsemver instead. This works well. We get bugfixes but no b/c breaks (be it a public api b/c break (semver) or non public api breaks (realsemver)).

Ideally I'd like it if npm keeps using ~ instead of ^, as it means the DocPad team would get blamed less for dependency authors mistakes when things inevitably do break in minors.

However, I've made my patch to DocPad to workaround this docpad/docpad@1656771

And would much rather spend my time making DocPad better, and spending a little amount of time working around npm, than spending an infinite amount of time in a debate that seems is never going to be accepted.

[isaacs]

If you want to pin deps, then there are many ways to do that. You can even use npm shrinkwrap to ensure that the entire tree looks a certain way, or edit your package.json to pin your deps to a specific version, or check the node_modules folder into git (and/or add as bundledDependencies) as I do on many of my own modules. I've already weighed in with +1 on making --save-exact a thing, so there'll be that as well.

Ultimately, a convenience option like --save has to balance the needs of very many people, so there's a compromise involved.

"Always pin deps to a specific version" isn't really reasonable in many cases. It means that you won't get updates, even security patches or minor bugfixes. It is too biased in favor of stability rather than convenience for many use-cases.

"Always allow any updates" (eg, "foo":"*") is also obviously a bad idea.

Most npm module authors I've talked to are at least intending to use versions according to the SemVer spec, and most expect that ~ does what ^ actually does. People who have been using npm literally from day one are frequently confused about what ~ does in version ranges. It does not faithfully capture what most people intend by their version number changes.

Meaning comes from humans, not from specs. If you are using modules where a change from 1.2.3 to 1.3.0 means "breaking changes", then, well, you aren't using the SemVer 2.0 spec at http://semver.org. In your versioning dialect, things are different, so you have to use different notation to express your needs. But your versioning dialect is not the standard, so you have to do non-standard things. That doesn't mean that the standard should change.

If we're going to have a --save that isn't exact by default, then what should it use, if not a publicly specified "this should be ok to update" pattern? And we have pattern like that, in the SemVer specification, which most people are already following.

[isaacs]

I find this definition to be more realistic, as it seems semver's is shortsighted to only care about only API changes, rather than a vast array of other things that could possibly affect b/c compat.

I'd argue that the output of a transpiler is absolutely part of the API. That's the return value of the function.

[medikoo]

Is it just Node v0.10 that supports ^? I've tried to use it with v0.8 and npm crashed with No compatible version found (when same configuration plays well on v0.10).
Since which exactly version of node (or npm) ^ support was added?

If that's the case, then it's probably too early to setup ^ as default.

[kenany]

@medikoo Since semver@2.1.0, which was included in npm@1.3.7 I believe.

[timoxley]

@medikoo oh wow, this is a good point. Afaik npm@1.3.7 is in node >= v0.10.16.

With this change, any modules published with ^ in their deps will break with No compatible version found, where node version < 0.10.16 and they haven't updated npm. @isaacs this is terrible for backwards-compatibility with 0.8, at least until there's a 0.8 release with a more up-to-date npm. Is 0.8 officially supported though? If not, maybe that's fine.

Hm, the default --savecould also update minimum node version but that seems a bit heavy-handed.

[rlidwka]

this is terrible for backwards-compatibility

meh... just add --semver-range, so people who really care about that will be able to use ~ :)

PS: yeah, I see a reason for reverting the change

[isaacs]

Node 0.10.16 is 9 months old at this point, and there have been numerous
serious security issues fixed between then and now in the v0.10 branch.

If npm provides yet another reason to upgrade, then that's a GOOD thing for
our users.

On Wed, Feb 19, 2014 at 3:50 PM, Alex Kocharin notifications@github.comwrote:

this is terrible for backwards-compatibility

meh... just add --semver-range, so people who really care about that will
be able to use ~ :)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/4587#issuecomment-35566621
.

[ssafejava]

Sorry to beat a dead horse, but we are seeing lots of compatibility issues with this all over the internet, in a few modules I contribute to, in #node.js and elsewhere. Part of it is due to the opaque error message you get when installing a module with a package.json using a caret, e.g.

npm ERR! Error: No compatible version found: inherits@'^2.0.1'
npm ERR! Valid install targets:
npm ERR! ["1.0.0","2.0.0","2.0.1"]

Changing the default behavior of --save has created a situation where every single node module written by an author using node >= 0.10.16 now may have an implicit "engines": { "node": ">=0.10.16" }. This is not at all clear to the average developer who likely does not know exactly what the ~ does in the first place, much less the ^. Any developer using v0.6 or v0.8 can easily be bitten by this on a patch release, if the maintainer uses --save at any point.

This should not have been in a minor revision and I would like to posit that we would be much better served by a less clever default format for --save, such as >=2.0.1 <3.0.0. It's just a few more characters in an auto-generated line, but with far less room for misinterpretation and no backcompat breaks. I see questions about ~ and ^ constantly and it appears that it is a never-ending source of confusion. In cases like this I think it is far better to be explicit.