flex-sdk dependency broken?

[timdp]

I just installed grunt-mxmlc into a blank npm module using npm i --save grunt-mxmlc. For some reason, it insisted on installing flex-sdk@3.5.0 rather than the latest version. Moreover, when I tried to make it install version 4.6.0-0, I got this:

C:\tmp\temp>npm i --save flex-sdk@4.6.0-0
[installation of playerglobal-latest@0.1.6 redacted]
npm ERR! peerinvalid The package flex-sdk does not satisfy its siblings' peerDependencies requirements!
npm ERR! peerinvalid Peer grunt-mxmlc@0.5.1 wants flex-sdk@>= 3.0.0-0

If I'm interpreting that error correctly, npm seems to think 4.6.0-0 is lower than 3.0.0-0.

I'm convinced this behavior changed recently. I have a bunch of projects that use grunt-mxmlc and none of them contain an old Flex SDK. Did something change in the way npm and/or semver handles >=?

This is on Windows with node v0.10.31 (on nodist) with npm v2.1.2.

[timdp]

I reinstalled nodist and node. My npm is now down to 1.4.6, which doesn't have this issue.

[JamesMGreene]

Yeah... NPM has been having loads of issues with large modules like flex-sdk lately. 😕

The general lame "answer" I recall seeing was "make smaller modules". sigh

[timdp]

Could you point me to the underlying issue(s) though? I couldn't figure out why it basically seemed to think that 4 was less than 3. Thanks!

[JamesMGreene]

Oh, my bad, I honestly didn't read your comment closely enough. Yeah, that's just peculiar and nothing to do with the module size I was mentioning (sorry for the confusion).

Turns out it is being caused by newer behavior in npm/node-semver, which is responsible for validating and comparing semver versions... and unfortunately, this update will make the particular peerDependencies version range completely unusuable (yay! 😡).

Here's the info on how and why they decided to break this existing, straight-forward mechanism: https://github.com/npm/node-semver#prerelease-tags

[timdp]

Interesting (and pretty annoying). What would be the recommended course of action then?

[JamesMGreene]

Can you try the following on your local system?

• Change this module's "package.json" file to list the version string for the "flex-sdk" (in both peerDependencies and devDependencies, if you are also installing the devDeps) entry as "^3.0.0-0 || ^4.0.0-0"
• Verify that it still works as expected on npm@1.x
• Upgrade back to npm@2.x and verify it works there as well?

This version string should work on both from what I understand... but it is very annoying. 😕

[timdp]

Yep, that seems to solve it on both 1.x and 2.x.

[JamesMGreene]

Thanks for verifying, I'll try to release an update for this module tonight then.

[JamesMGreene]

Fixed in grunt-mxmlc@0.5.2. Thanks!

[timdp]

I just ran into this again with npm v2.1.18, but I can't explain why.

If I run npm i --save-dev grunt-mxmlc, I get version 0.5.2 just fine, but it also installs flex-sdk 4.0.0 rather than 4.6.0.

I subsequently tried to fix it with npm i --save-dev flex-sdk, but that gave me the dreaded error about peerDependencies:

npm ERR! peerinvalid The package flex-sdk does not satisfy its siblings' peerDependencies requirements!
npm ERR! peerinvalid Peer grunt-mxmlc@0.5.2 wants flex-sdk@^3.0.0-0 || ^4.0.0-0

The latter is supposed to mean >=4.0.0-0 <5.0.0, right?

[JamesMGreene]

The latter is supposed to mean >=4.0.0-0 <5.0.0, right?

Correct. 😕

Any luck if you update NPM to the latest (v2.2.0)?

[timdp]

I just tried that right after posting the comment. No luck.

I also tried the following semver strings with mixed results:

• *: Works
• ^4.0.0: Doesn't work (huh?)
• ^4.0.0-0: Doesn't work either
• ^4.6.0: Still no luck
• ^4.6.0-0: Works!
• ^3.0.0-0 || ^4.6.0-0: Also fine

I sense it might have something to do with the -0 prerelease string.

Incidentally, the semver docs say || means an intersection rather than an or. Wouldn't the intersection of >=3.0.0-0 and >=4.0.0-0 be >=4.0.0-0 anyway? Seems more like a union to me.

[JamesMGreene]

The latter is supposed to mean >=4.0.0-0 <5.0.0, right?

I thought so but, upon rereading the npm/node-semver @ README.md#prerelease-tags, apparently not. 😕

Prerelease Tags

If a version has a prerelease tag (for example, 1.2.3-alpha.3) then
it will only be allowed to satisfy comparator sets if at least one
comparator with the same [major, minor, patch] tuple also has a
prerelease tag.

For example, the range >1.2.3-alpha.3 would be allowed to match the
version 1.2.3-alpha.7, but it would not be satisfied by
3.4.5-alpha.9, even though 3.4.5-alpha.9 is technically "greater
than" 1.2.3-alpha.3 according to the SemVer sort rules. The version
range only accepts prerelease tags on the 1.2.3 version. The
version 3.4.5 would satisfy the range, because it does not have a
prerelease flag, and 3.4.5 is greater than 1.2.3-alpha.7.

The purpose for this behavior is twofold. First, prerelease versions
frequently are updated very quickly, and contain many breaking changes
that are (by the author's design) not yet fit for public consumption.
Therefore, by default, they are excluded from range matching
semantics.

Second, a user who has opted into using a prerelease version has
clearly indicated the intent to use that specific set of
alpha/beta/rc versions. By including a prerelease tag in the range,
the user is indicating that they are aware of the risk. However, it
is still not appropriate to assume that they have opted into taking a
similar risk on the next set of prerelease versions.

In other words, ^3.0.0-0 || ^4.0.0-0 actually translates to >= 3.0.0-0 < 3.0.1 || >= 4.0.0-0 < 4.0.1.

There is no good way to write a generic NPM version range for prerelease versions any more. 😞

So... this string will need to be constantly updated, or else we'll have to just put the wildcard (*) there, or else we'll have to stop using prerelease versions (e.g. 4.0.0-0) altogether. Unfortunately, using "build number" versions (e.g. 4.0.0+1) don't result in the module dependencies getting updated regularly either as the SemVer spec dictates that the build number is not to be considered for sub-version comparison (4.0.0+1 == 4.0.0+2).

[JamesMGreene]

Incidentally, the semver docs say || means an intersection rather than an or.

Not so. Space-delimited comparator sets are intersections, ||-delimited comparator sets are unions:

Comparators can be joined by whitespace to form a comparator set,
which is satisfied by the intersection of all of the comparators
it includes.

A range is composed of one or more comparator sets, joined by ||. A
version matches a range if and only if every comparator in at least
one of the ||-separated comparator sets is satisfied by the version.

[JamesMGreene]

Local test:

$ node index.js
Does "4.1.0-0" satisfy range expection "^4.0.0-0"?  No....
Does "4.1.0-0" satisfy range expection "^4.0.0"?    No....
Does "4.0.0-0" satisfy range expection "^4.0.0-0"?  YES!
Does "4.0.0-0" satisfy range expection "^4.0.0"?    No....
Does "4.0.0-0" satisfy range expection "^4.*.*-*"?  No....
Does "4.0.0-0" satisfy range expection "^4.x.x-*"?  No....
Does "4.0.0-0" satisfy range expection "^4.x.x"?    No....
Does "4.0.0-0" satisfy range expection "^4.*"?      No....
Does "4.0.0-0" satisfy range expection "^4.x"?      No....
Does "4.0.0-0" satisfy range expection "*"?         YES!

[timdp]

I misread the part about intersections. My bad.

Probably too obvious a question: do we really need prerelease versions for this?

[JamesMGreene]

More tests to demonstrate my earlier point about build metadata (+) versions:

$ node index.js
"4.0.0-0" is less than "4.0.0-1"
"4.0.0-0" is less than "4.0.0"
"4.0.0" is equal to "4.0.0+1"
"4.0.0+1" is equal to "4.0.0+2"

[JamesMGreene]

Probably too obvious a question: do we really need prerelease versions for this?

The reason I—at least previously 😞—preferred to use prerelease versions is that the real version is dependent on the external source (Adobe/Apache's versioning of the Flex SDK). Ideally I would've been able to use "postrelease" ("build metdata", +) versioning but that became reclassified between SemVer spec 2.0.0-rc1 and 2.0.0-rc2.

In 2.0.0-rc1 (emphasis added):

A build version MAY be denoted by appending a plus sign and a series of dot separated identifiers immediately following the patch version or pre-release version. Identifiers MUST be comprised of only ASCII alphanumerics and dash [0-9A-Za-z-]. Build versions satisfy and have a higher precedence than the associated normal version. Examples: 1.0.0+build.1, 1.3.7+build.11.e0f985a.

In 2.0.0-rc2 and 2.0.0 (emphasis added):

Build metadata MAY be denoted by appending a plus sign and a series of dot separated identifiers immediately following the patch or pre-release version. Identifiers MUST comprise only ASCII alphanumerics and hyphen [0-9A-Za-z-]. Build metadata SHOULD be ignored when determining version precedence. Thus two packages with the same version, but different build metadata are considered to be the same version. Examples: 1.0.0-alpha+001, 1.0.0+20130313144700, 1.0.0-beta+exp.sha.5114f85.

[JamesMGreene]

My inclination at this point is just to continue using prerelease versions [to preserve the external versioning scheme] but set the dependency version range in downstream modules like this one ("grunt-mxmlc") to be * and let it be. 😩

[timdp]

Couldn't you also keep the package version synchronized with the SDK version? That'd be more work, of course. (Also, the Apache version is actually at 4.13.0 while the npm package is at 4.6.0-0, which is confusing.)

I guess that, in most cases, the asterisk makes the most sense. It's not as elegant, but yeah, at least it'll work.

[JamesMGreene]

Couldn't you also keep the package version synchronized with the SDK version? That'd be more work, of course. (Also, the Apache version is actually at 4.13.0 while the npm package is at 4.6.0-0, which is confusing.)

They are synchronized. The latest Node package version (of flex-sdk) that I've been able to find time to publish was 4.6.0-0 (which corresponds to the last Adobe release of the Flex SDK, 4.6.0).

Apache made a bunch of immediate changes to the Flex SDK before releasing their first version (4.8.0) that aren't compatible with the existing Node package. See JamesMGreene/node-flex-sdk#4 for more info, and feel free to help me out with that if you have time/interest. 👍

[JamesMGreene]

To clarify, I use the prerelease versioning for when the Node package has issues that require updating. If I didn't, I would publish 4.6.0 the first time but then be S.O.L. if I needed to fix something on my side because the version 4.6.0 is already taken and cannot be republished on NPM with the necessary fixes.

The preferred alternative is using post-release versioning ("build metadata", +) but, as mentioned above, that becomes an issue, too, because that additional string is not considered during the version comparison, e.g.

$ node index.js
Does "4.0.0+0" satisfy range expection "^4.0.0+1"?  YES!  (Should be "No....", IMHO)
Does "4.0.0+0" satisfy range expection "^4.0.0+2"?  YES!  (Should be "No....", IMHO)

[timdp]

I see. Maybe you could use a hack on semantic versioning with multiples of 1,000? Perl uses this to represent its version number as a float. E.g.:

• Package version 4.6000.0 could correspond to your first release of Adobe's 4.6.0.
• If you needed to patch something, you could still release 4.6000.1 and following.
• If Adobe comes out with 4.6.1, that would become your version 4.6001.0.

Alternatively, you could use only the minor version. E.g. 5.4006001.0 would be your first build of Adobe's 4.6.1. The 5 would just be for continuity.

Not the most elegant solution either, but at least it would avoid the prerelease madness.

[JamesMGreene]

Actually, my previous mention of switching to post-release ("build metadata", +) versioning isn't even possible anyway because NPM strips the build metadata during prepublish (so 4.0.0+0 becomes 4.0.0, 4.0.0+1 would also become 4.0.0 and thus get rejected during publish as a duplicate/taken version): npm/npm#6379

Really, really unfortunate situation.

[timdp]

How do you feel about the alternative versioning scheme I proposed above?

[JamesMGreene]

I appreciate the info and examples but I am very much not a fan of that alternate scheme. 😣

[JamesMGreene]

Related open issues filed against the SemVer spec:

• Nightly builds not supported semver/semver#200
• Suggesting a fourth number to indicate project generation semver/semver#213
• Handling hotfix versions semver/semver#241

[JamesMGreene]

The PhantomJS Node module that wraps PhantomJS went through the same versioning limitations. They eventually just gave up and switched to strictly follow SemVer, resulting in fewer dependency version selection issues (like this one) but also confusing their users. 👎

[timdp]

Wow, lots of reading material there. I'll go through it when I'm feeling more adventurous again.

But essentially, there's no fix then. Are you still going with the * dependency?

[JamesMGreene]

At least for this module ("grunt-mxmlc"), I think using * for the flex-sdk peerDependency makes sense. Otherwise, with the pre-release issues, it will be need to be updated with every new release of flex-sdk and the initial string would be:

^3.0.0-0 || ^3.0.1-0 || ^3.1.0-0 || ^3.2.0-0 || ^3.3.0-0 || ^3.4.0-0 || ^3.4.1-0 ||
^3.5.0-0 || ^3.6.0-0 || ^4.0.0-0 || ^4.1.0-0 || ^4.5.0-0 || ^4.5.1-0 || ^4.6.0-0

[timdp]

Yeah, that's what I was afraid of as well. But I mainly don't like the idea of the prerelease suffix for something that isn't a prerelease, so I still prefer the hack on semver I proposed. I know it looks terrible and it doesn't make that much sense semantically, but it would at least solve that. And without having to cave in and use the asterisk.

[JamesMGreene]

FYI, I created a new issue against the SemVer spec that I think is really the only legitimate fix we might have a chance of arriving at in SemVer 3.0.0: semver/semver#242