Clarify what a failure means

[jdalton]

I'm looking at lodash on http://next-update.herokuapp.com/ and noticed it states 2 failed upgrading from 2.2.1 to 2.4.1. What does the fail mean? What can cause a fail?

[bahmutov]

John, these 2 failures could happen for any number of reason, not necessarily due to lodash itself. Might be flaky npm environment and install failed. Or the lodash version was conflicting with some other peer dependency, or the unit tests themselves were flaky. Could be lodash api change.

The stats sent by next-update are completely anonymous and do not send the reason, just dependency name, from and to version, test success or failure. You can see all data the MongoDB collects if you try http://next-update.herokuapp.com/package/lodash

Older version of next-update did not check if the unit tests pass before starting updates, so it could be that the tests were not working even for 2.2.1!

I could not think how to send a good reason for the failed test, especially due to privacy concerns. But this could make a perfect enterprise dev application, you agree? One team keeps developing a module and if any other teams tries to upgrade and fails the first team gets meaningful information.

As far lodash itself - I am surprised I do not see more data points. It is a super popular package, and has lots of versions. On the other hand, no one tried to upgrade underscore yet ;)

g

[jdalton]

or the unit tests themselves were flaky.

Is it testing against Lo-Dash's unit tests or the packages unit tests?

[bahmutov]

next-update is testing against the package's own unit tests.

[jdalton]

next-update is testing against the package's own unit tests.

Ah ok. So currently do you all check to see if their unit tests pass before upgrading and then again after upgrading?

[bahmutov]

Yup, now next-update runs their unit tests first, and if they pass tries
each dependency one by one.
BTW, there is another mode: install latest version for all dependencies at
once and run tests. I do not remember if I send stats to each dependency in
this case. This might be another source of false positives.

gleb

On Thu, Jul 10, 2014 at 1:21 PM, John-David Dalton <notifications@github.com

wrote:

next-update is testing against the package's own unit tests.

Ah ok. So currently do you all check to see if their unit tests pass
before upgrading and then again after upgrading?

—
Reply to this email directly or view it on GitHub
#19 (comment)
.

[jdalton]

Ah nice. I've seen npm hiccups cause issues too so it'd be nice if that was detectable (like a failed install) and not counted either. I'm curious because Lo-Dash follows semver so versions failing between non-major bumps raises a flag for me (smth I should look at).

[bahmutov]

Agreed, I will make a separate issue for next-update to not count failed
install as a failure. Good that you take failures over minor versions
seriously

On Thu, Jul 10, 2014 at 1:47 PM, John-David Dalton <notifications@github.com

wrote:

Ah nice. I've seen npm hiccups cause issues too so it'd be nice if that
was detectable (like a failed install) and not counted either. I'm curious
because Lo-Dash follows semver so versions failing between non-major bumps
raises a flag for me (smth I should look at).

—
Reply to this email directly or view it on GitHub
#19 (comment)
.