Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

No more `npm publish -f` #148

Closed
isaacs opened this Issue · 138 comments
@isaacs
Owner

In b054e48 and 516262d, the ability to publish over a previously published version is removed. This hasn't been published live yet, but it has been tested extensively, and I think it's a Good Thing.

With this change, if you publish foo@1.2.3, you can still un-publish foo@1.2.3. But then, you will not be able to publish something else to that same package identifier. Not now, not never. Deleting documents entirely will be forbidden (though of course old revs still get compacted out) and we'll have a lot more visibility into what each change in the db does.

I wouldn't characterize it as a security fix, per se, but it does reduce a significant gotcha that users have run into over the years, which is especially annoying when deploying Node servers to production. If the bits are gone, ok, it's an error, you notice and deal with it. But if the bits are changed, then this can be really hazardous.

I'm about 85% sure that this is the right approach. It enforces a best-practice that is pretty much universally agreed upon. But, since it is reducing functionality in a way, I think it'd be good for people to speak up if they have a use case that'll be impacted.

@ded

:+1: right on right on. it should have been this way from the start.

@davglass

+10000000

@davemo

\o/ thank you!

@robashton

Oh god, about time.

@benjamn

I've used npm publish -f before when a network connectivity problem or NPM error corrupted the original npm publish. Is there another recommended way to repair snafus of that sort? Not changing the bits, per se, but making sure they're all there, and correct.

@jkrems

How does this work with completely removing a package from npm? Is the package name burned forever?

@isaacs
Owner

@benjamn My first go-to in this situation is to use the npm cache command to fetch and unpack a module to see if it's valid. npm cache clean foo will remove all foo package entries from the cache. Then npm cache add foo@$version will download stuff into ~/.npm/foo/$version/... and you can inspect at will. (Or it'll error out, or yell at you, or whatever.)

@jkrems Fully-removed packages are whittled down to a state where anyone can publish, but the old version numbers are still kept around as permanent tombstones that can never be reused.

@isaacs
Owner

Note that server admins can still DELETE docs directly, so there's still an escape hatch for when such a thing is needed.

UPDATE 2015: PLEASE SEE COMMENT AT THE BOTTOM OF THIS THREAD.

@jkrems

:+1: Sounds great!

@terinjokes
Collaborator

I think @jdalton uses this feature to republish previous versions of Lo-Dash when needed. He might want to add something to this discussion.

@danmactough

I was literally just about to do this to republish an old version of a module with a different dist-tag. Turned out I wanted to fix the readme, so I bumped the version anyway, but it still seems like a valid reason to publish -f.

@isaacs
Owner

@danmactough You can use the npm tag pkg@$oldversion some-tag to do that without a re-publish. Yeah, the documentation update option is where it's annoying. Maybe it'd be nice to add a client command to just update the readme? Seems like that's a common enough use-case to just bless in a nicer way without forceful re-publishing.

@danmactough

You can use the npm tag pkg@$oldversion some-tag to do that without a re-publish.

Ah. I thought that was the case and tried it, but I forgot to add some-tag, so it didn't work. Not something I do frequently, and it wasn't worth another thought as I wanted to update the readme anyway (which is a good enough reason for a patch version bump to me).

Maybe it'd be nice to add a client command to just update the readme?

Tough call. I can see an argument that what's done is done -- even if it's only the readme. Wouldn't bother me, though. But, I never, ever look at the readme that's sitting in the node_modules directory; I just assume there's a more up-to-date readme on github.

@isaacs
Owner

Yeah, it's mostly about what goes into the doc itself, since that shows up on the website, can be searched, etc.

@danmactough

shows up on the website, can be searched

Good points. Still kind of hinky for that shasum to change, though.

@isaacs
Owner

Right, so, I add a new command, npm doc-publish that just reads in your README.md or whatever, and updates that to the root package doc, without changing any tarballs. Actually, that's a really good idea, and easy.

@indexzero

:+1: Some secondary thoughts around

"Deleting documents entirely will be forbidden"

Assuming this is designed to prevent the naive work-around where a user would unpublish the entire package and then republish all versions (including the one single version they wanted to force push). e.g.:

npm unpublish my-package --force // removes all versions (including latest)
npm publish my-package           // Publishes latest again

This is a good thing as preventing deletes all together is the only way to prevent this behavior (or at least to make it auditable which is really what most package consumers want). There are a couple of side-effects though:

  1. Squatting: Although package name squatting has never really been a serious problem, it will be even harder to transfer ownership now if we can never delete documents.
  2. Accidental publishes: We dealt with this a few times when users would contact us to ensure that their private code was actually removed when it was accidentally published. The existing CouchDB "tombstones" made them uncomfortable enough, having living breathing doc records I imagine would be even more so.
  3. Force publish still possible: Reading the delete update function and the changes to validate_doc_update the registry now makes some effort to ensure that the person publishing the package isn't the same person that unpublished it. Wouldn't the following work though?
npm unpublish pkg --force                  // removes all versions (including latest)
npm publish pkg --userconfig second-user   // Intermediary publish to fool the registry
npm unpublish pkg --force --userconfig second-user // Remove intermediary package
npm publish // Now able to republish all versions however I want

Although (3) feels like a nit, given that developers (like @jesusabdullah) see npm publish -f as the moral equivalent to git push --force (even though it clearly isn't) we have to assume that any work around will be used heavily.

@jdalton

I think @jdalton uses this feature to republish previous versions of Lo-Dash when needed. He might want to add something to this discussion.

Yap, I've used it before. It was handy to back port patches when things like a Node compat issue cropped up, or an external resource changed under our feet, or I had a non-functional doc/package typo. It was a win for me at least :)

I've shot myself in the foot before, but recovered. It was useful.

@jkrems Fully-removed packages are whittled down to a state where anyone can publish, but the old version numbers are still kept around as permanent tombstones that can never be reused.

Yikes, so devs using a package name, unaware it was previously owned and deleted, may run into issues with their versioning?

@grncdr

expanding on @benjamn's point a bit, it seems that npm publish should prepare a checksum for the package on the client, and the server should fail the publish if the checksums don't match. That's probably a separate issue though.

@danmactough

updates that to the root package doc, without changing any tarballs

:+1:

@terinjokes
Collaborator

Yikes, so devs using a package name, unaware it was previously owned and deleted, may run into issues with their versioning?

Per @indexzero's comment above, someone new coming along would be able to publish any version they want. But that's probably by mistake…

@indexzero

The more I think about it there doesn't seem to be any good way to preventing force publish without preventing unpublish of entire packages. Any checking we do around users could easily be circumvented with multiple accounts by anyone who care enough to do so.

@isaacs what are your thoughts on removing this API all together? i.e. only unpublish of specific versions is allowed. It makes the name squatting case more prolific, but I'm pretty sure it's the only way to be certain.

@isaacs
Owner

@indexzero

Squatting: Although package name squatting has never really been a serious problem, it will be even harder to transfer ownership now if we can never delete documents.

Yeah, that sucks. Since an admin usually has to step in there anyway, they can just delete the doc entirely.

Accidental publishes: We dealt with this a few times when users would contact us to ensure that their private code was actually removed when it was accidentally published. The existing CouchDB "tombstones" made them uncomfortable enough, having living breathing doc records I imagine would be even more so.

Either way, it's still in the db until a compaction, and previous revs are just as gone as _deleted revs after a compaction, so nothing changes.

Force publish still possible: Reading the delete update function and the changes to validate_doc_update the registry now makes some effort to ensure that the person publishing the package isn't the same person that unpublished it. Wouldn't the following work though?

No, actually, but thanks for making me walk through this. The error message was completely useless there :) Fixed on eff893e

@jdalton

@isaacs Can you explain a little bit why force publish is being removed (I know you had a summary at the top but I'm wondering if there there was a long history of abuse/misuse to point to) and why it was prioritized over something like getting npm stats back up? I found force publish useful. It wasn't something to abuse but I was glad it existed when I needed it.

@indexzero

@isaacs Ok I see, very clever. For the other 90% of people who don't understand CouchApps this is how it works:

  • When a package (say winston) is "unpublished" it's document becomes:
{
  _id: "winston",
  _rev: "42-somerev",
  name: "winston",
  time: {
    // All the old time stuff
    "unpublished": {
      "name": "indexzero",
      "time": 1392186915505
    }
  }
}
  • When a new publish request comes in later it hits the validate_doc_update which gets the new document and the old document it has to pass this bit of code:
for (var v in oldTime) {
  if (v === "modified" || v === "unpublished") continue
  assert(doc.time[v] === oldTime[v],
    "Cannot replace previously published version: "+v)
}
  • Because both the old document and the new document have the same unpublished.time value it is forbidden.

@isaacs ramp up that support team on social engineering sir because I suspect they're going to get a fair amount of requests for this. Any thoughts on say ... only doing this for packages with more than 10 versions? I know lots of folks who publish to the same test package name and then unpublish. Preventing that is going to get pretty ... frustrating.

@contra

@indexzero You could use a few things to determine if a package shouldn't be able to force push (dependents, number of installs, how long it has existed, is it 1.0+ etc.).

IMO I think removing it completely is the best way to go and documenting this properly is going to be really crucial to prevent frustration. npm publish -f should link to something useful instead of ceasing to exist

@isaacs
Owner

The more I think about it there doesn't seem to be any good way to preventing force publish without preventing unpublish of entire packages. Any checking we do around users could easily be circumvented with multiple accounts by anyone who care enough to do so.

I don't think that's the case. But I'd love a patch that adds a test. Try checking out this repository and running whatever tests against it that you'd like.

I know lots of folks who publish to the same test package name and then unpublish. Preventing that is going to get pretty ... frustrating.

Are any of them on this thread? If not, can you please @-mention them so that they can be a part of this discussion? Thanks. (The npm tests themselves do this, but clearly I'm ok with this, so we can refactor those tests.)

Because both the old document and the new document have the same unpublished.time value it is forbidden.

Why is that? doc.time.unpublished.time is never checked. I'm confused?

@isaacs Can you explain a little bit why force publish is being removed and why it was prioritized over something like getting npm stats being restored?

@jdalton force publish changes the bits that are returned for a specific version number. This means that pinning your version is not a reliable way to ensure that you have a specific version of a package. It reduces data integrity, and can lead to difficult-to-resolve conflicts in the database. Many users have asked for this over the years.

This wasn't prioritized "over download stats". I worked a bunch on the npm stats system recently, in fact, and @seldo is researching time-series databases for us to use to efficiently deliver this information more up to date and more on-demand that it was before. The raw data is still all backed up, and we will be providing the counters again soon, and better. Hooray for multiple brains doing multiple things in parallel!

We're not going to do anything where some arbitrary limit means you get to start doing forced publishes. Changed bits in a package with a single version is still changed bits that can break your program in mysterious ways that are hard to detect.

@jdalton

force publish changes the bits that are returned for a specific version number. This means that pinning your version is not a reliable way to ensure that you have a specific version of a package. It reduces data integrity, and can lead to difficult-to-resolve conflicts in the database. Many users have asked for this over the years.

Yaaa buuut... it was pretty useful, & I'm glad I had it available when it was needed.

This wasn't prioritized "over download stats".

The reason I ask is because there are a few issues open for the download stats and it seems like a more pressing issue than removing working/useful features, like force publish, from npm.

Was there an open issue to remove this feature or was it on some roadmap? This kind of looks like you're dropping it on your users, :confetti_ball:, and using the issue tracker as a way to broadcast the change. Seems a bit backwards (usually goes the other way, an issue is opened, discussed, PRs made, & then merged).

@isaacs
Owner

To explain the checking algorithm:

For old (existing document) and doc (new PUT body)

  1. If doc._deleted, and not an admin then just fail. (No outright DELETE reqs, except by server admins.)
  2. If old.time[v] exists, and doc.time[v] is different from old.time[v], then fail.
  3. If old.versions[v] and doc.versions[v] are (deeply) different, then doc.time[v] must be different from old.time[v]. (Along with previous rule, this means old.time[v] must be unset.)
  4. If you DELETE to registry/_design/app/_update/delete/pkg, then it'll set the write body to something like {..., {"time":{..., "unpublished":{"name":"some-user","time":"2014-02-12T06:51:45.827Z"}}}. The contents of the time.unpublished object are mostly irrelevant. Strictly there so that a human can one day have an extra clue about who deleted it. (Booleans are a waste of information density ;)
  5. Docs with a time.unpublished entry may not have versions.
  6. If you PUT to a doc that has a time.unpublished entry, then you must remove that entry, and also you can do that even if you're not a "maintainer" (since the maintainers list was removed in the unpublish!)
  7. If doc.time.unpublished and old.time.unpublished, then fail (cannot unpublish an already unpublished doc, obscuring the human-aiding clue)
  8. If adding a doc.time.unpublished then the doc.time.unpublished.user must be the currently logged-in user.
  9. If adding a doc.time.unpublished then the currently logged-in user must be in old.maintainers.

There might still be a hole in here somewhere, so if anyone has a suggestion or patch, I'd be happy to review it. AFAICT, this ensures that unpublishes are always accountable, and that versions can never be re-used, causing weird surprises for other users of those modules.

@aheckmann

Sounds great. I've seen a few module authors abuse republishing in the past. +1

@isaacs
Owner

Was there an open issue to remove this feature or was it on some roadmap? This kind of looks like you're dropping it on your users, :confetti_ball:/, and using the issue tracker as a way to broadcast the change. Seems a bit backwards.

I am using the issue tracker as a way to solicit feedback, yes. It is backwards, but I've found issue trackers to be most useful as two-way information conduits :)

I don't know of an open issue to remove the feature. If there was one, it was probably in npm, not here, but here is where the change needs to be, for security reasons, since the client is just sugar around an HTTP endpoint.

However, I have been literally begged for this, in person, via email, and over twitter, numerous times, from friends and colleagues at several different companies. And on a personal level, as one of the people that people go to when their npm stuff is confusing and weird, I've shaken my own fists at myself for ever allowing publishes over pre-existing versions in the first place.

@jdalton

I've shaken my own fists at myself for ever allowing publishes over pre-existing versions in the first place.

In cases where force publish might be needed, would a solution be to contact the registry admins for assistance? TBH I'm cool if that's an option.

@isaacs
Owner

In cases where force publish might be needed, would a solution be to contact the registry admins for assistance? TBH I'm cool if that's an option.

Yeah, anyone with the admin bit can still delete whatever they want. If you npm publish something where your social security number is the version, or something, a human can manually intervene. But then, there's an email thread about it, some process, etc. Not just "Whoa, how come that line number in the error is different for you than it is for me?"

@jdalton

But then, there's an email thread about it, some process, etc.

I can dig it :+1:

@glenjamin

It might be useful to allow an un-un-publish - the exact same version can be restored.

Even if no UI, I think it makes sense to keep the tarball shasum in the unpublished doc - unless this info exists elsewhere

@ljharb

What about packages that have, say, zero downloads? I fail to see any value in preventing force pushes to versions that have never been used, and that would cover the "accidental publish" use case pretty well, if it was caught quickly. Are there difficulties I'm unaware of wrt npm mirrors?

@medikoo

It's great decision.

Still, if it's possible it might be good to allow republish for first 5 minutes after publish, so we can clear obvious mistakes (e.g. inclusion of files that should not be there). At least that was the only use case when I used forced publish feature.

@terinjokes
Collaborator

My use of force publish is consistent with @medikoo's, I screwed up a publish and quickly resolved it.

@jdalton

One of my more recent force publish uses was like @ljharb, @medikoo, & @terinjokes but multiplied by ~150 as it was when I first published lodash functions as individual modules.

@Raynos

For the 5 minute issue that @medikoo @ljharb @jdalton & @terinjokes mentions maybe we should have an npm publish --dry that allows you to verify a publish before running it.

I'm not quite sure what you want to verify before doing it though. maybe a dry run should generate a tarball you can inspect.

I'd imagine its definitely useful for @jdalton case where he wants to script publishing of a 150 modules and needs to test the script.

@davglass
@jdalton

In my case I had typoed the dependency version ranges (they were fixed instead of a range). So I forced published shortly after to correct the issue. Not sure a dry run would help there as it was a doh moment and not an issue with the package contents.

@davglass

What if --force only worked on the latest version, once and only within a few minutes (like 2)?

We have something similar at Yahoo, but we call it quarantine. A package is published there and verified before promoting it out.

@isaacs
Owner

There are technical complications around having a time based limitation.

Remember, couchdb is a sequence-based rather than time-based updating paradigm. If you do time-based stuff, in a way that the functional semantics of the system depend on (a) the clock time being the same on all servers and/or (b) the clock time when a PUT comes in being near-in-time to the clock time when a PUT was initiated, then that starts to fall apart in systems where you have distributed architectures (including re-playing writes in down-stream replicas, which we do quite a lot of).

So, we won't be doing that.

Only allowing a --force publish of the latest module might be ok, but it's tricky to implement in a secure way. And without the problematic time sensitivity, it's really still the same disruptive problem. What's to stop you from just force publishing over the same module a thousand times?

@jdalton's use case of messing up 150 modules, and then being able to clean them up, is a valid use-case. But it's rare, and probably fine in those cases to reach out to a server admin for help.

@raynos Re npm publish --dry, isn't that exactly npm pack? Your point is a good one, though: more people should know about pack and use it to inspect artifacts locally and test scripts. Write a blog post about it! (Or, you know, nudge me to :)

In the cases where you genuinely have published something you shouldn't've, unpublishing, bumping the version, and publishing a new thing seems like it's basically always the right choice, no?

@isaacs isaacs referenced this issue from a commit in npm/npm-registry-client
@isaacs isaacs Remove publish --force 535e167
@Raynos

In the cases where you genuinely have published something you shouldn't've, unpublishing, bumping the version, and publishing a new thing seems like it's basically always the right choice

That's the best approach. patch versions are cheap ! create lots of them.

@davglass
@isaacs
Owner

@ljharb Tying it to download counts would be really tricky. A lot of new moving parts to make that work, since downloads aren't tracked in the same place.

@davglass Allowing a single re-publish (or n republishes) of a version would be doable. It complicates the data structure a little bit, but not very much, and doesn't rely on anything external (like download counts or clock drift).

Fundamentally, my goal here was just figure out if there are actually use-cases for re-publishing that cannot for whatever reason be addressed via admin intervention. What I've learned (which isn't surprising, really) is that there ARE use-cases, but they're rare, can be easily addressed by human intervention, and that human intervention is probably a good idea in those cases anyway.

I'm going to go ahead and make the change, and if it leads to a flood of administrative overhead, we can explore a max-republish count, or some other reasonable limit.

@aredridel

Ow ow ow ow ... okay. Yes, this is the right answer.

@davglass
@jdalton

I'm going to go ahead and make the change, and if it leads to a flood of administrative overhead, we can explore a max-republish count, or some other reasonable limit.

Rock! @phated scared me straight dishing up worst case scenarios.
[mental note: I must always lock computer around @phated :grinning:]

@phated

I see uses of force publish, but the security concerns far outweigh convenience. So :+1: from me

@henrikhodne henrikhodne referenced this issue from a commit in travis-ci/dpl
@henrikhodne henrikhodne fix(npm): don't publish with --force
force publishing is going away in the near(-ish) future, see
npm/npm-registry-couchapp#148.
3a40380
@henrikhodne henrikhodne referenced this issue in travis-ci/dpl
Merged

fix(npm): don't publish with --force #101

@mzgol

We've recently had to re-publish jQuery 1.11.0 & 2.1.0 packages since npm publish did the publish but didn't properly register the tarball so doing npm install jquery was actually breaking.

I guess we could publish to 1.11.0+1 or similar in such cases though that would create a confusion for jQuery users; they would need to understand why there is no 1.11.0 package on npm (whereas there is such a package on bower & when distributed as a source file) but 1.11.0+1.

@elgs

Definitely +1.

@coderaiser

I also sometimes need to republish some package. Anyway I do it on a couple minutes after publish. So would be great if on period about half an hour (or 24 hours) there was ability to republish package. If someone do not republish package in time, package is frozen up.

@elgs

@coderaiser I used to have the same requirement until I found that I could do npm install /my/local/module/path. I had some test code which required my module to be published to the npm registry. If anything went wrong, I had to modify it, and publish it again. Now I can do a local npm install for such test. So I can be quite confident when I do the real npm publish. I'm not sure whether you are in my case, and I hope this could be helpful.

@rlidwka

I guess we could publish to 1.11.0+1 or similar in such cases though that would create a confusion for jQuery users

You can't. AFAIR, npm is cutting out build metadata before publishing, so it'll be the same old 1.11.0 which will fail.

@coderaiser

@elgs, Well it's a good point, I didn't know about this feature.
Anyway without force unpublish life would be much more stable.

@Bartvds

+1 on a grace period that allows to force publish for something like an hour to fix stupid mistakes and random errors.

Like how npm and github all have slightly different markdown parsers it is a bit too much to have to bump a version just to fix some markdown.

Also very handy when publishing had a hickup (like not picking up the README.md or other random glitches).

This would also be safer then allowing a certain number of republishing as the security risk is limited to the short grace period.

@mzgol

In our specific jQuery example publishing was successful and we didn't notice the problem until some time later when we got a bug report about the package. Bumping the version from 1.11.0 to 1.11.1 in such a situation is not really a solution - we're primarily a client-side library so we wouldn't release a new version just because publishing on npm failed. It would be confusing for users to understand that 1.11.1 on npm is the same as 1.11.0 outside; we'd also need to skip 1.11.1 on the next patch release. This is a lot of problems.

If you remove the -f option, I hope there will at least be a process we could go through in case of such (maybe rare but still happening) problems to be able to republish.

@WebReflection

I came here to hate everything … then I've read the whole thread and changed my mind.

package versions as if these were MD5 or SHA1 version of a binary :+1:

I still believe what @mzgol asked is more than reasonable: a process for major libraries or even some npm gotcha/problem that goes through "manual" updates or it's supervised and rare enough to allow the force … maybe keeping the -f flag with a --because=reasons and a sort of quick human admin yep/nope from npm would be great?

I understand you guys don't have an help desk though so … no way we gonna do that would be an acceptable answer.

my 0.02

@cayasso

Thanks @isaacs for this change, I have had situations before where my apps just stop working and was mainly because of a silent "publish -f" someone did on a module, so I am happy that this will not happen again :+1:

@hemanth

Me: Okies let me do a npm publish -f and @isaacs:

clap

@jamesbloomer

++many. Been broken by a republish and it was extremely annoying.

@IgorMinar

I'm all for removing npm publish -f. We need the package manager to be reliable more than anything else. Having versions be immutable is a major contributing factor to reliability.

I do understand that sometimes a deployment might go wrong. What if there was a grace period (1h?) while -f was allowed.

Alternatively keep -f but somehow verify shasum automatically so that if the dependency changed the npm install failed. (lockdown does this but you must opt in).

@garthk

Strikes me many of the concerns driving use of npm delete could be handled with a hypothetical npm deprecate, e.g. persuading upgrades for security reasons.

npm install would loudly complain about deprecated modules, but still proceed. The registry would also display warnings about a module's direct or indirect dependency on a deprecated version. (Does it do so on deleted versions?)

Would that take enough pressure off npm delete to ease some of the --force "workaround" issues raised above?

@rlidwka

Alternatively keep -f but somehow verify shasum automatically so that if the dependency changed the npm install failed

That's shrinkwrap's job. Sadly, it doesn't verify shasums, but I hope it will be changed eventually.

@isaacs
Owner

Sadly, it doesn't verify shasums, but I hope it will be changed eventually.

Yes, it is planned. Requires some extensive cleanup that hasn't risen to the level of getting done yet, sadly.

@jbach jbach referenced this issue in npm/npm
Closed

Unpublish & Publish is Failing. #4588

@isaacs
Owner

This is now done and pushed live. And the latest npm release doesn't even allow you to try to overwrite a version, even with --force.

@isaacs isaacs closed this
@Bartvds

@isaacs so the short one-hour (or whatever) grace-period didn't make the cut?

Is it out of undesirable complexity, or maybe just a bad idea? (I ask out of interest in the reasoning)

@mike-spainhower

@Bartvds Please read the thread - #148 (comment)

@isaacs
Owner

A time-based approach is problematic because it relies on PUTs being near-in-time to a user typing npm publish. Since we routinely replicate these docs all over the place, that is not a reasonable solution.

Also, since CouchDB does not treat time as a first-class thing, this could potentially open security holes.

Unpublishes are rare anyway. If someone truly needs this, then they can reach out for admin help. If we start to see a lot of support requests for this, we can perhaps work something out that is more automated.

@mikeal

yeah, the concept of a time is in direct conflict with offline conflict resolution schemes which CouchDB attempts to support.

@elgs

-1 to unpublish. If something goes wrong, just simply increase the minor version number by 1, and publish it again. There is huge (actually infinite) integer space for the version numbers.

@Bartvds

Thanks for the clarifications, makes sense (TIL :)

@isaacs
Owner

@elgs If only disk space were as unlimited as integer space :)

@RubenVerborgh

Unpublished releases still seem to show up with npm info. Is that desired behavior?

@IgorMinar

Primarily for archival purposes: there have been cases when package authors unpublished their packages without considering the impact such action would have on everyone depending on them.

For example grunt-contrib-jasmine-node was unpublished and republished as grunt-jasmine-node without any warning. This caused lots of broken builds around the world.

It's quite hard to prevent people from doing stupid stuff without disabling features that are occasionally needed. Maybe displaying a warning and asking maintainer to confirm the unpublishing by typing the name of the package would raise awareness about the possible dangers of unpublishing.

@ljharb

I know it's difficult wrt Couch, but honestly download counts should come into play here imo. If something's never been downloaded, deleting it shouldn't be restricted - i should be able to erase a package-version from existence that nobody's used.

@glenjamin

You can do exactly that, unpublish still works.

You just can't later re-use that number to mean something else.

@IgorMinar

@glenjamin I'm arguing that unpublishing can screw many people if not communicated well and done carefully for packages that people depend on. Because of this the maintainer unpublishing a package should go through extra hoops and we warned of consequences.

@MarcDiethelm

Oh boo. This change just bit me. I almost only ever use unpublish / publish to a) fix docs, b) add a tag to a pre-release. (We really could use a better way of doing pre-releases...).

So I just released a project generator with in a beta version, but messed up the tag. ...Because it's --tag foo and not --tag=foo as I expected. Sigh. I immediately unpublish and re-publish. No, I didnt know about npm tag...

You know what happens now. npm refuses to let me re-publish... the exactly same package.
Which is unfortunate because the version numbers of this package and the framework it belongs to are supposed to be in sync! (...Unless there is a genuine blocker that needs fixing.)

So yeah, something like npm restore would be incredibly useful right know, if only for the specific use case of people like me who didn't know about this change.

@MarcDiethelm MarcDiethelm referenced this issue in npm/npm
Closed

`npm publish` fails #4653

@raoulmillais

Should the advice in the FAQ on checking in node_modules for deployables be updated? I.e. Assuming a decent continuous integration/deployment pipeline, can we now consider shrinkwrap a sensible option for deployables too?

@aredridel

I consider shrinkwrap workable at this point, though once checksums are added, that'll be all the better.

@isaacs
Owner

@raoulmillais There are still advantages to checking in deps for deployed artifacts, but really, that's a very subtle set of tradeoffs that you must make for yourself. Many folks just trust version numbers as-is, or use shrinkwrap, or do a pre-deploy step that saves and verifies things, etc.

@MarcDiethelm --tag=foo should absolutely work fine. What version of npm are you using?

You know what happens now. npm refuses to let me re-publish... the exactly same package.

It's not exactly the same, though. It has a different checksum, different modified date, different published date. You may say it's the same, and I believe you, but leaving that door open is dangerous.

What package? What version? I'll unblock it for you.

@rockbot rockbot referenced this issue in npm/npm-www
Closed

Error 500 at /package/femto-logger #628

@mike-feldmeier

This is a very unfortunate change that I hope gets revoked. If want to change any metadata or readme information, I shouldn't have to version bump. If I change a paragraph in my readme, I wouldn't bump the version in git, but now I have to just to get npm to publish correctly.

@isaacs
Owner

@mike-feldmeier Working on a npm doc-publish command (or some such) that will JUST update the documentation.

@Raynos

Does npm doc-publish mutate the README in the package.json and thus on npmjs.org website ?

@isaacs
Owner

@Raynos It'd update the root doc on the registry, and on the website, without doing another publish or requiring a version bump.

@riggerthegeek

I've just gotten caught out by this. I understand the reasons behind this (and agree with it completely), but it seems strange that you can unpublish a package (thus breaking other people's stuff) and then not put it back.

How can you get around that?

@isaacs
Owner

@riggerthegeek Don't unpublish unless it is critically important to remove something from the registry. Publish the update with a bumped patch version, so they're likely to be able to pull it if they are not pinning their versions explicitly. If they ARE pinning their versions explicitly, then they'll probably be better off being broken (and noticing!) rather than silently getting changed bits.

@QETHAN

@isaacs "npm cache clean foo". it works! thank you

@dannycoates dannycoates referenced this issue in mozilla/fxa-auth-server
Closed

switch from lockdown to shrinkwrap #603

@skidding skidding referenced this issue from a commit in skidding/dragdealer
@skidding skidding Rename package version #28
You can can no longer override a published version
npm/npm-registry-couchapp#148
4fd80d4
@skidding skidding referenced this issue in skidding/dragdealer
Merged

Add to npm #28

@highsource

I have to say, this is quite unfortunate that I can't republish an existing version.

Here's my story.

A user has reported that he can't install my module (jsonix):

highsource/jsonix#6

After short investigation it appeared that the tarball for the 2.0.1 version is missing:

https://registry.npmjs.org/jsonix/-/jsonix-2.0.1.tgz

I'm no npm pro so I googled and came upon an advice "unpublish the republish an existing version". So I did unpublish and then could not publish it again.

Well, great. But when I tried to publish a new version (2.0.2), I've started getting 502 error on publish:

npm/npm#4878

The thing is that I'm getting new INVALID version in the registry even if publishing fails. But I can't republish existing invalid versions. I am forced to bump the version. It's already 7th or 8th version I had to bump in an attempt to publish a version of my package.

Is there any way to allow these invalid versions to be republished? I actually just want to publish the 2.0.1. I don't need all these version 2.0.2 and above, they appeared only due to the failed publish attempts.

@IgorMinar

@highsource in your case publish -fis not a solution but a workaround to a different issue that should be solved. It should not be possible to successfully publish a package and not have it available for download. At the same time, if publish is not successful due to some internal issue (symptom http 5xx response), the version number should remain available for republishing.

@rockbot
Owner

@highsource the situation you're describing is totally valid and 100% understandable. It's for that exact reason (i.e. a missing tarball for a specific version) that we encourage you to contact us directly at support@npmjs.com. It's our experience that only the tiniest percentage of people run into this situation, and thus we make special exceptions for them.

As for the 502 errors - please upgrade your node and npm instances (they're rather behind, unfortunately). If you need your node and npm versions to remain at their current versions, let us know in the support email you send us.

Send us an email and we'll fix it all up :-D

@highsource highsource referenced this issue in highsource/jsonix
Closed

npm installing error #6

@highsource

@IgorMinar You're totally right. I've reported the other issue here:
npm/npm#4880
Already closed since the problem does not persist anymore with 1.4.3.
@rockbot Thank you for the response! I could successfully publish with npm 1.4.3 and the latest node.js. I'll send a support mail in a few moments.
Thank you people for your kind help. :)

@blakeembrey

@isaacs Not sure if it's related to this fix, but I just found I can no longer republish an entire module after I had just unpublished it. npm unpublish is giving me Error: forbidden already unpublished while npm publish is throwing Error: forbidden Cannot replace previously published version: 0.0.1. Any ideas on a workaround?

@mzgol

@blakeembrey You need to bump a version, did you do it? Otherwise it has to fail after this change, it wouldn't make sense otherwise.

@blakeembrey

@mzgol I can bump the version, but seems counterintuitive since the module is completely gone.

@blakeembrey

@isaacs Awesome, thanks. Makes sense, just following the errors didn't make sense and it seemed like it could be a cache issue or something. Maybe an update on the error messaging could help (as well as better warning/link when unpublishing).

I had only published for a split second before unpublishing it because of a tiny typo. This was 0.0.1 so I figured it hardly mattered since I could just reset git and republish the module fresh with the correction.

@nathanaschbacher

This is ridiculous. I just published a new version of a module with:

$ npm publish http://url.to.git/module.tar.gz

Except I did it from my working directory, so npm picked up my local working directory instead of using the URL provided. I went to force re-publish... and I can't.

So now I have to version bump my module purely to deal with this bizarre new npm idiosyncrasy.

@IgorMinar
@nathanaschbacher

The command was valid. The problem was I supplied a URL that it should have used because it was explicit, but instead it first looked in my working directory for a package.json and found one so it went ahead and used that and ignored my URL argument.

I'll file another issue. However. I still don't think republishing functionality should have been removed. If I had a more complex release cycle than just myself for this project it would have been a nightmare to version bump purely to satisfy this new npm behavior.

@patriksimek patriksimek referenced this issue from a commit in pekim/tedious
@patriksimek patriksimek 0.2.1 because of npm/npm-registry-couchapp#148 badbf8a
@WebReflection

can still un-publish … But then, you will not be able to publish something else to that same package identifier

I wonder if this also means that death packages will be death forever so that time will pass and meaningful packages/modules names will be all gone … rephrasing: how to file death package and remove them from npm if nobody downloaded them for, let's say, 3 months and/or these are clearly unmaintained? Similar approach with domain or … is this even a concern?

Thanks

@jkrems

remove them from npm if nobody downloaded them for, let's say, 3 months

There's no way of telling if nobody is relying on them. The registry is replicated by different people so there are no central stats about which packages were downloaded and which are not in use anymore.

@WebReflection

I'll try again: let's say I took a package name and I am the owner, and I also want to remove it forever from the registry instead of living it there as rotten package … is this possible?

If not, this is a very bad long term plan, IMO, plus it is misleading for nom users if this package will show up after searching for it .. maybe it was a mistake, maybe it was wrong.

Thoughts? Thanks

@Raynos

@WebReflection you can unpublish.

You just cant republish over an existing version.

@rockbot
Owner

@WebReflection you can definitely unpublish. You can also start with deprecating the package and then unpublishing after a period of time, to warn people to stop using your package (and to find alternatives) before their whole system breaks.

We're talking about the best ways to show unpublished modules, either so that you can freely give up the name to someone else (because it's a good name, but you just don't want it anymore), or stop the name from ever being taken (because it's a terrible name, and no one should ever use it).

I'm totally open to hearing thoughts from the community about this - but it should probably be in a different thread ;-)

@jgerigmeyer jgerigmeyer referenced this issue from a commit in asciidisco/grunt-qunit-istanbul
@asciidisco asciidisco version bump 667aaf3
@jsdevel

@isaacs I completely understand and agree with this. npm publish should be an action of no return. It would be really cool if we could add some sort of a soft publish that would allow republishing.

Consider the use case that others have mentioned where network activity hinders a proper npm publish, or something on the back end hangs up throwing the infamous error parsing json issue.

It would probably be a lot of effort to add npm publish --soft or something similar, but I see the benefit as 2 fold:
1. soft publishes would never be used for npm install, so npm publish --soft would effectively be npm publish -f without the security / consistency issues mentioned herein.
2. completely forcing patch bumps for edge cases really messes up release history and tagging on github. I've been facing a lot of issues lately publishing node-soap that have already caused 2 releases to be lost forever on npm. Soft publishes would allow the publish process to go unhindered 99% of the way, allowing npm publish to make the referenced version live.

@jsdevel

Opened #171 for soft publishes.

@jason0x43

The registry should have a window (minutes, hours) during which a package can be republished to deal with user error. There is nothing like the sinking feeling of pressing enter only to realize that you've made a mistake somewhere, and you can't fix it in a timely fashion.

@aredridel

Yes, but that's actually hard to do with the current architecture, and doesn't solve the problem that people can end up with code that changes under them.

@jason0x43

It may be hard, but that doesn't mean it's not important. Being able to depend on SHAs rather than semvers for specific instances of packages seems like it would have been a much less disruptive way to prevent the surprise code change issue for projects that require/desire that level of lock down, and it's a more semantically correct use of the available information about a package (SemVer rule 3 notwithstanding).

@highsource
@jason0x43

I like the staging area idea. It's another checkpoint to catch mistakes, and it would take care of failed publishes.

@jason0x43

A similar, but potentially simpler, option to the staging area would be to have public and private/unlisted publishing options. A "private" package would be installable by anyone using the exact semver, but it wouldn't be listed when running npm info or installed via version range specifiers like ~1.0.0. Private packages could be overwritten as many times as necessary. Once the developer is satisfied with the package it could be made public, at which point it would be locked.

@aredridel

How about having that private version just be worked on locally, and not published until it's ready? Doesn't seem that that actually solves a problem.

@jason0x43

Say you have a project that depends on one of your npm packages, and you run CI on something like Travis. It'd be awesome if you could test your final configuration so that when you publish, you're publishing what you tested.

@aredridel

Exactly! That's what git is good at!

@jason0x43

Github in particular came to my mind. You can push anonymous commits to your repo as much as you want, and then you tag a release when it's ready to go. You don't have to work offline until you get it just right before making a final push. Of course, if you screw something up on Github you can still force push if necessary.

@isaacs
Owner

npm is not Sonatype Nexus or Maven. It has different semantics, different implementation, and serves a different platform. Expecting an exact 1-to-1 mapping of Nexus or Maven to npm is not going to happen.

Instead, let's zoom out and talk about use-cases.

You want to publish something, but not release it to the world. Only people explicitly asking for this thing should get it.

You can do that with npm, pretty easily.

In the "foo" package folder:

npm publish --tag=staging

Then, somewhere else, you can do:

npm install foo@staging

to get the beta version.

Later on, let's say you want to promote the "staging" package to become the new default install for people who do npm install foo. Do this:

npm tag foo@staging latest

Here's how this works:

  • Every package has a "dist-tags" object which maps "tags" to actual versions.
  • "tags" work equivalent to versions in most cases. npm install foo@staging is akin to doing npm install foo@1.2.3 if foo has "dist-tags":{"staging":"1.2.3"} in it. You can also have "dependencies":{"foo":"staging"} in your package.json file, and so on.
  • The default tag is "latest". If you don't specify a tag, then it uses that one. (Usually what you want is the latest published version, and in most cases, you just want to publish it as that version, so this works out.)
  • If you specify a different tag (ie, something other than "latest"), you can either use a different target for the top-level install, or even use "staging" as the default tag for all dependencies by doing npm install foo --tag=staging.

None of this necessitates re-publishing an already-used version number, and the benefits of preventing overwriting are too great to justify changing this decision in order to accommodate a use case in the same way as Maven, when npm already does accommodate that use case in a different way.

@isaacs
Owner

Note that "staging" here can be basically any arbitrary string. Only "latest" is magical.

@ljharb

@isaacs Thanks for the clear explanation of tags! Is there perhaps an environment variable that would set the default --tag for npm publish? If so, a team wanting the workflow you described, or someone wishing to prevent themselves from making a public mistake, could set their default tag to "staging" or something similar.

@jason0x43

I have relatively little experience with other package repositories, so I'm not trying to say npm should be equivalent to something else. I'm just suggesting that the experience could be improved, both for users and admins. A developer shouldn't have to burn a version number (this is not always trivial) simply because of a publishing error, and it's unfortunate that correcting mistakes requires the intervention of an admin.

Tagging sounds promising; I'll have to think about it some more. If I publish with tag "staging" (or whatever), can I publish the same version again with a new tag (until it's "latest"), or will I still have to version bump? I get the sense that that version number is still gone, which is the primary thing I'm trying to avoid.

@mzgol
@jason0x43

I understand that they're distinct concepts, although the fact that you can publish and tag at the same time (npm publish --tag foo) to avoid having a default tag applied shows that the operations can be closely related. Regardless, tagging doesn't appear to accomplish what I was going for, which is simply a way to stage a particular version of a package without having the version number irrevocably committed.

The decision to disallow updates to a particular version is going to occasionally cause problems, either due to human or technical error. This is known, and there is a mitigation strategy in place, which is to email support and ask for the version number to be manually deleted from the repository. All I'm suggesting is that there could be relatively simple technical measures in place that would accomplish the same goals without requiring 1) a version bump when package contents haven't substantively changed, and 2) the sometimes lengthy wait entailed in getting human-in-the-loop support.

@isaacs
Owner

Tagging sounds promising; I'll have to think about it some more. If I publish with tag "staging" (or whatever), can I publish the same version again with a new tag (until it's "latest"), or will I still have to version bump? I get the sense that that version number is still gone, which is the primary thing I'm trying to avoid.

No. You can never publish that same version again.

But Good news! You don't have to! Because you can point the tags at whatever published versions you want. There is no need to re-publish version 1.2.3, because it is already there, on the registry, just with a different dist-tag. It's already there. Just update the latest tag to point to it, and you're done. No need to republish.

The decision to disallow updates to a particular version is going to occasionally cause problems, either due to human or technical error.

In practice, this type of error is extremely rare, especially using the latest version of npm, because now publishes are a single atomic PUT that will either succeed or fail, 100%. There is not any need to publish the same version again. If the PUT fails, then it didn't make it to the database, and so the version number is not "taken". If it doesn't fail, then you don't have to republish, because it won't ever end up "half published", no matter what kinds of network errors might occur.

If the issue is: "Oh, no, I published version 1.2.3, but I forgot that I was supposed to have updated thus and so before doing that!" well, ok, get your processes in order. If you email support, we can help you with that. Or you can just accept it, and bump the version number.

Regardless, tagging doesn't appear to accomplish what I was going for, which is simply a way to stage a particular version of a package without having the version number irrevocably committed.

Sorry, that is not how npm works. A name@version is a specific thing. It cannot be changed to be a different thing. See the entirety of this thread, and the associated blog post, for more reasons why this is.

If you want to "stage" a publish in this way, then you can do this with git tags or branches quite easily. npm install "git://github.com/jason0x43/project-name.git#1.2.3" will install the 1.2.3 commit-ish (tag, branch, etc.) of that repo. You can force push to github all you like. Once it goes to npm, that version number is taken. Please plan accordingly. Thanks.

@jason0x43

I've read the thread and the blog post, and can see the reasoning behind this change. It just seems like that goal could have been achieved without entirely taking away a valuable ability from developers or necessitating significant changes to the registry's architecture. I suppose I assumed this kind of issue would be more common than it apparently is, though, so it probably wasn't worth the effort. Ah well. I'm not opposed to documentation trails or human intervention, but it's not nearly as fast as being able to self-correct.

Is support@npmjs.com the best place to go for admin intervention?

@highsource

Tagging plus atomic publishes probably solve my problems.

@jason0x43

Well, if nothing else I have a better idea of how tags and versions work now. :) One workflow that seems to work well is to always use patch versions for your package, and use tags to denote canonical versions. So you'd use 0.1.1-1, 0.1.1-2, etc. for the package, and just tag the last one as 0.1.1. That lets both 0.1.1 and ~0.1.1 work properly as dependencies.

@bryevdv

Is there a way to undo an unpublish? New to npmjs and just getting started, there was zero warning or (y/n) confirmation when doing an "npm unpublish" that this irrevocably discarded a version number. Perhaps it would be a good idea to warn and ask confirmation in such a drastic case? I was just trying to update the docs/README because they were malformed. In any case the project is:

https://www.npmjs.org/package/bokehjs

which is the front-end component of the Bokeh (http://bokeh.pydata.org, https://github.com/ContinuumIO/bokeh) library. The js version and the the python version are synced, so a different version number is not feasible.

@isaacs
Owner

I'm sorry for this. A confirmation on unpublish is a good idea.

No, there is not a way to do this. Once published, the version number is taken, for good. You can remove it, but you can't ever re-use it. The costs of allowing this are simply too great.

The only viable solution is to either bump the version of the python lib as well, or design Bokeh so that it doesn't rely on the versions being identical.

@bryevdv

It's sad to see doctrinaire policies become an excuse to abdicate real judgment in all instances.The original upload was also only playing around to gain familiarity with npmjs. I will extend my suggestion to warnings and confirmations on publish as well, so that your users may have some indication, any idea at all, that the publication is also an irrevocable action from which they may not recover simple issues.

@ljharb

It makes sense not to allow differently publishing to an unpublished version - but is there any way to revert an unpublish?

@isaacs
Owner

It makes sense not to allow differently publishing to an unpublished version - but is there any way to revert an unpublish?

Not without significant human intervention.

@bryevdv Perhaps I misunderstood, if so, I apologize. Is there something you would like un-un-published? Can you please email support@npmjs.com with the details?

@bryevdv

@isaacs I sent a message there the other day but I will resend thanks.

@shawnzhu shawnzhu referenced this issue from a commit in drone/drone
@shawnzhu shawnzhu support publishing to npm d0628bb
@felixgirault felixgirault referenced this issue from a commit in essence/essence.js
@felixgirault felixgirault Bumped version
I fucked up and can't republish the broken 0.1.0 version
(npm/npm-registry-couchapp#148)
afd2b46
@isaacs
Owner

This issue is resolved. If you have a rare issue that requires re-publishing over an old version of a package, please email us at support@npmjs.com and explain your situation.

However, note that if you are tying the npm version to some external versioned thing, such that libfoo 1.2.3 MUST be published to npm as foo-js@1.2.3 or else it won't work, then that is a brittle design. It's better to have the external package's version incorporated into the version somehow, so that the two are not so intimately tied to one another.

The change discussed in this issue has been in production for several months. Locking it now, so that further discussion is directed to other channels.

@isaacs isaacs locked and limited conversation to collaborators
@isaacs
Owner

UPDATE 2015:

Back in the winter and spring of 2014, the npm registry was a much smaller and simpler piece of technology.

Now, it is much more dangerous and difficult to properly remove every trace of a package in such a way that it does not introduce data conflicts. Removing and re-publishing old versions is not supported.

If your situation is truly unique, and for some reason it is fundamentally impossible to bump the version number and proceed, please let us know by emailing support@npmjs.com.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Something went wrong with that request. Please try again.