Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Git urls don't allow semver ranges #3328

Closed
NickHeiner opened this Issue Apr 8, 2013 · 41 comments

Comments

Projects
None yet

According to the documentation:

Git URLs as Dependencies
Git urls can be of the form:

git://github.com/user/project.git#commit-ish
git+ssh://user@hostname:project.git#commit-ish
git+ssh://user@hostname/project.git#commit-ish
git+http://user@hostname/project/blah.git#commit-ish
git+https://user@hostname/project/blah.git#commit-ish

The commit-ish can be any tag, sha, or branch which can be supplied as an argument to git checkout. The default is master.

I would like to be able use the full semver range syntax for the commit-ish part:

git://github.com/user/project.git#1.0.x
git://github.com/user/project.git#~2

I'm dependent on git urls because my company uses a number of repos on our internal github instance, and we don't have our own npm registry. (Based on digging through some old issues, it wasn't even clear that setting up our own npm registry would be possible.)

Could I submit a PR for this?

Owner

isaacs commented Apr 8, 2013

If you want that to work, then tag or branch so that 1.0.x points to something in git. We're not going to walk through commits finding semver-ish-looking things and try to be clever about them.

@isaacs isaacs closed this Apr 8, 2013

Member

edef1c commented Apr 9, 2013

@isaacs We can walk through tags, though? I believe I proposed this to you in the past.

Owner

isaacs commented Apr 10, 2013

Yeah, it's not a good idea, though. Very complicated and weird. If you have loose git deps, then point at a branch, rather than a commit.

Write a pre-procressor for package.json, call your sourcefile clever-package.json or so. It's a known fact that pre-processors are the solution to just about any gripe you have with the syntax for your platform of choice. ;)

But seriously, the pre-processor could fix up the package.json for you so that it points to a real commit. It'd be certainly a useful little tool, if open source.

Ideally, npm would have a "preinstallpackages" script hook for this. Then you can be sure that the generated package.json is always up to date before it gets evaluated.

bcherny commented Jul 16, 2014

@isaacs It's certainly not a trivial implementation, but it would be very nice to have feature parity with bower here.

Contributor

rlidwka commented Jul 16, 2014

It's certainly not a trivial implementation

If we're talking about github only, it's trivial. I can't point out the exact commit, but it was done in visionmedia/npm and yapm a long time ago using github api and gh-lookup. Just run yapm install visionmedia/commander.js@^1.2.0 and see for yourself.

Actual git url's are more tricky than that, because you have to fetch the entire tree before you know what commit hash you need, and there are a few corner cases. But people usually look just for github repos anyway.

Amusingly, although I opened this issue originally, I've now come to agree with @isaacs that this would be "very complicated and weird". I would advise people to just use an npm registry instead of conflating source control with published package management.

@NickHeiner How's that working out with your private packages? Even today, years since npm started, private package hosting is only just in beta.

Contributor

rlidwka commented Jan 30, 2015

Even today, years since npm started, private package hosting is only just in beta.

npm private package hosting isn't the only one hosting available out there. I'm using private packages for 2 years now, and it works just fine.

But git urls are useful for public stuff, if people don't want to use npm registry for one reason or another.

npm private package hosting isn't the only one hosting available out there. I'm using private packages for 2 years now, and it works just fine.

Link to service provider?

Contributor

rlidwka commented Jan 31, 2015

@jasonkuhrt , if you are looking for a npm-compatible private registry installed on your servers, look at sinopia. If you're looking for 3rd party service provider (SaaS), check cnpm.

@jasonkuhrt, we are hosting our own copy of the npm registry on couchdb. It's painful to maintain at times but generally works.

Aaike commented Mar 3, 2015

how come jspm is able to install git packages using a semver range ? can the same technique not be used ? i think it's just getting the tags using git ls-remote --tags and then do the semver match. no need to over complicate it ?
it would really be fantastic to be able to specify a semver range for github packages...

ciekawy commented Mar 16, 2015 edited by iarna

being impressed by the richness of node packages and the well working idea of reusable modules I am convinced that this is the very case where having an option of working semvers for git urls is worth extracting the job already done for bower.

dehru commented Mar 17, 2015

bower.json has no problem with this syntax

"thingy": "http://thingy.lss.com/scm/cuc/emc-login.git#1.x.x",

I wish I could add a similar syntax to package.json.

ciekawy commented Mar 17, 2015

Contributor

othiym23 commented Jul 16, 2015

GOOD NEWS, everybody! It may not work exactly like it does in Bower, but adding this functionality is now on npm's roadmap for the near- to medium-term, as part of making GitHub-hosted repositories work more like registry-hosted packages.

@othiym23 othiym23 reopened this Jul 16, 2015

It may be more related to #6236, but when this is addressed, it seems like npm update should always reset to the specified commit-ish even if it's not a semver range. Ideally npm install would as well (it's still confusing that those are two different things instead of one idempotent operation)

I'd really love to see this added. Would make git installs way more robust. What's the timeline?

gasteve commented Dec 12, 2015

This is badly needed.

d3mi3n commented Jan 12, 2016

is this still part of the roadmap?

Member

KenanY commented Jan 12, 2016

Contributor

othiym23 commented Jan 12, 2016

Yes, with the possibility that we may not get to it this year. We're in the early stages of pretty drastically trimming down the CLI's roadmap to match what we reasonably believe we can deliver in a 6-12 month timeframe. Expect to hear more about this from the team soon.

raveclassic commented Feb 16, 2016 edited by iarna

Currently I need to maintain several proprietary projects and libraries in local bitbucket, and the only way to reference them with support of semver (it is crucial, because it's an overkill to raise version in each project on library patch) is using bower which is very painful to support along with npm.
Guys, this is needed realy badly...

Glad to hear this is happening. My usecase is similar to @raveclassic's one.

dimfeld commented Feb 26, 2016

@raveclassic @filipesilva I'm achieving this by maintaining multiple tags in git for each revision. For example:

git commit package.json -m "v1.5.3"
git tag v1.5.3
# Update existing major/minor version tags to point to the new commit.
git tag --force v1.5
git tag --force v1
git push --tags --force

Then in the other project's package.json, I get semver-ish functionality by just referencing the v1 or v1.5 tag:

"dependencies": {
    "mypackage": "git+ssh://server/mypackage.git#v1",
}

First, the obvious problem: changing things already pushed to a git repo is inadvisable. My team finds it to work ok here since it's only overwriting tags instead of full commits, but your tolerance of such a practice may vary.

More importantly, if the package is already installed in a project as described above and you want to bump the version, npm install on its own will not pick up the changed tag, as npm reasonably assumes that established things in the repository aren't changing out from under it. npm install --force mypackage will refetch the repository and pick up the new version though.

EDIT: I just discovered that npm update actually does fetch the updated tag from the repo and update appropriately, so the above isn't as much of a problem now.

So, this isn't perfect, but we find it works reasonably well in the absence of full support from npm. Hope it helps.

@dimfeld Thanks for the tip, but unfortunately this won't help with version ranging with ^ and ~ which is critical.

UPD: Moreover, we have force push restricted in repos

@dimfeld thanks for pitching in your strategy! The team I'm in is a small one and as is atm, I think moving the tags would be more complicated (and thus error prone) for everyone to grok correctly than just being thorough with updating platforms.

Experience might show me otherwise, and in that case your strategy is probably the soundest one. The only other sane alternative is branches for minor version ranges, as someone suggested earlier.

dimfeld commented Feb 29, 2016

Yeah, I agree it's a high-maintenance strategy... It could maybe be automated using a post-commit hook that pulls the package version from package.json and updates the tags if needed. https://coderwall.com/p/mk18zq/automatic-git-version-tagging-for-npm-modules may be a good starting point, though I haven't tried it yet.

As far as ^ and ~ compatibility, my thinking was that using the vX tag is similar to using ^X.Y.Z, in that it get the latest version with a matching major version , and the vX.Y tag is like using ~X.Y.Z, as it gets the latest version matching the major and minor versions. When you update a library from 1.5.7 to 1.6.0, for example, the v1.5.7 and v1.5 tags would not be updated, so puling in the library at tag v1.5 would continue to be similar to semver matching on ~1.5. Of course, this is a simplified model compared to the full power of semver matching, and there may be some subtleties you're thinking of that I'm forgetting. And if you're unable to force push it's a moot point anyway.

Ultimately, I agree fully that having official support for semver ranges on git urls would be much better :)

Honestly, although I highly support npm adding this possibility, why doesn't GitHub itself do this?
https://github.com/npm/npm/releases/tag/v3.8.x would be rather awesome. The only caveat would be that a tag could literally be v3.8.x and in that case, you'd have a pretty poor naming convention and you just don't deserve any better.

Contributor

arikon commented Mar 23, 2016

@StephanBijzitter This should be done in npm, not in Github.

Contributor

othiym23 commented Apr 21, 2016

The CLI team still intends to ship this as part of giving hosted Git dependencies as much parity with registry dependencies as is practicable. If somebody wants to take a stab at implementing this, it's mostly in the caching logic (and fetchPackageMetadata) where things would need to change. Thanks to all for your patience!

cellvia commented Sep 11, 2016 edited

thank you @dimfeld i ended up implementing the same thing for my team as well, only difference is i opted to use v1.1.x and v1.x.x notation to help make obvious to them whats going on with the semver-ing ... works like a charm til semver on git urls is officially supported

as far as automation, i have the script that re-creates those tags run anytime i do a "npm run publish-{major,minor,patch}" along with testing, babel transpiling, npm version, etc

dimfeld commented Sep 11, 2016

I'm glad it's working well for you @cellvia. Good idea with the v1.x.x style notation. I think I'll start using that too.

Contributor

sankethkatta commented Dec 24, 2016

@othiym23 I took a stab at implementing this over at #15308. Would love to get your feedback!

If you really need this feature you can just switch to Yarn utility instead of NPM. It supports regular package.json and has this feature already.

Example

$ yarn add visionmedia/commander.js#^1.2.0
yarn add v0.20.3
info No lockfile found.
warning No license field
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
success Saved 2 new dependencies.
├─ commander@1.3.2
└─ keypress@0.1.0
warning No license field
Done in 1.98s.

@ondratra That's exactly what we've ended up with.

Collaborator

npm-robot commented Jun 19, 2017

We're closing this issue as it has gone thirty days without activity. In our experience if an issue has gone thirty days without any activity then it's unlikely to be addressed. In the case of bug reports, often the underlying issue will be addressed but finding related issues is quite difficult and often incomplete.

If this was a bug report and it is still relevant then we encourage you to open it again as a new issue. If this was a feature request then you should feel free to open it again, or even better open a PR.

For more information about our new issue aging policies and why we've instituted them please see our blog post.

@npm-robot npm-robot closed this Jun 19, 2017

Worth noting this is supported now: https://github.com/npm/npm/releases/tag/v5.0.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment