Better instructions for running a private repo #106

Closed
celer opened this Issue May 28, 2013 · 21 comments

Projects

None yet

7 participants

@celer
celer commented May 28, 2013

Hi,

So we've been trying to setup our own NPM repo for private packages, but with no success. (We have very little couchdb experience, but it shouldn't matter - we don't want to become experts in couchdb, we just want a private NPM repo!)

Our goal is to publish just a few packages in our private repo and then use a mix of public and our private packages for application development.

When following the directions here: https://github.com/isaacs/npmjs.org we are unable to get it to work.

Our assumptions

  • That it is possible to have a private repo without replicating the public repo
  • We have no defined admin user for couchdb
  • The steps we've omitted are for replicating the public repo
npm@npm:~$ cat /etc/issue.net
Ubuntu 12.04.2 LTS
npm@npm:~$ couchdb -V
couchdb - Apache CouchDB 1.3.0
npm@npm:~$ node -v
v0.10.8

Our config file for couchdb looks like so:

[couchdb]
uuid = 08daf17661b67a76a671f66ea031d88b
[httpd]
[httpd_global_handlers]
[couch_httpd_auth]
[log]
[os_daemons]
[daemons]
[ssl]
verify_ssl_certificates = false
ssl_certificate_max_depth = 1
[vhosts]
[update_notification]
[admins]

Notice that we have no admin users defined

npm@npm:~$ curl -X PUT http://localhost:5984/registry
{"ok":true}
npm@npm:~$ git clone https://github.com/isaacs/npmjs.org.git
Cloning into 'npmjs.org'...
remote: Counting objects: 1291, done.
remote: Compressing objects: 100% (742/742), done.
remote: Total 1291 (delta 609), reused 1200 (delta 531)
Receiving objects: 100% (1291/1291), 649.70 KiB | 1.10 MiB/s, done.
Resolving deltas: 100% (609/609), done.
npm@npm:~$ cd npmjs.org
npm@npm:~/npmjs.org$ sudo npm install couchapp -g
... (no error)
npm@npm:~/npmjs.org$ npm install couchapp
... (no error)
npm@npm:~/npmjs.org$ npm install semver
... (no error)
npm@npm:~/npmjs.org$ couchapp push registry/app.js http://localhost:5984/registry
Preparing.
Serializing.
PUT http://localhost:5984/registry/_design/scratch
Finished push. 1-8b4b7cde241179296b34d437d6fcbec3
npm@npm:~/npmjs.org$ couchapp push www/app.js http://localhost:5984/registry
Preparing.
Serializing.
PUT http://localhost:5984/registry/_design/ui
Finished push. 1-342abe0ba85f3fbb4f23eda78fe27a46
# Skip synchronization task as we do not want to replicate the public repo
npm@npm:~/npmjs.org$ npm config set registry http://localhost:5984/registry/_design/app/_rewrite

These steps have been gleamed via reading bug reports and should be added to the documentation if they are correct:

npm@npm:~/npmjs.org$ npm config set _npmjs.org:couch=http://localhost:5984/registry
npm@npm:~/npmjs.org$ npm start
npm WARN package.json npmjs.org@1.0.1 No repository field.

> npmjs.org@1.0.1 start /home/npm/npmjs.org
> ./push.sh

/home/npm/npmjs.org/node_modules/.bin/couchapp
Preparing.
Serializing.
PUT http://localhost:5984/registry/_design/ghost
Finished push. 1-63e60faadf6c7483dd1d68c1ef1fee37
Preparing.
Serializing.
PUT http://localhost:5984/registry/_design/scratch
Finished push. 2-fb48ce9e7c7e674f89030c5569a8b6d8

Pushed to scratch ddoc. To make it real, use a COPY request.
Something like this:

curl -u "$username:$password" \
  $couch/registry/_design/scratch \
  -X COPY \
  -H destination:'_design/app?rev=$rev'

But, before you do that, make sure to fetch the views and give
them time to load, so that real users don't feel the pain of
view generation latency.

You can do it like this using npm:

    npm run load
    # Go get lunch and come back.
    # Go take the dog for a walk, and play some frisbee in the park.
    # If you don't have a dog, go get a puppy, provide it with love
    # and appropriate discipline until it gets older, train it to play
    # frisbee in the park, and then take it to the park and play frisbee.
    # Wait some more.  It's probably almost done now.
    # Make it live:
    npm run copy

It is unclear if we need to run npm run load and npm run copy as we do not wish to mirror the public repository - are these steps required???

Regardless let's see what happens when we try to create a private package

npm@npm:~/npmjs.org$ cd ..
npm@npm:~$ mkdir testpkg
npm@npm:~$ cd testpkg/
npm@npm:~/testpkg$ npm init
This utility will walk you through creating a package.json file.
It only covers the most common items, and tries to guess sane defaults.

See `npm help json` for definitive documentation on these fields
and exactly what they do.

Use `npm install <pkg> --save` afterwards to install a package and
save it as a dependency in the package.json file.

Press ^C at any time to quit.
name: (testpkg)
version: (0.0.0) 0.0.1
description: testpkg
entry point: (index.js)
test command:
git repository:
keywords:
author:
license: (BSD)
About to write to /home/npm/testpkg/package.json:

{
  "name": "testpkg",
  "version": "0.0.1",
  "description": "testpkg",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "repository": "",
  "author": "",
  "license": "BSD"
}


Is this ok? (yes) yes
npm WARN package.json testpkg@0.0.1 No repository field.
npm WARN package.json testpkg@0.0.1 No readme data.
npm@npm:~/testpkg$ npm --registry  http://localhost:5984/registry add-user
Username: npmtest
Password: (npmtest)
Email: npmtest@npmtest.test
npm http PUT http://localhost:5984/registry/-/user/org.couchdb.user:npmtest
npm http 201 http://localhost:5984/registry/-/user/org.couchdb.user:npmtest
npm@npm:~/testpkg$ cat ~/.npmrc
_auth = bnBtdGVzdDpucG10ZXN0
registry = http://localhost:5984/registry/_design/app/_rewrite
_npmjs.org:couch = http://localhost:5984/registry
email = npmtest@npmtest.test
npm@npm:~/testpkg$ npm publish
npm WARN package.json testpkg@0.0.1 No repository field.
npm WARN package.json testpkg@0.0.1 No readme data.
npm ERR! publish Failed PUT response undefined
npm ERR! Error: login error
npm ERR!     at RegClient.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/npm-registry-client/lib/request.js:82:22)
npm ERR!     at CouchLogin.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/npm-registry-client/node_modules/couch-login/couch-login.js:177:14)
npm ERR!     at CouchLogin.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/npm-registry-client/node_modules/couch-login/couch-login.js:159:46)
npm ERR!     at Request.self.callback (/usr/local/lib/node_modules/npm/node_modules/request/index.js:148:22)
npm ERR!     at Request.EventEmitter.emit (events.js:98:17)
npm ERR!     at Request.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/request/index.js:876:14)
npm ERR!     at Request.EventEmitter.emit (events.js:117:20)
npm ERR!     at IncomingMessage.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/request/index.js:827:12)
npm ERR!     at IncomingMessage.EventEmitter.emit (events.js:117:20)
npm ERR!     at _stream_readable.js:910:16
npm ERR! If you need help, you may report this log at:
npm ERR!     <http://github.com/isaacs/npm/issues>
npm ERR! or email it to:
npm ERR!     <npm-@googlegroups.com>

npm ERR! System Linux 3.2.0-38-virtual
npm ERR! command "/usr/local/bin/node" "/usr/local/bin/npm" "publish"
npm ERR! cwd /home/npm/testpkg
npm ERR! node -v v0.10.8
npm ERR! npm -v 1.2.23
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR!     /home/npm/testpkg/npm-debug.log
npm ERR! not ok code 0

I guess we have a few a requests:

  • Updated documentation showing the process from end to end
  • Unit test for standing up a private repo (we can provide a basic shell script for this, if we knew what the process was supposed to be)
@spunone
spunone commented Jun 6, 2013

It would also be useful to know the right way to leverage public_users rather than _users to overcome the issues with _users no longer being world readable. Should the registry point to ghost, app, scratch or something else?

Is there some other forum where people are discussing this? As node gains traction in the corporate world, the need for a private repository will grow. Right now, it's impossible to create and maintain a private repo effectively.

@celer
celer commented Jun 6, 2013

This is a huge problem for us from a corporate stand point as well, it really hinders our ability to get node leveraged within side of our organization. Sadly we never got any further then discussed here.

Essentially it is one more tick against using node inside of large organizations :(

@spunone
spunone commented Jun 6, 2013

Have you tried using the repo after you push the couchapps? I've been able to run them with some success without doing any of the npm run load stuff, though I have been replicating from the public repo. In one of those bug reports, I believe it was mentioned that Couch 1.3.0 doesn't need the view generation tricks since it handles it on its own.

@spunone
spunone commented Jun 6, 2013

Another thing I found I needed was to allow insecure rewrites in couchdb. I found a little bit of information here, though it doesn't go into nearly enough detail:

http://clock.co.uk/tech-blogs/how-to-create-a-private-npmjs-repository

@aakn
aakn commented Aug 12, 2013

I've run into the same problem. I've spent the entire day trying to setup a private repo but with no luck.
DId you figure it out yet?

@celer
celer commented Aug 12, 2013

I never got it working, so I gave up. This is still an blocker for corporate adoption of NodeJS.

@spunone
spunone commented Aug 12, 2013

Pretty much. Nodejitsu has a commercial service hosting NPM they'd be willing to sell you.

What problem specifically are you experiencing? Couch constantly crashing when you try to replicate from the public repo, perhaps?

@aakn
aakn commented Aug 12, 2013

Exactly the same problem as OP. says login error when I try to publish it to the repo.

[I have the same requirements as OP. Need a private repo in the organization, but don't want to replicate the public repo.]

@celer
celer commented Aug 12, 2013

It has been a while, but if I recall it was that I couldn't ever get it to authenticate.

I didn't want to replicate the entire public repo, and I just needed to host or own NodeJS modules in it. We got around it in a very cruddy fashion by putting our modules up on an HTTP server and using the URL in the package.json to fetch them.

But this was just for our team, for inter-team interactions having a NPM repo in-house would be a requirement for corporate adoption.

@spunone
spunone commented Aug 12, 2013

Have you setup a continuous replication that copies _users to public_users? CouchDB made a change so that _users isn't world readable anymore. I believe the latest version of NPM will look for public_users.

And yeah, having to keep an entire copy of the repo in tow continues to be an annoyance.

@aakn
aakn commented Aug 12, 2013

@celer yeah, thats the requirement.

@spunone just setup the continuous replication. Doesn't seem to make a difference.

The weird thing is the only request that shows up in the couchdb logs is a POST /_session 401. there is no request for _user or public_user

@spunone
spunone commented Aug 12, 2013

And your npm client is configured to hit your private repo with the right username? Did you create a user in your private repo? Can you see it in _users and/or public_users?

Maybe try this npm config option, it's helped us in a few situations:

npm config set always-auth=true

@aakn
aakn commented Aug 12, 2013

so after running npm config set always-auth=true, I get the following error when i try to publish

npm ERR! publish Failed PUT response 401
npm ERR! Error: unauthorized Name or password is incorrect.: registry/testpkg

I checked the _users db and it doesnt have my user name in it.

I used the following command to create a user:

npm --registry  http://localhost:5984/registry add-user    

My .npmrc file:

cat ~/.npmrc                                                   
_auth = YWxpOmFsaQ==
registry = http://localhost:5984/registry/_design/app/_rewrite
http://localhost:5984/registry/_design/app/_rewrite = 0
email = ali@test.com
_npmjs.org:couch = http://localhost:5984/registry
always-auth = true
@spunone
spunone commented Aug 12, 2013

Have you tried doing an npm login?

@aakn
aakn commented Aug 13, 2013

Got it working. Had to use this for creating a new user

npm --registry http://localhost:5984/registry/_design/scratch/_rewrite adduser
@risingfish

+1 for this request. There are not a lot of resources out there right now for running your own npm server.

@scherler

hi all,

first of all thanks to the npm and node community for my new "love".

I have a mirror working but VERY limited see #117 for details.

My main problem for the recommendation to use npmjs.org.git as a private repo is that I think it is not build do be that. As a summary I found that is awesome to have your own mirror of the official repo, but I guess that is not what somebody is looking when creating a private rep.

I have a strong background on java so maven reps are the things I am comparing the whole registry stuff. Now if you you look into something like http://archiva.apache.org (to name one from many alternatives), it is a "repository management software that helps taking care of your own personal or enterprise-wide build artifact repository".

The main concept is different since the idea that you can configure a chain of repositories to resolve against. Where the mirroring is on demand. Let us see an example:

  1. I need dependency a
  2. I contact the the rep
  3. the rep looks whether the dep is cached and returns the version if so
  4. if not it works the chain of public/private reps to look for the dep
    • if it can be found in a public rep it got cached in the public mirror
    • if it cannot resolved any public one we look whether the user can access the private rep chain (yes you can have various private reps ... one for clients, one for dev only, ...

Now in npmjs.org and npm in general there is no concept of repository chain. You need one (!) registry that stores all your resources. Further npmjs.org is not designed to be "protected", see #117 for background but imagine that you protect your _user and registry to only work with your corporate users. The main problem that you will run into is that

  1. The rewrite rule is in a "protected" database "registry" meaning to actually be redirected to the _user db you need to be already authorized. Chicken/egg problem.
  2. To overcome that fact you need to use --registry http://dev:pass@testing.es:5984/registry/ which basically auth you against the "registry" which then will do the rewrite to auth you against the actual _user
  3. However using dev:pass in the url renders security senseless since it is plain text and any sniffer will know your pass.

My conclusion is that is not possible to have real private AND protected repository with the current concept, since the above chicken-egg problem.

One solution to our "chicken" I see is

  • create a wrapper for the rewrite part from testing.es:5984/registry/_design/app/_rewrite/ and exposing it somewhere like testing.es:5984/rewrite/.
    This would then only be used for login the in the user and to get the _auth with

    npm login --registry http://testing.es:5984/rewrite/

  • After this in theory we worked around the chicken egg. However if you look again in #117 one problem may still remain (but I think it is more a general bug then any other thing)

npm verb adduser back [ [Error: forbidden Only _admin may edit roles: registry/_design/app/_rewrite/-/user/org.couchdb.user:dev/-rev/8-73e74732a42155a1fb8dddfec9fc68ca],
npm verb adduser   { error: 'forbidden', reason: 'Only _admin may edit roles' },
npm verb adduser   '{"error":"forbidden","reason":"Only _admin may edit roles"}' ]
npm WARN adduser Incorrect username or password

This is another big problem I run into, if you bound your security for dbs to roles and store them in the _user db in each user, this roles will be nuked once you do a npm login I assume it is due to the fact that your are doing an inject each time you login instead a "simple" comparison of user:pass but I am not sure whether this due to the way couchdb works or the way npm login works. However fact is that roles are not written, meaning each time you do a login you need to set again roles.

Bottom line I think a fair warning for people looking into create they private registry is to at least point out that you will never be able to protect the registry/_user with couchdb . Meaning would need to implement a prior proxy tier for user auth and make sure that your rep is only accessible through the proxy tier.

I have to second the many voice in this thread that:

  1. for a corporate environment the current state of "private" repository is unacceptable.
  2. the replication way is further more then problematic since ATM we are at around 63GB but as more people start to adopt node that rep can quickly explode and consume way too many resources.

This limitation are a barrier for companies to adopt node and npm and it will be very bad for the future of node, since many of us devs are bound in cooperate work and need to obey the security rules of our companies.

I am not sure whether it is in the interest of npmjs.org to provide a system similar to archiva, nexus, ... where you can use your private reps in an efficient way with the global public one.

@scherler

Actually regarding the login problematic I may found a solution ...see #117

@scherler
scherler commented Sep 3, 2013

with npm/npm#3853 the only thing that is missing to close this issue to update the docs and add a section about using the updated version.

The above fixes all my mirror problems. I do not need to use --registry http://dev:pass@testing.es:5984/registry/ anymore.

However to really close this issue I think we need to allow multiple repositories where we are resolving the deps, but since I already have a 63 gb mirror, it is very low priority to implement a chain resolver for registries for me.

@rlidwka
rlidwka commented Sep 29, 2013

When following the directions here: https://github.com/isaacs/npmjs.org we are unable to get it to work.

My advice is: don't. CouchDB is used when you really need all that 60Gb of packages and performance starts to be a bottleneck. If you want to setup a private registry for dozen of packages, there are simpler solutions.

I'm developing sinopia server that aims to do just that (feature requests are welcome), and there's a handful of links to other ideas/projects at the bottom of the readme.

The main idea is: instead of replicating the entire database, you can use a software that proxies all relevant requests to npmjs.org keeping in the local database only a small number of packages you really use. Well, I guess it's similar to "chain of repositories" described before.

@terinjokes
Member

New documentation went out to the README last month, that vastly improves the the process for setting up a private registry with or without replication. Setting up a "chain of repositories" must still be done via a proxy or delegation system.

@scherler While _users is no longer world-readable, I think the issue you're getting at is there's no way to easily lock down the db, e.g. to prevent people creating new users? It might be worth creating a new issue specifically to discuss this.

@rlidwka Thanks for the contribution to the ecosystem.

@terinjokes terinjokes closed this Mar 4, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment