Add search functionality

[yannickcr]

I've implemented the search functionality, it should fix #36

Basically, it will fetch package list from npm and simply append the local ones in the response.

This patch:

• add a new route to match the search queries (/-/all/*).
• add a new get_recent_packages method in local-storage to list the packages more recent than a given timestamp.
• add a new search method in storage to run the request to npm and to append the local packages (found with get_recent_packages) to the response.

[rlidwka]

My biggest concern here is DoS. With this patch anyone who have access to /-/all can crash the process by creating several requests because of buffering.

Does it make sense to include local packages there? Maybe just self.uplinks[uplinks[0]].request(...).pipe(response) would suffice (by which I mean piping output from the first uplink as is without changing it)?

[yannickcr]

Sinopia is more likely used on a private network than on a public one so I don't see DoS as a real issue.

We could use the find *nix command in get_recent_packages to directly get the new files list, and not loop around all the files to filter them after. I think it would improve the file listing but sadly this command is not available under Windows. Maybe we can make some OS conditions here.

About the local packages, our needs are to be able to search on the npm registry but also to be able to find the private packages we published on Sinopia.

[danyshaanan]

I think I'm in a similar situation as @yannickcr . I want to be able to search private packages, (even: want to be able to search only private packages), and since this will most likely be internal, I don't care about DDOS. But if that's a problem in general, it could be only enabled in the config, or even (config.yaml):

search: local_storage storage npmjs

I'd be happy to contribute here if needed.

[phillipj]

+1

Would really appreciate being able to search for packages! Also willing to contribute if needed?

[leftieFriele]

+1 search would be awesome

[mllrjb]

+1 on searching private. @rlidwka - thoughts?

[ghost]

+1 on searching private. Not concerned with DoS here either.

[ecowden]

Another +1 here. Would looove to see this feature. 👍

[jalateras]

great. I will give this a go

[rlidwka]

Thanks, merging. I also changed it a bit (see commits before), most notably adding access control (with your version anybody who have access to "" package would see anything). Please check that out.

There is still a lot of issues with this, for example it polls only the first uplink (if it's unreachable, returns {} which confuses npm a lot), but I suppose it would work for now.

(sorry for the delayed response, was on vacation for a month)

[yannickcr]

Great ! Thanks.

[jmls]

perhaps I'm misunderstanding something, but I can't get the search to work .

https://registry.npmjs.org/-/_view/byKeyword??startkey=["http"]&endkey=["http",{}]&group_level=2

works fine, however,

http://dev.npm.myrepo/-/_view/byKeyword?startkey=["http"]&endkey=["http",{}]&group_level=2

doesn't

am I missing something, or have I completely misunderstood what the search function does in sinopia ? thanks!