Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Request: get rid of LevelDB #1925

Closed
rix0rrr opened this issue Apr 20, 2020 · 6 comments · Fixed by #2043 or #2145
Closed

Request: get rid of LevelDB #1925

rix0rrr opened this issue Apr 20, 2020 · 6 comments · Fixed by #2043 or #2145

Comments

@rix0rrr
Copy link

rix0rrr commented Apr 20, 2020

Is your feature request related to a problem? Please describe.

Right now, the Verdaccio local storage plugin uses LevelDB to store tokens.

This is a rather heavyweight dependency; it requires compiling a native extension using node-gyp (which itself requires Python and a compiler).

It also means that using Verdaccio out of the box on a plain Windows machine is out of the question, and makes Verdaccio a pain to use in CI/CD setups where we use simple plain Windows images. I'm now forced to build a pipeline to manage my Windows testing image.

What's worse, Verdaccio seems to use simple gets and puts in a way that doesn't seem very performance sensitive, so no need for a compiled component. A simple JS-powered storage solution would do the just just as easily.

When LevelDB was introduced here: verdaccio/local-storage#168 there were some initial misgivings, but they were ultimately put aside.

Describe the solution you'd like

Use a pure-JavaScript KV store so I don't need to install Python and a C compiler before I can use Verdaccio. If you're concerned about performance, make the token storage backend configurable, or opportunistically use leveldb (turn leveldb into an optional dependency and use it if it has been installed).

Describe alternatives you've considered

I've considered not using Verdaccio, but you seem to be the only game in town so that would mean reimplementing my own NPM registry and I really don't want to do that.

@juanpicado
Copy link
Member

Verdaccio uses node-bunyan which also I think requires node-gyp. Somehow, Token feat was introduce as an experimental feature ( it is disabled by default) and of course that requires node-gyp. There was a lot of discussions back then and the main objective was get feedback, so, thanks. I'm completely open to change things but I have no interested to invest time on this because I don't have to. Are you willing to PR a posible solution?

In the best scenario would be a breaking change so until Verdaccio 5 arrives we could not change it.

@rix0rrr
Copy link
Author

rix0rrr commented Apr 21, 2020

Yep, I ran into the bunyan dependency as well.

I did nasty nasty things to get rid of these dependencies, that are likely to blow up on me but haven't so far (by using npm-shrinkwrap and cutting the dependencies I didn't want from the shrinkwrap):

https://github.com/rix0rrr/serve-npm-tarballs/blob/master/shrink-snip.ts#L14
https://github.com/rix0rrr/serve-npm-tarballs/blob/master/serve-npm-tarballs.ts#L23

My immediate problem is solved, so that puts me in the same boat as you: I don't have to change it anymore. I'll give it some thought though.

@juanpicado
Copy link
Member

Thanks for share that, interesting project.

I recently did the first step to remove any sort of dependency (removing node-bunyan) on install (#1760) for the future Verdaccio 5. As I mentioned LevelDB is just an experiment, I don't know wether will be replaced or not, still soon, but the main idea is improve performance, easy scalability and keeping the project small as possible allowing plugins.

Cloud we close this?

@rix0rrr
Copy link
Author

rix0rrr commented Apr 21, 2020

I have a design question though: another solution alluded to in the PR thread is that we could substitute a different storage "plugin".

That is fine, I can also write a different storage plugin that doesn't depend on LevelDB. It would be a less hacky solution.

I'm just not sure how in NPM in general we can control the dependency tree. As long as you depend, out-of-the-box, on @verdaccio/local-storage, then the LevelDB dependency will always be pulled in, and if you don't then you won't work out of the box.

I just don't see how NPM can be used to mix and match plugins. Maybe by setting up a meta-package, @verdaccio/default-setup which depends on all default plugins, and having people build their own setup if they want something else.

But that feels like a bigger change than I could be making in a PR.

@rix0rrr
Copy link
Author

rix0rrr commented Apr 21, 2020

Or I guess, to be somewhat backwards compatible, to turn verdaccio into a meta-package, where all it does is depend on @verdaccio/core, @verdaccio/local-storage, etc.

@juanpicado juanpicado transferred this issue from verdaccio/monorepo Aug 28, 2020
@juanpicado juanpicado added this to the 5.x.x milestone Nov 23, 2020
@juanpicado juanpicado self-assigned this Nov 23, 2020
@juanpicado
Copy link
Member

I'll replace level by https://github.com/typicode/lowdb , buyan #1334 is not part anymore of 5.x branch so the issue here should be solved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment