New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Question: verdaccio v4 scalability. #1459
Comments
@wzrdtales do you have comment on this? Since you posted the original issue #103. |
hi @favoyang The #103 basically triggered the idea of do not depend of the file system, which was the base of Sinopia. At Verdaccio 3 we introduced the idea of Storage API, which basically allows you to use your own storage. For example, using AWS buckets https://github.com/verdaccio/docker-examples/tree/master/amazon-s3-docker-example/v4 The default authentication is also file system based by default, which can also being replaced by plugins with their own storage persistence.
The
For this and experiences scaling Verdaccio you need ask the community, unfortunately I cannot help in such field which time on time provide feedback how they use it, I've heard crazy things but I've never had the time to run crazy experiments aside of the docker-example repository mostly with mentoring purposes. |
I put my 50 cents here https://twitter.com/verdaccio_npm/status/1170374861618892800 that's all I can do from my side. |
Thanks for sharing the info. My major concern was from https://verdaccio.org/docs/en/amazon documentation, which described a stack with one balancer + multiple verdaccio worker instances + Elastic File System. Because the .verdaccio-db.json file is obviously a shared stateful point, it won't work unless some kind concurrent tech are using, like a trivial master / slave mechanic used by most relation databases, or simply delegate it to a faster enough backend like redis, to make verdaccio worker stateless. However the s3-docker link you shared, shows it is using the Remitly/verdaccio-s3-storage plugin. If no .verdaccio-db.json happens in this backend, we should be fine. Most time verdaccio operations fall into a simple write / massive read scenario, which relative easy to scale. I'll close this ticket, but feel free to add more comments. |
The docker example uses https://github.com/verdaccio/monorepo/tree/master/plugins/aws-s3-storage which is under control :-) |
🤖This thread has been automatically locked 🔒 since there has not been any recent activity after it was closed. |
Just a note on this - this file is used in the S3 storage plugin. |
Hi @apexskier A quick update. Any store relies on the The aws-s3 backend is not scaleable, the discussion moved to verdaccio/monorepo#317 and PR candidate is verdaccio/monorepo#275. But the review process is very slow. The verdaccio-minio is relatively new to me. It uses a single JSON (db) to store the package info list, but the good news is it won't cache the list into memory, every action will reload that file from the disk. It has a solved issue barolab/verdaccio-minio#13 about the cluster issue and resulted in removing the memory cache. However, probably not good enough, there still could be a racing issue. |
I'm planning to operate an open registry service which may have more traffics than average private registry usage at certain time. My major candidates are verdaccio, cnpm and codebox-npm. And verdaccio looks promising. I'm curious at the time of 2019, how scale is verdaccio v4.
Except an old thread #103, I didn't find much up-to-date information about scalability. By doing some researches, I narrow down to two similar solutions
It seems very straightforward that the elastic file system/s3 + multiple instances shall work, until I find there's a .verdaccio-db.json file, seems act as a simple shared db contains all package lists. So how it works in a cluster environment even with s3? Multiple instances write operations (adding new package) try to modify the same file, and it may out of sync, not to mention how it broadcast the changes to other instances.
I haven't dig into verdaccio code too much, is there something obvious I missed? Any real world scalability tips would be also helpful.
The text was updated successfully, but these errors were encountered: