Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

verdaccio-cli external tool to handle an intensive task #343

Closed
juanpicado opened this issue Oct 5, 2017 · 12 comments
Closed

verdaccio-cli external tool to handle an intensive task #343

juanpicado opened this issue Oct 5, 2017 · 12 comments

Comments

@juanpicado
Copy link
Member

@juanpicado juanpicado commented Oct 5, 2017

My reason:

My suggestion comes inspired by #142 and #138

The CLI might reside in:

  • Mixed with the current verdaccio cli
  • Create an external and new tool verdaccio-cli

Caveats

  • How to this will affect on running instances without corrupt tarballs?
  • How this will work with future pluggable storages
  • User management is hot topic, but I'm not sure how this will work properly

Usage

  • Cache clean
  • Integrity check
  • Backup
  • Restore Backup
  • User management?

Clean cache

Clean cache might be a high intensive task base on the local storage size.

Normal clean cache

verdaccio-cli cache clean 

Clean specific package

verdaccio-cli cache clean @scope/package

Clean cache for specific package + version

verdaccio-cli cache clean @scope/package@1.0.0

Clean cache for specific version

verdaccio-cli cache clean @scope/package@1.0.0

Clean cache specifying a config file

verdaccio-cli cache clean -c /path/config.yaml @scope/package

Backup and Restore

Backup and restore a backup, useful for offline environments

A complete backup

verdaccio-cli backup

Only cached

verdaccio-cli backup --ignore-private

Only private

verdaccio-cli backup --ignore-local

Restore backup

verdaccio-cli restore --path /path/backup.tar.gz

Check Integrity

Check the integrity of tarballs

Package check

verdaccio-cli check @scope/package@1.0.0

User management ⚠️

Add/Remove users (this part is just a suggestion and will require a discussion) and perhaps only useful with the default auth plugin.

Add user

verdaccio-cli user-add 

Remove

verdaccio-cli user-rm

I think this will require a lot of thoughts, please comment and let me know what do you think.

@arsdehnel
Copy link

@arsdehnel arsdehnel commented Jan 26, 2018

Are these function (backup, clean, restore) already existing and these are just CLI commands to get at them? I'm not seeing where these things exist in the current codebase but maybe I'm just missing something. Have hit some gross invalid tar header errors and I'd be happy to solve them with a proper tool rather than some bash nonsense.

@juanpicado
Copy link
Member Author

@juanpicado juanpicado commented Jan 26, 2018

@arsdehnel this is just a Feature Request, none of this does no exist in the code base yet.

@juanpicado
Copy link
Member Author

@juanpicado juanpicado commented Mar 15, 2018

I don't think I will work on this. If someone thinks is a good idea and is willing to contribute, please reopen.

@juanpicado juanpicado closed this Mar 15, 2018
@juanpicado juanpicado modified the milestones: 3.0.0, Future Apr 10, 2018
@juanpicado juanpicado added this to To do in Roadmap via automation Apr 16, 2018
@juanpicado
Copy link
Member Author

@juanpicado juanpicado commented Apr 16, 2018

I'll reopen in case someone is interested on work on it.

@juanpicado juanpicado reopened this Apr 16, 2018
Roadmap automation moved this from To do to In progress Apr 16, 2018
@juanpicado juanpicado moved this from In progress to To do in Roadmap Apr 24, 2018
@p3x-robot
Copy link

@p3x-robot p3x-robot commented Jan 23, 2019

@juanpicado FYI: I have created a verdaccio-cli, for now, it is in my name p3x-verdaccio-cli. I would not reserve the name, but if you like it, we can move to verdaccio main repo.

https://github.com/patrikx3/verdaccio-cli
https://www.corifeus.com/verdaccio-cli
https://www.npmjs.com/package/p3x-verdaccio-cli

@juanpicado
Copy link
Member Author

@juanpicado juanpicado commented Mar 9, 2020

FYI, a new interesting tool for storage migration #1740

@immanuelfodor
Copy link

@immanuelfodor immanuelfodor commented Jun 4, 2020

Cache cleaning or metadata refresh would be great to have inside the Docker container. Currently, it can only be solved by setting up a small (e.g., 2m) timeout to get new metadata from the upstream repo, then restarting the container, then setting it back (e.g., 24h), then another restart.

@juanpicado
Copy link
Member Author

@juanpicado juanpicado commented Jun 4, 2020

Cache cleaning or metadata refresh would be great to have inside the Docker container. Currently, it can only be solved by setting up a small (e.g., 2m) timeout to get new metadata from the upstream repo, then restarting the container, then setting it back (e.g., 24h), then another restart.

Could you elaborate a bit more? ... how your scenario looks like?

@immanuelfodor
Copy link

@immanuelfodor immanuelfodor commented Jun 4, 2020

I had a package eslint-scope cached in verdaccio up to v5.0.0, but v5.1.0 was published 2 hours ago: https://www.npmjs.com/package/eslint-scope (now it's 5 hours).

ls -la /verdaccio/storage/data/eslint-scope/
total 180
drwxr-xr-x    2 root root   146 Jun  4 15:35 .
drwxr-xr-x 1688 root root 49152 May 25 12:23 ..
-rw-r--r--    1 root root 15576 Mar  5 10:50 eslint-scope-3.7.1.tgz
-rw-r--r--    1 root root 16124 Mar  5 10:49 eslint-scope-4.0.3.tgz
-rw-r--r--    1 root root 16289 May 15 12:26 eslint-scope-5.0.0.tgz
-rw-r--r--    1 root root 16508 Jun  4 15:35 eslint-scope-5.1.0.tgz  # this is the wanted new package, originally not cached and so not found locally

Verdaccio had the following config:

uplinks:
  npmjs:
    url: https://registry.npmjs.org/
    maxage: 24h

As far as I understand, this means given a new package request, upstream is not checked even for metadata, the local versions are served.

In my CI config, npm install runs through the local Verdaccio package proxy, and as the build requested v5.1.0, the pipeline has failed. No such version existed, and the metadata refresh was not due until tomorrow, so it did not pick up the recent new version.

I restarted the docker container in hope of a metadata refresh/clean, but it did not help. Then I looked for how could I trigger a metadata refresh or invalidate it. Without any meaningful results in the container (verdaccio flags, helper script) or online search results, I modified the above config to have 2m instead of 24h, then restarted the container. Through a new CI npm install, Verdaccio has picked up the new package version (v5.1.0), then I put back the 24h, and restarted the container again.

This is why it would be great to force a metadata refresh for a given package overriding the global config.

@juanpicado
Copy link
Member Author

@juanpicado juanpicado commented Nov 4, 2020

I think #343 (comment) does the job, please try that and 🙏 help him if any feature is required there.

@juanpicado juanpicado closed this Nov 4, 2020
@immanuelfodor
Copy link

@immanuelfodor immanuelfodor commented Nov 4, 2020

Would you please integrate this CLI into the Verdaccio Dockerfile? It would be great to have it there and configured to use the local instance by default, e.g., with using a bash alias. As we are running Verdaccio with Rancher, it would be so easy to step into the container, and do the cache clean when needed.

@juanpicado
Copy link
Member Author

@juanpicado juanpicado commented Nov 4, 2020

Would you please integrate this CLI into the Verdaccio Dockerfile? It would be great to have it there and configured to use the local instance by default, e.g., with using a bash alias. As we are running Verdaccio with Rancher, it would be so easy to step into the container, and do the cache clean when needed.

@immanuelfodor I cannot include code is not part of the Verdaccio GH organization within Dockerfile (assuming would be possible which I don't know), I'm quite strict on that regard.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Roadmap
  
To do
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants