verdaccio-cli external tool to handle an intensive task

[juanpicado]

My reason:

My suggestion comes inspired by #142 and #138

The CLI might reside in:

• Mixed with the current verdaccio cli
• Create an external and new tool verdaccio-cli

Caveats

• How to this will affect on running instances without corrupt tarballs?
• How this will work with future pluggable storages
• User management is hot topic, but I'm not sure how this will work properly

Usage

• Cache clean
• Integrity check
• Backup
• Restore Backup
• User management?

Clean cache

Clean cache might be a high intensive task base on the local storage size.

Normal clean cache

verdaccio-cli cache clean 

Clean specific package

verdaccio-cli cache clean @scope/package

Clean cache for specific package + version

verdaccio-cli cache clean @scope/package@1.0.0

Clean cache for specific version

verdaccio-cli cache clean @scope/package@1.0.0

Clean cache specifying a config file

verdaccio-cli cache clean -c /path/config.yaml @scope/package

Backup and Restore

Backup and restore a backup, useful for offline environments

A complete backup

verdaccio-cli backup

Only cached

verdaccio-cli backup --ignore-private

Only private

verdaccio-cli backup --ignore-local

Restore backup

verdaccio-cli restore --path /path/backup.tar.gz

Check Integrity

Check the integrity of tarballs

Package check

verdaccio-cli check @scope/package@1.0.0

User management ⚠️

Add/Remove users (this part is just a suggestion and will require a discussion) and perhaps only useful with the default auth plugin.

Add user

verdaccio-cli user-add 

Remove

verdaccio-cli user-rm

I think this will require a lot of thoughts, please comment and let me know what do you think.

[arsdehnel]

Are these function (backup, clean, restore) already existing and these are just CLI commands to get at them? I'm not seeing where these things exist in the current codebase but maybe I'm just missing something. Have hit some gross invalid tar header errors and I'd be happy to solve them with a proper tool rather than some bash nonsense.

[juanpicado]

@arsdehnel this is just a Feature Request, none of this does no exist in the code base yet.

[juanpicado]

I don't think I will work on this. If someone thinks is a good idea and is willing to contribute, please reopen.

[juanpicado]

I'll reopen in case someone is interested on work on it.

[p3x-robot]

@juanpicado FYI: I have created a verdaccio-cli, for now, it is in my name p3x-verdaccio-cli. I would not reserve the name, but if you like it, we can move to verdaccio main repo.

https://github.com/patrikx3/verdaccio-cli
https://www.corifeus.com/verdaccio-cli
https://www.npmjs.com/package/p3x-verdaccio-cli

[juanpicado]

FYI, a new interesting tool for storage migration #1740

[immanuelfodor]

Cache cleaning or metadata refresh would be great to have inside the Docker container. Currently, it can only be solved by setting up a small (e.g., 2m) timeout to get new metadata from the upstream repo, then restarting the container, then setting it back (e.g., 24h), then another restart.

[juanpicado]

Cache cleaning or metadata refresh would be great to have inside the Docker container. Currently, it can only be solved by setting up a small (e.g., 2m) timeout to get new metadata from the upstream repo, then restarting the container, then setting it back (e.g., 24h), then another restart.

Could you elaborate a bit more? ... how your scenario looks like?

[immanuelfodor]

I had a package eslint-scope cached in verdaccio up to v5.0.0, but v5.1.0 was published 2 hours ago: https://www.npmjs.com/package/eslint-scope (now it's 5 hours).

ls -la /verdaccio/storage/data/eslint-scope/
total 180
drwxr-xr-x    2 root root   146 Jun  4 15:35 .
drwxr-xr-x 1688 root root 49152 May 25 12:23 ..
-rw-r--r--    1 root root 15576 Mar  5 10:50 eslint-scope-3.7.1.tgz
-rw-r--r--    1 root root 16124 Mar  5 10:49 eslint-scope-4.0.3.tgz
-rw-r--r--    1 root root 16289 May 15 12:26 eslint-scope-5.0.0.tgz
-rw-r--r--    1 root root 16508 Jun  4 15:35 eslint-scope-5.1.0.tgz  # this is the wanted new package, originally not cached and so not found locally

Verdaccio had the following config:

uplinks:
  npmjs:
    url: https://registry.npmjs.org/
    maxage: 24h

As far as I understand, this means given a new package request, upstream is not checked even for metadata, the local versions are served.

In my CI config, npm install runs through the local Verdaccio package proxy, and as the build requested v5.1.0, the pipeline has failed. No such version existed, and the metadata refresh was not due until tomorrow, so it did not pick up the recent new version.

I restarted the docker container in hope of a metadata refresh/clean, but it did not help. Then I looked for how could I trigger a metadata refresh or invalidate it. Without any meaningful results in the container (verdaccio flags, helper script) or online search results, I modified the above config to have 2m instead of 24h, then restarted the container. Through a new CI npm install, Verdaccio has picked up the new package version (v5.1.0), then I put back the 24h, and restarted the container again.

This is why it would be great to force a metadata refresh for a given package overriding the global config.

[juanpicado]

I think #343 (comment) does the job, please try that and 🙏 help him if any feature is required there.

[immanuelfodor]

Would you please integrate this CLI into the Verdaccio Dockerfile? It would be great to have it there and configured to use the local instance by default, e.g., with using a bash alias. As we are running Verdaccio with Rancher, it would be so easy to step into the container, and do the cache clean when needed.

[juanpicado]

Would you please integrate this CLI into the Verdaccio Dockerfile? It would be great to have it there and configured to use the local instance by default, e.g., with using a bash alias. As we are running Verdaccio with Rancher, it would be so easy to step into the container, and do the cache clean when needed.

@immanuelfodor I cannot include code is not part of the Verdaccio GH organization within Dockerfile (assuming would be possible which I don't know), I'm quite strict on that regard.