Pages for cached npm packages

[MKRhere]

I'm currently using verdaccio because I have to be somewhere without internet for a few days. I've cached a lot of npm modules via verdaccio, and everything seems good, but I'd love to see pages generated for them on the localhost frontend like it appears on npmjs.com. This already happens when you publish from verdaccio, but I'd like to (optionally?) enable these pages for packages pulled from npm as well.

Oh, and well done! I love what you've been doing so far.

Thoughts on this?

[juanpicado]

I'm currently using verdaccio because I have to be somewhere without internet for a few days.

I needed to quote this 👍

but I'd love to see pages generated for them on the localhost frontend like it appears on npmjs.com

This is not possible right now, but, why not? I'd love it as well. I think would be a nice and loved PR.

Oh, and well done! I love what you've been doing so far.

🙃😊

[MKRhere]

Aww, I would love to make a PR, but I can't now. I'll keep an eye on it if I ever get time 😄 If someone else can, it'd be a welcome addition! 😄

[TheILPlace]

Hi,
currently i am using verdaccio with several projects, in which there is NO internet connection at all.
i use verdaccio to cache the npm packages in an external network. zip the storage folder, move it into the disconnected network, and merge it with the existing storage folder.

as far as using verdaccio to install the npm packages in the disconnected env. - perfect.
the main problem - you cannot search for a package using the webpage !

this looks like a trivial query - check what is in the cache.
is this not a simple task to accomplish ?

[juanpicado]

@TheILPlace two issues here:

1. the web page lookup for updates on uplinks for private pages, it was a design issue heritage since the fork. We will fix that, just lack of time and resources. This slow down the search process.

Refers to: #1005

2. About cached nonprivate packages, that might work well with 100, 200 but with thousands is not that trivial.

[nanchen]

Starting verdaccio will generate a file called ".sinopia-db.json" under storage folder, adding package names to "list" array will do the trick to show the cached packages. Though I don't know if it's recommended.

my file content:
{"list":["blob"],"secret":"1adfa7e39bd3fc88c4106fbe6808620de6361237e920edce77c45a71e928639c"}

So package 'blob' is shown in my web page:

[juanpicado]

@nanchen :) that's hacky, you can do it and works, but, it's not a good practice.

[juanpicado]

Once #1126 is merged, we can experiment easily to solve this.

[fengyuchenglun]

this feature can use?

[HawtinZeng]

this feature can use?

[juanpicado]

No

[juanpicado]

Currently, as a first step, I'm adding required logic to the local-storage for retrieving all packages from storage + private ones. Once that's done let's see how is the performance with huge storages. In any case, this will be optional and enabled by config demand. I haven't chosen any possible config name but I'd love to listen to proposals.

[fengyuchenglun]

how config ?

[juanpicado]

@fengyuchenglun currently not possible, we haven't get there yet. The unique part is ready is the UI which is able to handle thousands of packages without crash, but the backend not ready yet. There has been progress with the #2340 that search in the whole private database and could be easily hooked in the UI, but the /packages endpoint is only able to fetch private packages, not all cached ones.

[MKRhere]

When does the retrieving happen? On starting the server? Once the server has started it can persist when fetching new packages and if a package was deleted from fs, and the page was navigated to, you can say package was deleted and go to previous page and remove from storage. How is it handled with private packages though? 🤔

[juanpicado]

When does the retrieving happen? On starting the server?

No, this will hit performance on booting and the API does not care about this feature, it is just for UI and those has enable the UI.

Once the server has started it can persist when fetching new packages and if a package was deleted from fs, and the page was navigated to, you can say the package was deleted and go to the previous page and remove from storage. How is it handled with private packages though? 🤔

The original design fetch packages under demand. Thus, each time user visits the page the list of private packages and his metadata is retrieved and a web api endpoint provide list to UI after the check permissions take place.

Having enabled this for the whole storage will increase the fetching time since the default plugin is based on file sytem. In any case, this has to be done for each plugin since verdaccio delegate this job through the Storage API.

The unique cache that exist is for the search endpoint using the lunr-mutable-indexes library for private packages. I'm not sure yet how would be the impact if I extend this to the whole storage.

[chauthai]

Do you still need help for this issue? I've just found your project and I want to help out.

[juanpicado]

@chauthai yes, I was working on this, the current status is the following

• https://github.com/verdaccio/local-storage must be modified in order to be able to fetch all packages, this is a design change, not that trivial that might affect other plugins.
• Once that is solved, we need to modify verdaccio Storage and LocalStorage handlers.
• Also, we need a new option to enable this by demand, this has to be optional

If you want more insights about this. Feel free join our chat (discord).

[artemdudkin]

pls look at
verdaccio/local-storage#164
#1281
verdaccio/ui#35

[carlos-penton]

Perhaps you might like verdaccio-offline-storage: it lists all the locally available packages in the web ui AND rewrites on-the-fly the packages definitions so only locally available versions are recognized. Also, it's compatible with the default local-storage plugin, so the registry won't get broken. Hope it helps.

[juanpicado]

Perhaps you might like verdaccio-offline-storage: it lists all the locally available packages in the web ui AND rewrites on-the-fly the packages definitions so only locally available versions are recognized. Also, it's compatible with the default local-storage plugin, so the registry won't get broken. Hope it helps.

Excellent move 👏

[carlos-penton]

Excellent move 👏

Thanks!

[juanpicado]

Update about this, on the current master branch (not 5.x)

• The new search service is able to perform better search either private, local and public via uplinks
• If the update the web search api with the new refactor: search v1 endpoint and local-database #2340 endpoint, the search suggestion could search in any package, thus navigate in cache packages
• What's not ready (the hardest part) is display cache packages on the front-end page, that might require more thinking.

PR are welcome for the point 2.

[Stevearzh]

Hello, anything update here?

I met the same problem in a closing network environment, since I'm using 5.x version via docker, I see both two plugins local-storage and verdaccio-offline-storage you mentioned before are not maintained for a long time...

[juanpicado]

You can try npm i -g verdaccio@next-7 is pretty much what I described here #706 (comment)