Slow zef performance and usability comments

[azawawi]

On my "slow" $work network, I am facing a problem while using zef. The "Initializing" part on every run is simply redundant. Why cant `zefcache module results likepanda`` for at least a day and provide an update command?

So far, zef is the superior candidate architecture-wise. However, it needs to be as fast as possible since zef is going to be used a lot by programmers around the world once Perl 6 is successfully launched on Christmas.

Also the -v option in the README.pod usage examples complicates the commands one has to type.

[ugexe]

Agreed. First, i'll point out that this can already be solved by providing your own projects.json file via the --projects-file argument, although I admit this is not an optimal solution for this problem. On the other hand tools like cpanm do not seem to require you to update your project list manually which is advantageous to the user (but then again I have no problem with tools like apt-get needing to be updated manually either). So I think the best solution may be to finally add a config file for keeping state, and allowing zef to be configured to a number-of-days between auto updating its cache (or 0 to do it every time) as well as a --update option to manually override.

Speed

The Initialization message at the start is shown so the user does not think the program is frozen when you start it, but is not tied to network activity.. Why does it start so slow? It's not because of the network connection (well usually its not, in your case it I understand it is), but rather because of rakudo's current CompUnit implementation (specifically CompUnitRepo::Local::Installation). panda uses the older style CompUnitRepo::Local::File which rakudo can load modules faster with, but this is a bug of rakudo itself (specifically the internal JSON parser). In short: currently more installed modules = slower rakudo startup. (as a side note: zef doesn't have to be installed to work so you could create an alias to wherever you clone zef and use that to invoke zef so as not to install extra modules, but thats more of something i'd expect a tool like rakudobrew to handle).

Do not let this speed discrepency worry you though! There is work being done on the new CompUnit setup for rakudo by lizmat and nine (the curli branch of rakudo). This new implementation (replacing both ::Installation and ::File) will speed things up because it will only need to read a json file when a module is used instead of the current parse one giant json file on startup (which is why currently more installed modules = slower). Those changes will also allow the Zef code base to shrink significantly, further speeding things up, as well as allowing more focus on parallelization and its general role as a recommendation manager + content retrieval.

README

Again I agree. -v was important initially when most people were interested in the test output, but not so much now that things are stabilizing in the ecosystem and core.

[azawawi]

I guess we can use a web service like cpanm for locating metadata for module X. That will fix the problem right away since the lookup will be faster with smaller JSON response parsing time. If it timeouts or fails, we should revert to the current method.

As mentioned before, my corporate network is way slower with DNS request (long story that started with a power failure problem :) ) so you can think of it as the worst case to happen. But still it happened.

I recently added rakudobrew build-zef to improve zef's installation procedure.

[ugexe]

Ah, but you misunderstand. The slow part is not parsing metadata per install, its rakudo itself loading the meta data json every time you start it (and also use at least 1 module). The filtering of the retrieved json is generally quick. This link shows the culprit: https://github.com/rakudo/rakudo/blob/nom/src/core/CompUnitRepo/Local/Installation.pm#L9 Note despite the name that it also loads modules, not just installs them (to be specific, it will load modules installed by it, while CompUnitRepo::Local::File can install and load modules installed by it) But again I don't expect this to last much longer. The new CompUnit work will have to be done before x-mas, and it won't take me long to adapt changes to zef.

I had an idea the other day related to this topic though. I could create a ecosystem projects list clone but host it on github. Because git is required anyway it would then mean the http client could be removed (less modules to install = more speed), which is certainly slower than the git command. Then to add support for module sources other than the ecosystem when they become available (like metacpan) it could bootstrap the appropriate plugin to make it work (so its only installed if its needed/wanted). The idea is to then allow it to make recommendations based on multiple sources of modules. This would match your web service idea.

We could also find a way to provide access to the project list via IP address instead of requiring a host name... maybe git will solve this as well. As you can tell there are quite a few improvements I would like to make, i'm just waiting for the the CompUnit changes.

Thanks for adding zef to rakudobrew!

[ugexe]

I've changed the default behavior so the package list is only updated when zef update or zef --update install X Y Z (and automatically once during the install of zef if the zef home directory does not exist). I will work on adding a configuration file soon so the default rules can be customized.

fb92b85

[azawawi]

👍

[ugexe]

NOTE: I need to re-add the mentioned functionality (to only update when needed or requested) to the new version of zef

[tony-o]

@azawawi is this still an issue or is it something we can close?

[azawawi]

Closing. Thanks for the reminder 👍