Cached modules of lower version given preference to newer modules in ecosystem

[zoffixznet]

Context

Install of IRC::Client fails with zef, but works with panda. I dug through the dep chain and found zef is trying to install META6 version 0.0.9, which likely is missing some bug fix, while the latest is 0.0.11.

I'm on bleed rakudo and zef; trying to just run zef install META6 fetches too-old version. This makes me think zef probably mistakenly thinks v0.0.9 is after v0.0.11

zoffix@leliana:~$ zef update
-------------------------------------------------------
Content Storage                     |Distribution Count
-------------------------------------------------------
Zef::ContentStorage::Ecosystems<p6c>|766               
Zef::ContentStorage::LocalCache     |57                
-------------------------------------------------------


zoffix@leliana:~$ zef install META6
===> Searching for: META6
===> Testing: META6:ver('0.0.9'):auth('github:jonathanstowe')
t/010-use.t .......... ok 
t/015-constructor.t .. ok 
t/020-basic.t ........1/?Saw 1 occurrence of deprecated code.
================================================================================
Sub from-json (from GLOBAL) seen at:
  t/020-basic.t, line 25
Please use JSON::Fast, JSON::Tiny or JSON::Pretty from https://modules.perl6.org/ instead.
--------------------------------------------------------------------------------
Please contact the author to have these occurrences of deprecated code
adapted, so that this message will disappear!
t/020-basic.t ........ ok 
t/030-versions.t ..... ok 
===SORRY!=== Error while compiling /home/zoffix/.zef/store/META6.git/31f5de4dd441985d7a684609a06269836dd93ed2/t/040-projects.t
An exception occurred while evaluating a constant
at /home/zoffix/.zef/store/META6.git/31f5de4dd441985d7a684609a06269836dd93ed2/t/040-projects.t:8
Exception details:
  Could not find symbol '&Class'
    in block  at t/040-projects.t line 8

t/040-projects.t ..... No subtests run
All tests successful.

Test Summary Report
-------------------
t/040-projects.t  (Wstat: 0 Tests: 0 Failed: 0)
  Parse errors: No plan found in TAP output
Files=5, Tests=38,  1 wallclock secs
Result: FAILED
===> Testing [FAIL]: META6:ver('0.0.9'):auth('github:jonathanstowe')
Aborting due to test failure: META6:ver('0.0.9'):auth('github:jonathanstowe') (use --force to override)
  in code  at /home/zoffix/.rakudobrew/moar-nom/install/share/perl6/site/sources/1DC0BAA246D0774E7EB4F5119C6168E0D8266EFA (Zef::Client) line 306
  in method test at /home/zoffix/.rakudobrew/moar-nom/install/share/perl6/site/sources/1DC0BAA246D0774E7EB4F5119C6168E0D8266EFA (Zef::Client) line 285
  in code  at /home/zoffix/.rakudobrew/moar-nom/install/share/perl6/site/sources/1DC0BAA246D0774E7EB4F5119C6168E0D8266EFA (Zef::Client) line 456
  in sub  at /home/zoffix/.rakudobrew/moar-nom/install/share/perl6/site/sources/1DC0BAA246D0774E7EB4F5119C6168E0D8266EFA (Zef::Client) line 453
  in method install at /home/zoffix/.rakudobrew/moar-nom/install/share/perl6/site/sources/1DC0BAA246D0774E7EB4F5119C6168E0D8266EFA (Zef::Client) line 559
  in sub MAIN at /home/zoffix/.rakudobrew/moar-nom/install/share/perl6/site/sources/A9948E7371E0EB9AFDF1EEEB07B52A1B75537C31 (Zef::CLI) line 122
  in block <unit> at /home/zoffix/.rakudobrew/moar-nom/install/share/perl6/site/resources/3DD33EF601FD300095284AE7C24B770BAADAF32E line 1

zoffix@leliana:~$ 

Expected Behavior

Should install META6 v0.0.11

$ perl6 -e 'say v0.0.11 cmp v0.0.9'
More
$ perl6 -e 'say v0.0.11 after v0.0.9;'
True

Actual Behavior

Tries to install META6 v0.0.9

Steps to Reproduce

with latest and greatest zef, run zef install META6 and observe that it tries to install wrong version.

Your Environment

$ perl6 -v
This is Rakudo version 2016.12-266-ged5c866 built on MoarVM version 2016.12-71-g331a6b4
implementing Perl 6.c.
$ perl6 -e 'say $*VM.config<os>'
linux

[ugexe]

tl;dr: the following commands would Do The Right Thing:

zef --/cached install META6
zef install "META6:ver<0.0.11>"

---

The caching mechanism needs to be fixed in relation to the recommendation manager parts

You ran zef update so you had an updated ecosystem list with that version in it (which could be confirmed via a zef search META6), but the current behavior is to take the first matching dist it finds. Why? Two reasons:

1. By having the cache checked first (btw this can be changed by reordering inside of resources/config.json) its possible to do installs without hitting the network (since the ecosystem defaults to auto-update itself when searched)
2. Since the cache is (assumed to be) checked first we can save on having to parse the updated ecosystem list that was downloaded with zef update earlier

Item 1 can be reworked to do what most people expect

Item 2 was a concern a yearish ago, because parsing the entire ecosystem json was fairly slow (and while rakudo is getting faster it may or may not be enough to make up for the increase in size). But I think people would rather deal with a little slowness for now if they get what they expect more often. Realistically a better format is needed that can be parsed faster while also (but simple enough to not require any external modules)

---

So zef needs to be changed to always check all module sources (ecosystem, cache, cpan, etc) that are enabled at the time (although this doesn't mean it has to download the package list each time - only that it needs to search the cached package list)

[rafaelschipiura]

You suggested compiling a faster version of the ecosystem instead of caching it. Combined with searching the newest version on this compiled list, would indeed solve the problem.