The top sites prioritization is faulty

[IuliaCristescu4]

https://bugzilla.mozilla.org/show_bug.cgi?id=1410180

[Note]:

• I started from the assumption that, without any default top sites, the Top Sites section should always display the most visited websites, in descending order

[Affected versions]:

• 58.0a1 (2017-10-19)
• 57.0b9 (20171016185129)

[Affected platforms]:

• Windows 10x64
• Ubuntu 16.04 x64

[Steps to reproduce]:

1. Launch Firefox with a clean profile
2. Open the following websites (in separate tabs):
   • https://9gag.com/ (8 times)
   • https://imgur.com/ (9 times)
   • http://www.psimadethis.com/ (12 times)
   • https://bugzilla.mozilla.org/ (4 times)
3. Restart the browser and access the AS New Tab page
4. Access http://www.imdb.com/ 11 times, restart the browser and reopen the AS New Tab page
5. Access https://github.com/ 13 times, restart the browser and reopen the AS New Tab page

[Expected result]:

• [step3] Top Sites section displays the above sites in the following order:
  http://www.psimadethis.com/, https://imgur.com/, https://9gag.com/, https://bugzilla.mozilla.org/
• [step4] Top Sites section displays the above sites in the following order:
  http://www.psimadethis.com/, http://www.imdb.com/, https://imgur.com/, https://9gag.com/, https://bugzilla.mozilla.org/
• [step5] Top Sites section displays the above sites in the following order:
  https://github.com/, http://www.psimadethis.com/, http://www.imdb.com/, https://imgur.com/, https://9gag.com/, https://bugzilla.mozilla.org/

[Actual result]:

• [step3] Top Sites section displays the above sites in the following order:
  https://imgur.com/, http://www.psimadethis.com/, https://9gag.com/, https://bugzilla.mozilla.org/
• [step4] Top Sites section displays the above sites in the following order:
  https://imgur.com/, http://www.imdb.com/, http://www.psimadethis.com/, https://9gag.com/, https://bugzilla.mozilla.org/
• [step5] Top Sites section displays the above sites in the following order:
  https://github.com/, https://imgur.com/, http://www.imdb.com/, http://www.psimadethis.com/, https://9gag.com/, https://bugzilla.mozilla.org/

[Additional notes]:

• The same steps provided the expected results using Mac OS 10.11.6

[Mardak]

Here's the various screenshots I took during the process of trying to reproduce on linux64 but got the expected results:

After visiting first set, before restart:

After visiting first set, after restart:

After visiting imdb, before restart:

After visiting imdb, after restart:

After visiting github, before restart:

After visiting github, after restart:

All the sites as in expected result:

I did verify that each of the top sites in the expected positions had the expected frecencies:
NewTabUtils.activityStreamProvider.getTopFrecentSites().then(v => console.log(v.map(s => s.frecency)))
Array [ 26000, 24000, 22000, 18000, 16000, 8000 ]

It's possible for the frecency scores to decay across restarts, so that a 12 visited site appears after a 11 visited site if those 11 visits were more recent than the 12 visits.

[IuliaCristescu4]

I followed again the above steps using Windows 10 x64 and 57.0b10 (20171019140425). From 2 attempts (with 2 different clean profiles), the first one was successful (the expected results were triggered - see the screencast https://goo.gl/oHEHKT) and the second one was the faulty one (the exact actual results from above were triggered - see the screencast https://goo.gl/fGMMyu). Is it possible that some active shield experiment to influence the behavior?

[Mardak]

I seem to reproduce something similar if opening many tabs quickly with the url where the issue is that the "typed" / pasted in page doesn't associate as a typed page probably because so many tabs are being opened.

» NewTabUtils.activityStreamProvider.getTopFrecentSites().then(v => console.log(v.map(s => [s.frecency, s.url]).join("\n")))
19440,http://www.psimadethis.com/
14100,https://9gag.com/
10400,https://imgur.com/
8300,https://bugzilla.mozilla.org/
sqlite> SELECT url, frecency, v.* FROM moz_historyvisits v JOIN moz_places h ON h.id = v.place_id;
https://www.mozilla.org/privacy/firefox/|25|1|0|11|1508786614509189|1|0
https://www.mozilla.org/en-US/privacy/firefox/|100|2|1|12|1508786614571997|5|0
https://9gag.com/|14100|3|0|13|1508786622171639|2|0
https://9gag.com/|14100|4|0|13|1508786622203836|1|0
https://9gag.com/|14100|5|0|13|1508786623399590|2|0
https://9gag.com/|14100|6|0|13|1508786623745059|2|0
https://9gag.com/|14100|7|0|13|1508786624709272|2|0
https://9gag.com/|14100|8|0|13|1508786625790396|2|0
https://9gag.com/|14100|9|0|13|1508786626371267|2|0
https://9gag.com/|14100|10|0|13|1508786627296569|2|0
https://imgur.com/|10400|11|0|14|1508786632576312|2|0
https://imgur.com/|10400|12|0|14|1508786633244981|2|0
https://imgur.com/|10400|13|0|14|1508786634507353|2|0
https://imgur.com/|10400|14|0|14|1508786636050321|2|0
https://imgur.com/|10400|15|0|14|1508786636458137|1|0
https://imgur.com/|10400|16|0|14|1508786640134732|2|0
https://imgur.com/|10400|17|0|14|1508786640190539|1|0
https://imgur.com/|10400|18|0|14|1508786640412034|1|0
https://imgur.com/|10400|19|0|14|1508786641036418|1|0
http://www.psimadethis.com/|19440|20|0|15|1508786647004844|2|0
http://www.psimadethis.com/|19440|21|0|15|1508786647816430|2|0
http://www.psimadethis.com/|19440|22|0|15|1508786648481147|1|0
http://www.psimadethis.com/|19440|23|0|15|1508786649493700|2|0
http://www.psimadethis.com/|19440|24|0|15|1508786650361411|2|0
http://www.psimadethis.com/|19440|25|0|15|1508786651159309|2|0
http://www.psimadethis.com/|19440|26|0|15|1508786652085877|2|0
http://www.psimadethis.com/|19440|27|0|15|1508786652886799|2|0
http://www.psimadethis.com/|19440|28|0|15|1508786653831779|2|0
http://www.psimadethis.com/|19440|29|0|15|1508786654645928|2|0
http://www.psimadethis.com/|19440|30|0|15|1508786655501652|2|0
http://www.psimadethis.com/|19440|31|0|15|1508786656354243|1|0
https://bugzilla.mozilla.org/|8300|32|0|2|1508786662587968|2|0
https://bugzilla.mozilla.org/|8300|33|0|2|1508786662955018|2|0
https://bugzilla.mozilla.org/|8300|34|0|2|1508786663972252|2|0
https://bugzilla.mozilla.org/|8300|35|0|2|1508786664966663|2|0

However, I don't see the issue when typing in the url once and loading it again by pressing enter.

» NewTabUtils.activityStreamProvider.getTopFrecentSites().then(v => console.log(v.map(s => [s.frecency, s.url]).join("\n")))
24000,http://www.psimadethis.com/
18000,https://imgur.com/
16000,https://9gag.com/
8300,https://bugzilla.mozilla.org/
sqlite> SELECT url, frecency, v.* FROM moz_historyvisits v JOIN moz_places h ON h.id = v.place_id;
https://www.mozilla.org/privacy/firefox/|25|1|0|11|1508787251464412|1|0
https://www.mozilla.org/en-US/privacy/firefox/|100|2|1|12|1508787251512556|5|0
https://9gag.com/|16000|3|0|13|1508787432000880|2|0
https://9gag.com/|16000|4|0|13|1508787432286841|2|0
https://9gag.com/|16000|5|0|13|1508787433064777|2|0
https://9gag.com/|16000|6|0|13|1508787435048495|2|0
https://9gag.com/|16000|7|0|13|1508787436843112|2|0
https://9gag.com/|16000|8|0|13|1508787437376302|2|0
https://9gag.com/|16000|9|0|13|1508787438840822|2|0
https://9gag.com/|16000|10|0|13|1508787453554179|2|0
https://imgur.com/|18000|11|0|14|1508787462172479|2|0
https://imgur.com/|18000|12|0|14|1508787462865469|2|0
https://imgur.com/|18000|13|0|14|1508787463666589|2|0
https://imgur.com/|18000|14|0|14|1508787464602241|2|0
https://imgur.com/|18000|15|0|14|1508787465477303|2|0
https://imgur.com/|18000|16|0|14|1508787466335721|2|0
https://imgur.com/|18000|17|0|14|1508787467307475|2|0
https://imgur.com/|18000|18|0|14|1508787468194139|2|0
https://imgur.com/|18000|19|0|14|1508787476077778|2|0
http://www.psimadethis.com/|24000|20|0|15|1508787487760948|2|0
http://www.psimadethis.com/|24000|21|0|15|1508787489803958|2|0
http://www.psimadethis.com/|24000|22|0|15|1508787491964565|2|0
http://www.psimadethis.com/|24000|23|0|15|1508787492615127|2|0
http://www.psimadethis.com/|24000|24|0|15|1508787493596708|2|0
http://www.psimadethis.com/|24000|25|0|15|1508787512907797|2|0
http://www.psimadethis.com/|24000|26|0|15|1508787513788632|2|0
http://www.psimadethis.com/|24000|27|0|15|1508787514738562|2|0
http://www.psimadethis.com/|24000|28|0|15|1508787515571891|2|0
http://www.psimadethis.com/|24000|29|0|15|1508787516457251|2|0
http://www.psimadethis.com/|24000|30|0|15|1508787533970946|2|0
http://www.psimadethis.com/|24000|31|0|15|1508787535057143|2|0
https://bugzilla.mozilla.org/|8300|32|0|2|1508787549943460|2|0
https://bugzilla.mozilla.org/|8300|33|0|2|1508787550750170|2|0
https://bugzilla.mozilla.org/|8300|34|0|2|1508787551707928|2|0
https://bugzilla.mozilla.org/|8300|35|0|2|1508787552460847|2|0

Morphing the bugzilla bug to a places bug.