Different results in Chrome Network tab and sitespeed report

[arnaudvalle]

Hello there,

I've been trying to run sitespeed on a page behind login (I believe that's all good now) but I am seeing different results (on requests numbers for example) between doing it "manually" in my Chrome browser (and looking in the Network tab) and when doing it via the sitespeed command.

I've setup a test page with a login / password for you to have a look here: http://lumsites-sandbox.appspot.com/a/lumapps/home/sitespeed-homepage-test-private (see credentials in the preScript login script I use).

Here's the command I use:

docker-compose run sitespeed.io --preScript sitespeed-login.js http://lumsites-sandbox.appspot.com/a/lumapps/home/sitespeed-homepage-test-private -n 3 -b chrome --browsertime.chrome.args no-sandbox --browsertime.chrome.args disk-cache-dir=/dev/null --browsertime.chrome.args disk-cache-size=1 --browsertime.chrome.args media-cache-size=1 --browsertime.pageCompleteCheck 'return (function() {try { return (Date.now() - window.performance.timing.loadEventEnd) > 15000;} catch(e) {} return true;})()' --graphite.host=graphite --summary-detail

These are the versions it uses on my machine:

Versions OS: linux 4.9.8-moby nodejs: v6.9.1 sitespeed.io: 4.4.2 browsertime: 1.0.0-beta.25 coach: 0.31.0

Which gives me the following results:

Total requests 31
Image requests 3
CSS requests 4
Javascript requests 7
Font requests 0

When I do the same process manually (logging in, then going to the test page and looking up the numbers in Chrome > Network with cache disabled) I get this:

Total requests 51
Image requests 4
CSS requests 4
Javascript requests 21
Font requests 5

I suspect it has something to do with the 'warm cache' due to the login page visited during the preScript step, but if that's the case I can't see how to clear that cache completely before running the tests on my target page.

Thanks in advance for your help or any hint on what I may have done wrong.

[arnaudvalle]

For info, I get different results depending on the version of the image of sitespeed I specify in my docker-compose file. The 'Transfer size' values look especially different.

sitespeed 4.0.3

4.0.3

    Score / Metric                Median

---

✗ Overall score 67
✗ Performance score 65
✗ Accessibility score 65
✗ Best Practice score 76
√ Fast Render advice 95
! Avoid scaling images advice 90
√ Compress assets advice 100
! Optimal CSS size advice 90
✗ Total size (transfer) 1.9 MB
√ Image size (transfer) 642.6 KB
✗ Javascript size (transfer) 1.1 MB
✗ CSS size (transfer) 97.0 KB
Total requests 43
Image requests 13
CSS requests 1
Javascript requests 11
Font requests 0
200 responses 43
Domains per page 7
Cache time 10 minutes
Time since last modification -1 second
RUM Speed Index 5121
First Paint 4840 ms
Backend Time 1530 ms
Frontend Time 7155 ms

sitespeed 4.5.1

4.5.1

Score / Metric Median

---

✗ Overall score 69
✗ Performance score 68
✗ Accessibility score 65
✗ Best Practice score 76
√ Fast Render advice 95
! Avoid scaling images advice 90
✗ Compress assets advice 80
! Optimal CSS size advice 90
√ Total size (transfer) 987.1 KB
√ Image size (transfer) 664.9 KB
✗ Javascript size (transfer) 126.5 KB
✗ CSS size (transfer) 113.3 KB
Total requests 46
Image requests 17
CSS requests 4
Javascript requests 7
Font requests 0
200 responses 46
Domains per page 8
Cache time 45 minutes
Time since last modification -1 second
RUM Speed Index 4440
First Paint 2015 ms
Backend Time 1389 ms
Frontend Time 6617 ms

[soulgalore]

Hey @arnaudvalle thanks for all the info! We should move this issue to https://github.com/sitespeedio/browsertime (I can do that tonight). We parse the log from Chrome and it is possible that something can be done there. We are moving the parsing to a new repo https://github.com/sitespeedio/chrome-har when that is finished and BT uses that one, we can take on and have a look at the size diffs.

Best
Peter

[soulgalore]

Or better move it to the new repo. But I still think #1376 (comment) is valid, right now the cli command for clearing the cache isn't getting it right.

[arnaudvalle]

@soulgalore Thanks for your help - I've created an issue like you suggested in the new repo (sitespeedio/chrome-har#1).

I'll leave it up to you to close this one if you prefer avoiding unnecessary duplicates.

[soulgalore]

Yep let us close this and continue in Browsertime.