Frequency of tests

[gety9]

Guys hi,

Not an issue, but more of a question

What is recommended frequency of tests?

If we'd like to test only 3 times a day should it be during peak load (when most visitors visit the website) or or least load periods?

[beenanner]

Depends on your visitors traffic patterns, caching, and how much your budget is willing to afford. I would try to have more than 1 run say 3 to 5 runs to get an avg for any single test and at a minimum you probably want 3 to 5 tests a day to get an idea of site impact throughout any one day. Remember the more you can test the more you can potentially learn about your user's experience and improve it!

[soulgalore]

@beenanner I think in general we are missing out on documentation on a high level, I've started "performance testing in practice" docs that aims to be tool independent, I'll share the branch in a couple of weeks when I had time to get further. @gety9 question is perfect for that.

I also would like to start to try out organising parts of the documentation as run books https://en.wikipedia.org/wiki/Runbook - Timo at work has started doing it for our performance related stuff and I think we could do the same.

[gety9]

@beenanner

thanks for reply, for now i made 3 tests a day 5 runs each.
The reason i asked the question cause when looking at dashboard.sitespeed.io
i see that wikipidia.com stats vary greatly by time of the day so for this kind of site hourly might be needed, but for ryanair it's pretty stable.

@soulgalore
it would be great!

One more question not related to this one.
What is the difference between transfer size and content size, and why are they so different for js here
https://puu.sh/CRtgf/dbf952dacd.png
?

[beenanner]

Transfer would take in account gzipping

[soulgalore]

The variation usually depends on either the server you use to run the test and the content that is served. For most sites First Visual Change is a ok metric to focus on since best case ads etc will not change that metrics (if it do you need to check your inplementetion).

Best
Peter

[gety9]

@beenanner @soulgalore

Transfer would take in account gzipping

got it, thank you

For most sites First Visual Change is a ok metric to focus

got it, thank you

One more question:

When recording site speed it must be exactly same website that users see (including all tags - like GTM, Google Analytic, FB pixel, Mouseflow, etc) - correct?

We have interal cookie that disables all tags, and that's what we were planning to use not to litter analytics/stats with sitespeed tester visits, but this way i guess results will not be 100% accurate.

So i as i understand we must enable all tags, and than filter this traffic inside the tools (GA, FB pixel, etc) .
Did i understand this right?

[soulgalore]

Aha, I think you should test both then, because then you can more easily keep track of how much the tags impact performance. It's easier then to see if some of your own code changes impact the performance vs the tags.

Best
Peter

[gety9]

@soulgalore

1

I think you should test both then

looked more into it, and tools (GA, FB pixel, etc) do not have option to exclude by cookie, so the only way is to attach static IP to testing instances and exclude by IP. But than you have to pay for unused IPs when instances are stopped. i guess we will just test with tags disabled for now (will disable all tags in GTM which has option to filter by cookie).

2
Also in your experience do slower speeds result in lesser location specific site speed difference?
For example first visual change on our desktop site with cable connection throttling is about ~50% faster for ohio (US) vs london(UK).
But for mobile site with 3g throttling there is no difference for first visual change between locations, and last visual change average for some pages is faster for london, which is a little strange to me.
https://puu.sh/CTuBM/3e9886f7d7.png

Is it expected behavior?

(our server is in US, and we don't have CDN for content, so london should be slower)

[soulgalore]

2

Yeah that seems strange, look into the HAR/waterfall and verify that everything looks right (look at the ttfb/download times to see that the throttling/connectivity really works). Adding latency should make things slower so the longer from the source, the slower load.

[gety9]

@soulgalore

look into the HAR/waterfall and verify that everything looks right (look at the ttfb/download times to see that the throttling/connectivity really works)

Do you mean to look at "receive"? For html itself and first css file london's receive is much much faster than for ohio https://puu.sh/CTwCr/554f40b012.png

Adding latency should make things slower so the longer from the source, the slower load.

that's what i expect too

[soulgalore]

Sorry I missed how did you setup https://www.sitespeed.io/documentation/sitespeed.io/connectivity/ ?

[gety9]

@soulgalore

Seems like problem was with how docker resolves DNS on different Ubuntu versions.
moby/libnetwork#2187

After Ubuntu update on london instance results look more trustworthy:
https://puu.sh/CUKhs/8c4e085730.png
Could you please confirm that now everything looks right?
(it's static page with 3 images and 1 css file)

Also do i understand it correctly that when comparing 3g ohio and 3g london that "receive" should be about the same, and difference will be in "connect" and "wait" ?

[soulgalore]

So what kind of connectivity are you using, you didn't answer that? If you don't set a connectivity it will run on the current internet connection on the host.

[gety9]

@soulgalore

we are using docker network
https://www.sitespeed.io/documentation/sitespeed.io/connectivity/#docker-networks

[soulgalore]

Anything more needed from us here?

[gety9]

@soulgalore

we've managed to solve it,
problem was OS version on instance server