Unrealistic fast TTFB when using connectivity cable for some runs

[soulgalore]

Hi Pat,
we got an issue where we sometimes get super fast unrealistic TTFB and wonder if we could be doing something wrong?

We are comparing and testing out two different versions of a page to see how our changes affect SpeedIndex & render. We configure our runs like this:
runs: 51 (we tested different amount of runs to try to get stable values).
fvonly: 1
medianMetric: SpeedIndex
medianRun: median
We use Chrome. We use connectivity cable, 2G and 3G. We run on AWS Europe.

Here:s an example of a run that gets TTFB 0.055s:
http://wpt.wmftest.org/result/160222_V7_7D/40/details/

More realistic times seems something like 200ms. It happens so often so that sometimes that run pop up as our median run. I'm pretty sure this happens only when we run on Cable for Chrome, I haven't seen it on 2G and 3G. We also test on Firefox but I can't see the TTFB there since we are using SPDY.

Should we run our tests in another way, is there a way to make sure this doesn't happen?

[pmeenan]

The issue is coming from Chrome's dev tools reporting of the timings and it looks like it is missing the start of the request (including socket and DNS) for some reason. The real fix is for me to move away from using dev tools for the timings for HTTPS requests and I'm really close to having that working (though then it will be like Firefox and SPDY will be an issue).

[soulgalore]

Ah ok, thanks. Well it's not so long until May 15.

Just of curiosity (I haven't checked how WebPageTest creates HARs): In the coming version of Browsertime we parse the performance log from Chrome to generate the HAR. Is devtools based on the same or is it a chance that these numbers will be "better"? You don't need to answer if you don't want :)

[pmeenan]

WebPageTest constructs it's own HAR's manually from the performance timing it records. Which "performance log" are you referring to from Chrome? If it is using the remote dev tools network.* events then it is likely to be the same as you're seeing in WPT right now.

[soulgalore]

Ah thanks. The "performance" log is the name you use when you configure Chrome in Selenium to get the network.* events log.

[nathanbower]

I'm encountering the same issue on occasion. Thanks for the fast attention to the issue, Patrick!

[soulgalore]

@nathanbower does it happen only for cable connectivity or do you see it for others also?

[nathanbower]

@soulgalore I've only noticed this for cable connectivity.

[pmeenan]

This should now be fixed for HTTP 1/2. I just finished implementing the code to get the decrypted byte streams for Chrome. The root cause of this specific issue may have been a navigation timing bug in Chrome 48 that was not noticed until after it shipped where it was sometimes trimming the start time of the navigation incorrectly (fixed in 49 from what I understand).

It will still be an issue for SPDY though. @soulgalore, if you think you will be using SPDY for a while even after deprecation I can see about integrating spdylay to get native support for SPDY as well.

[soulgalore]

Ah thanks. I'm not sure exactly when it will happen but when we have more h2 than spdy users we'll switch. I'll get back if it's going to be long after deprecation. Thanks again!

[soulgalore]

Lets close this for now, I haven't seen it in a while but I haven't run the amount of tests as I did before though.