PerformanceEntries are affected by backgrounded status

[npm1]

When a tab is backgrounded, it can become throttled, it will never paint, etc. In general, browsers apply heuristics to mitigate the impact of backgrounded tabs on performance of the visible tab. However, this affects aggregation of Performance APIs: values computed during a background status are not really comparable to values computed during a visible status because the browser can behave differently in these cases.

Loading is the most affected because (1) most performance metrics involve loading in one way or another and (2) it's common to be backgrounded while loading. I think the most common case of this would be opening a new browser window with multiple tabs: all but one will load in the background.

A prominent example of this is PaintTiming. The paint timing timestamps are only computed once the process actually paints, which means that the timestamps can be inflated arbitrarily while the tab is in the background.

I think the spec needs to provide some tools to allow more easily separating visible vs backgrounded metrics or at least provide some guidance about the necessity to do so.

[tdresser]

@n8schloss - Ilya mentioned you've had some thoughts here in the past? Are you aware of any previous discussion?

[igrigorik]

@tdresser @npm1 apologies about the delay. The related thread is @ w3c/page-visibility#29

[yoavweiss]

I think the use case raised by @n8schloss is the one where we need an answer to the question: "was this page ever visible?". That seems slightly different than the use case brought here.

Here it seems like we need different entries to have different definitions to whether the were affected by the page being BG:

• PaintTiming: paint never actually happens, so no timings will be displayed.
• ResourceTiming: resources loaded while the page was in the background can be affected
• Same for Navigation Timing, but for the whole page's resources (at least the ones that block onload)
• EventTiming: dispatching time or available CPU can be impactful

So, in some cases, it may make sense to add a flag on the PerformanceEntry itself, indicating that it was impacted by the tab being in the background.
In other cases, (e.g. PaintTiming, but potentially also in RT, ET, etc.), there is no entry, or the entry can be significantly delayed. So in these cases it makes sense to indicate this elsewhere.

I'm not sure what's the best way to tackle this:

• Page Visibility already gives us a global indicating overall visibility time intervals. Existing entries can match up against that.
• At the same time, it's possible that being in the BG is preventing or delaying entries from appearing. That is information that the browser currently doesn't expose.

Maybe the missing piece here is a "was impacted by BG throttling", which each of the features would be responsible for filling in, or would be entirely UA defined. Does that make sense?

[npm1]

Page Visibility only has an event handler for visibility change. It does not say what the previous visibility state was, and to use in this case would require early event handler registration via Javascript. So the current API is not good for our use case.

[gi11es]

Based on what was discussed during the meeting, is it fair to say that being able to look up a reliable visibility history would solve this problem, by way of identifying "tainted" page views?

The current set of JS APIs doesn't allow catching all background tab status reliably, because the status during the timespan between the tab being open (possibly in the background) and when the first piece of JS can run remains unknown.

As @yoavweiss suggested, it would make sense to record "VisibilityEntries" in the performance timeline for visibility changes, as well as an origin event that would contain whether the tab was initially opened in the background of in the foreground.

[Zizzamia]

@gi11es great point!
In Perfume.js I added an onVisibilityChange to disable logging and analytics when the tab becomes hidden because any data during a visibility change become high unreliable.

[yoavweiss]

As @yoavweiss suggested, it would make sense to record "VisibilityEntries" in the performance timeline for visibility changes, as well as an origin event that would contain whether the tab was initially opened in the background of in the foreground.

To add to that, we discussed on the call that out-of-viewport may also be throttled in the future, so we would then need to change PageVisibility (if possible) to reflect that.
Another alternative is to include some kind of "was throttled" signal that may better reflect what we're trying to measure here (so instead of tying the signal to visibility, we could add a ThrottlingEntry which may correlate with visibility at first, but might diverge from it in the future and in different implementations).

[gi11es]

Indeed, entries marking throttling sound like a more generic and future-proof solution!

It probably requires having separate entries when the throttling starts and when it stops. Otherwise relying on the startTime/duration paradigm of PerformanceEntry, it would only get recorded when throttling stops, not letting us know if there is ongoing throttling if we are in the middle of it.

[npm1]

The only problem with ThrottlingEntry is that it would be harder to specify. I would propose starting with page visibility (possibly adding out-of-viewport), which would likely cover most cases?

[gi11es]

I’m not sure how « out of viewport » would work in regards to visibility entries. Pages are usually larger than the viewport, which means that there’s always something out of view.

I think it’s pretty clear what constitutes throttling, it’s whenever the browser slows down JS execution purposefully. Specific examples can be provided, like not honoring small timeout values in setTimeout, etc.

What I think is most useful about making the entries about throttling is that it represents what truly affects the performance metrics. Even right now, browsers don’t necessarily throttle background tabs, and it’s possible that when they do it’s not at the exact moment the tab is put in the background. There’s a correlation in some implementations, but all the people currently discarding performance metrics for any tab that’s been in the background at some point are discarding perfectly valid, unthrottled data out of caution. If the actual throttling was recorded, we wouldn’t have to discard more than we need to.

[igrigorik]

FWIW, I think we're converging on the ~same solution here that we came up with at last year's F2F: w3c/page-visibility#29 (comment).

One change I'd make to what we described in that issue is migrating away from buffers and to PerfObserver + buffering of events prior to onload.

[tdresser]

"I think it’s pretty clear what constitutes throttling, it’s whenever the browser slows down JS execution purposefully. Specific examples can be provided, like not honoring small timeout values in setTimeout, etc."

Throttling isn't a binary thing though. Browsers can use all kinds of heuristics to prioritize frames and tabs, many of which will slow down JS execution to some degree. I don't think it will be clear which of these should set the "throttled" bit. It certainly isn't all of them, as some heuristics will only slow things down by a minute amount. It's not unreasonable to imagine a browser in which effectively every page has it's JS throttled by some degree.

[gi11es]

In the case where every page is throttled, is it really that bad that they all get marked as such? It's important information when looking at performance metrics.

I'm not familiar enough with browser internals to know if these different types of throttling need to be categorized. But as a consumer of performance metrics, I would like to know when these deliberate slowdowns happen. If it's something large enough to affect the reliability of the RUM metrics, it should be flagged. Maybe that effect should be the criteria, rather than focusing on the nature of the slowdown.

[tdresser]

"If it's something large enough to affect the reliability of the RUM metrics, it should be flagged."

Yeah, I think this is roughly correct, but extremely difficult to spec, and it's tricky to define "large enough".

If the browsers have acceptably similar behavior for when "large enough" throttling takes place, it may be just as good (and a lot easier) to specify the signals browsers use to decide to do "large enough" throttling, instead of specifying what it means to be doing "large enough" throttling.

[mikesherov]

heres another related thread of this: w3c/paint-timing#40

[noamr]

This is also true when documents are unloaded, for the purpose of being put in session history - it is said in the spec (and in the implementation) that the document object may be cached.

I personally like the idea of having visibility and suspend / resume entries in the timeline rather than throttle - throttle may be problematic in terms of exposing browser internals and its definition may be ambiguous.

[npm1]

PageVisibility observer will be able to address the concern of this bug. I agree that exposing throttling signals and such is a more tricky subject since it's not even well defined, and hence we should aim to solve the major problem here, which is visibility.

[rniwa]

Note that WebKit may throttle fully visible web page if the user hadn't interacted with it recently in some cases as well so depending on what you're trying to measure, the visibility may not be the right thing to keep track.

[npm1]

For the purposes of validation of paint metrics, throttling is irrelevant and visibility needs to be exposed. But it's true that there are other uses-cases for which network or CPU throttling would be useful. I was thinking of moving forward with VisiblityStateEntry (see slides from WebPerf call) but after chat with @yoavweiss it might make sense to take a step back and think about a future-proof solution, so in particular the name of the entry (and the entryType value) may need to be different if we want to expose other kinds of situations, like throttling, within the same entry. We should probably discuss what the API shape for this would look like if we were to expose visibility and throttling in the future.

[bokand]

The critical difference between hidden and more general throttling is that, in the hidden case, it's not just that resources are limited but the (compositing-level) rendering pipeline is halted. That does seem like a qualitative difference and one that may be easier to describe/specify.

OTOH, I do sympathize with the argument that categorizing performance measurements shouldn't make assumptions about how they're impacted by visibility; that does over-specify the implementation for what is effectively a signal of "was the rendering pipeline blocked for reasons out of the page's control?", some examples:

• It's plausible to me that on some platforms/UAs we might submit frames to the OS compositor even for minimized windows so that hidden doesn't block the pipeline.
• There are also cases where the pipeline might be halted but the page remains visible: e.g. modal dialog boxes (e.g. IIRC, the iOS low battery modal pauses all rendering).
• Today, a foreground tab in a background (but unminimized) window is considered visible but it seems like it would be completely reasonable for the UA to halt rendering in this case.

Using visibility would break in all these cases.

I personally like the idea of having visibility and suspend / resume entries in the timeline rather than throttle - throttle may be problematic in terms of exposing browser internals and its definition may be ambiguous.

The use case for this issue is by definition trying to measure those browser internals - getting it by proxy doesn't change that.

What do folks think of using a halted pipeline (i.e. the UA isn't sending pixels to the OS compositor) as the exposed bit? It seems like we could come up with reasonably specific language for that and seems to address what was wanted from visibilityState here without tying directly to it.

FWIW pageVisibility itself is somewhat fuzzily specified (and overly desktop-centric) - it doesn't sound crazy to me to give some similar amount of wiggle room to UAs to also expose more general "isThrottled" information but the use case there seems less clear to me given the likely divergent behavior.

[npm1]

A summary of recent discussions:

• In Oct 8 meeting, we discussed The spec's definition of hidden is unclear and should be updated to better account for mobile use cases page-visibility#59 and some ideas surfaced about having a third state to cover prerendering and possibly mobile states that are currently unclear.

• @bokand presented at TPAC. I think the AI was to create a matrix with all possibilities, and the results are in Bring back support for prerendering page state page-visibility#69.

• In Dec 3 meeting David presented again, making the case for a separate prerendering bool and handler.

[npm1]

FWIW pageVisibility itself is somewhat fuzzily specified (and overly desktop-centric) - it doesn't sound crazy to me to give some similar amount of wiggle room to UAs to also expose more general "isThrottled" information but the use case there seems less clear to me given the likely divergent behavior.

It seems reasonable to me to consider exposing throttling states as a separate indicator. We do get into the concern of having too-many-things-to-look-at (visibility, prerender, throttling of various sorts, etc.)