Media query evaluation in browser preloader will hurt performance #9

Closed
igrigorik opened this Issue Nov 3, 2012 · 16 comments

Comments

Projects
None yet
8 participants

I've brought this up across several threads in bits and pieces, but here is a (more) coherent document describing why changing the current preloader behavior will, in fact, hurt performance:

https://docs.google.com/document/d/1HanD5V0ousYmHZNPt21myXtGm5IKxXr6CNGBoWVj8Sk/edit

Conserving bytes is an important consideration, and I hope we can find a way to address this case. But conserving bytes is not the only consideration, there is also battery life, incurred latency, and inconsistent UX.

Related tickets: #7, ResponsiveImagesCG/ri-usecases#16

The root issue is that the current proposal conflates two issues: (a) downloading resources which you may need later, and (b) downloading resources which you won't need at all. The best behavior for (a) is as implemented currently (fetch all on initial load), and (b) is what we need to figure out how to solve.

Owner

yoavweiss commented Nov 3, 2012

Thanks for the well thought out issue & document.
I'll try to answer your points here:
1)
Your point is that avoiding download of "possibly needed" resource in the start of the download, will hurt performance in case of an MQ change (e.g. orientation change)., mainly because of radio start-up times.
IMO MQ based resources can be divided into 4 resource groups:

  • required - resources that should be displayed to the user straight away
  • probable - resources that will be probably needed when the user starts interacting with the site (e.g. "landscape" resources on a tablet, where orientation change is highly probable)
  • possible - resources that may be needed during interaction, but with low probability (e.g. window resize event for desktop, etc)
  • Unlikely - Resources that are unlikely needed during interaction (e.g. orientation change on devices where orientation change is currently disabled, screen change when device is hooked up to a projector)

I think we can all agree that "required" should be downloaded straight away, and "unlikely" should be downloaded on demand. IMO, the "possible" should also be loaded on demand, and the "probable" resource should probably downloaded with "PREFETCH" priority (i.e. after all other resources were downloaded) where it makes sense (e.g. on 3G/4G, where the user is not roaming and has unlimited data plan), possibly related to some user preference.
In any case, this part can and should be optimized by the browser to accommodate all the various trade-offs, but I believe it's not "all or nothing".

a. Your point is that today no real MQ evaluation happens in the PreloadScanner. That is true, but it does not have to be. I have played around with the PreloadScanner and by wheeling in the document's Frame and RenderStyle, I was able to evaluate real MQs in the PreloadScanner. You can see the code at: https://github.com/ResponsiveImagesCG/RespImg-WebCore/blob/master/html/parser/HTMLPreloadScanner.cpp
I haven't yet exhaustively tested it for all MQs, but it seems to work pretty well for all "screen" MQs that I have tested.
I'd love your input on that. Binaries to play around with are also available in the "Downloads" section.

b. The @viewport rule does impose a problem in that aspect and may change MQ evaluation that are based on the viewport width after the resources were preloaded. I'm still hashing this one out, but a simple (even if confusing) way out is to advocate to Web developers to always use this rule as part of an inlined <style> rather as an external CSS.

c. I agree that all the above will increase Preloader complexity & CPU usage, but I'm not sure it's prohibitive.

Finally, in my personal view, the above should also be true for external CSS, and for future resources that may have a media attribute.

I think we can all agree that "required" should be downloaded straight away, and "unlikely" should be downloaded on demand

No, that's my entire point. These resources must be downloaded at the beginning for best performance on mobile apps. What we're discussing here is the default behavior for the browser, and the default behavior should be optimized to deliver the best experience: least unexpected and reliable performance results. Don't forget that you can get the behavior you want by moving this logic to the client - if you want to do that, that's fine.

Finally, in my personal view, the above should also be true for external CSS, and for future resources that may have a media attribute.

No. For the same reasons above. Once again, you can get this behavior if you want, today - no modifications required.

P.S. A good explanation of some of the considerations for optimizing network access on mobile: http://developer.android.com/training/efficient-downloads/efficient-network-access.html

Owner

yoavweiss commented Nov 4, 2012

Just to make sure I understand you correctly:
If, for example, a certain picture element contains one resource for viewport width below 1500px and another for viewport width above 1500px, in your view both resources should be downloaded for a device with initial viewport of 600px in portrait and 800px in landscape, since the device may be connected to an external screen with resolution that exceeds 1500px?

If we can reliably eliminate conditions which will never be satisfied on the page once the page is loaded, then there is no reason to download them. But just to be clear: if the stylesheet can be applied at later time (orientation change, or even print), then it should be downloaded at the start.

The external screen example is a tricky one because the behavior will vary based on your windowing system, but for example.. on OSX, connecting to a larger screen does not automatically resize your windows. This behavior, is in part to "address" this issue.. But that's a complete aside.

Coming back to the "reliable" part. If we think the conditions we're after are effectively limited to dip counts, then there is no reason to bring in the entire MQ evaluation machinery. Perhaps its worth thinking about a different attribute which could serve as a simple evaluation attribute against the device dip count - not unlike what the current meta viewport tag does.

Owner

yoavweiss commented Nov 4, 2012

Regarding different machinery than MQs, I'm open for suggestions, as long as it is something accepted by the Web developer community. I also heard from browser vendors that the fact that MQs constitute a "contract" as opposed to srcset's "hints" gives browser more room for optimization in case the different sources are the same image in different resolutions.

Machinery aside, the issue you're raising is a principle one. Should most users download unnecessary data, just because a small minority might need it, and suffer from some lag in its absence?

Let's add some numbers to the example case above. We can assume that 99% of mobile users don't dock their device to a big screen. We can also assume that the "mobile" resource is 20K, while the "big screen" resource is 200K.
In this case, you suggest that it is better to let 99% of users download an extra 200K (and possibly pay the mobile charges for this data) than to let 1% of users wait an extra 2 secs for 3G warmup, and another 1-3 seconds to download the image once they need it.
You can replace "big screen" with "print", if the external screen example is tricky, but the issue remains the same. Should the majority of users suffer extra data download in order to minimize lag for the edge cases?

It might also be possible to resolve this with a mix of browser heuristics & user preferences. E.g. by default the browser would download only the "required" and "probable" resources. If the user has a history that increases his chances of needing "possible" or "unlikely" resources (history of printing/docking/window resizing/other) or a "optimize for speed on the expense of data savings" preference, the browser can prefetch other resources as well.

All in all, I hope we can agree that the trade-off here is between lag in some cases and extra data download in others. I think that a middle ground can be found. IMO, downloading all resources as part of the initial load is too aggressive, and will result in a situation that is even worse than the "just serve retina images for everyone" approach that exists today.

Owner

Wilto commented Nov 4, 2012

I’m curious about why WebKit (somewhat) recently changed the default behavior of background images within media queries from the approach @igrigorik is describing to the approach we’re advocating for picture—they’re now requested as-needed. I’m having trouble digging up the bug or commit, but I’ll keep digging.

Unfortunately, we’re sort of at mutual dead-ends in all this: we’re speaking purely in the theoretical based on what we know about the delivery of stylesheets—much smaller assets, where inapplicable requests don’t stand to do anywhere near as much damage to users’ bandwidth. Perhaps there’s some way we could put together a test case illustrating both behaviors? @igrigorik, can you point to some resources where we could learn more about the incurred costs of making asynchronous requests, so we can try to cobble together a representative test? It would make for an interesting read anyway, as it basically speaks against using AJAX in a mobile context.

@yoavweiss: we've deviated from the original discussion here, and you're putting words into my mouth.. which I didn't say. The external screen example is definitely a case to consider, but it's not the most pressing case. The more important one is, for example, the orientation case, which is something you guys have highlighted in the use case -- which, also happens to be a much, much more frequent case.

I come back to, once again: if we can reliably eliminate conditions which will never be satisfied by the device rendering the page, then there is no reason to download those resources. Meaning, all orientations (that are possible given the viewport) should be downloaded to minimize impact once the orientation change occurs (and to save battery and avoid variable latency). But, if I attach a new physical screen, and my windowing system changes all of the viewport settings (and perhaps even a retina screen), then this is a separate discussion - let's not sidetrack on this one, yet.

Re, different machinery: what about the srcset width and height hints? Why is this not sufficient?

@Wilto: bandwidth is important from cost perspective. However, bandwidth doesn't have to be in the way of a fast first paint of the page - we can paint the page, but continue downloading low-priority resources in the background. The energy and latency savings are anything but theoretical. Here are a few resources worth reading:

Owner

yoavweiss commented Nov 4, 2012

@igrigorik I apologize if I misunderstood or misinterpreted what you've written. I had no intention to put words in your mouth.
Regarding the orientation change case, I believe you're right and there are cases where is makes perfect sense to fetch images for both orientations. I also believe that there are cases where it does not make sense (e.g. roaming user with orientation switch disabled), and differentiating between the two is in the realm of the browser's prefetching mechanisms.
My main issues with srcset width & height hints (besides the resistance they've encountered in the dev community) is that they don't enable the "art-direction" case, at least not in a straight-forward way.

@yoavweiss right, if the user has an explicit toggle to do an orientation lock, or some other browser mechanism (ex, prefer low-res images because I'm roaming), then this would fall under "we can reliably eliminate this resource" rubric. I'm all for that. Having said that, I'm not yet 100% that this is best solved at preloader / doc parser level - server-side adaption could fill this gap even better.

But speaking of saving bytes and art-direction.. You know, if we really wanted to save bytes, then we would reuse the same image across both orientations - after all, fetching two crops of the same image is duplicate content. If we restricted the case to same visual pixels but different "viewports for the image", then doing a client-side crop (ala css sprite positioning), would actually be the most efficient route from network perspective. Hmm?

Owner

Wilto commented Nov 5, 2012

@igrigorik These links are great; thanks so much!

Mind you, I stand by the idea that we’re imposing a huge cost on every user ever to hit the page, for the sake of the relatively small number that are gonna re-orient—likely only once while browsing a site, only to spin up the radio one additional time to request asset(s), and even then: only in the event that there are separate breakpoints for the portrait/landscape orientation of that device.

However, I also understand that there’s no such thing as an “inapplicable” media query. We can’t make assumptions there. But when we consider the size of high-resolution images designed for a Retina MacBook being served to users on a lower-end Android device and a per-kilobyte data plan: always serving everything to everyone is an untenable approach, even if done at a lower priority. We’d be looking at several megabytes served to every single user on every single page every single website using picture, for the sake of the occasional user reorienting their device one time, which may or may not mean an additional request.

Don’t get me wrong, this is genuinely great feedback—this is exactly the kind of data we’re looking for, and it’s hugely helpful. We’re not entrenched in terms of any one approach; our goal is to find the best possible solution for the all users of the web (but hey, no pressure or anything).

I think it might be best if we move this discussion to the mailing list. This is a problem that needs to be carefully considered in terms of both picture and srcset, and we should loop more people into the conversation. I know I’ll continue to research this in the meantime.

@Wilto do we have any data to prove or disprove the fraction of orientation changes? Amongst the different threads, I've seen people claim both sides: it happens all the time, and it doesn't... Doesn't seem like a hard thing to quantify for anyone with a semi-popular mobile site.

High-resolution images + retina: this is an orthogonal concern, which is already addressed by srcset. The UA can decide which one it wants to fetch, based on device properties, or other user specified considerations.

Agree with Mat's comment here. It's essential to keep loading performance at a high priority, but overall download size is a real concern that users care about too - perhaps often even more than they care about perceived loading speed. My cell data plan caps at a certain number of megs per month, and that's very common nowadays not only with prepaid fixed-data sim cards, but also with the growing commonality of shared data plans across one's personal devices. I already self-restrain my media consumption when I'm not on wifi to ensure I don't use it up. Anything we can do to prevent waste will be appreciated.

beep commented Jun 26, 2013

Agree with Mat's comment here. It's essential to keep loading performance at a high priority, but overall download size is a real concern that users care about too - perhaps often even more than they care about perceived loading speed.

+1 from me.

Owner

yoavweiss commented Jun 26, 2013

@scottjehl @beep Hey! :)
There is no technical problem preloading only the required image resources.
Whether or not the optional resources (ones that may be needed later upon an orientation change, etc) should be loaded as low priority resources is up to the browsers (like today's CSS).
However, since image have a much higher price tag than CSS files, IMO, downloading them should be something that the browsers do only after the user expresses explicit interest in that (like Chrome does today with preloading/prerendering on Android).
I'm not sure specifying that is something that falls under the picture spec, though (Nor am I sure this should be specced at all)

Member

aarongustafson commented Jun 26, 2013

+1 from me too. The implementors should make the call on how it’s done, but conserving bandwidth is key.

On Wednesday, June 26, 2013 at 12:29 PM, Yoav Weiss wrote:

@scottjehl (https://github.com/scottjehl) @beep (https://github.com/beep) Hey! :)
There is no technical problem preloading only the required image resources.
Whether or not the optional resources (ones that may be needed later upon an orientation change, etc) should be loaded as low priority resources is up to the browsers (like today's CSS).
However, since image have a much higher price tag than CSS files, IMO, downloading them should be something that the browsers do only after the user expresses explicit interest in that (like Chrome does today with preloading/prerendering on Android).
I'm not sure specifying that is something that falls under the picture spec, though (Nor am I sure this should be specced at all)


Reply to this email directly or view it on GitHub (#9 (comment)).

Member

anselmh commented Jun 27, 2013

I agree with @yoavweiss here – we should not care about how something is implemented in specifications. This is something platform specific and may vary.
However loading initially required images should have high priority as already said in consensus here.

baamenabar referenced this issue in ResponsiveImagesCG/meta Aug 20, 2013

Closed

Add a tool or reference to test browser preloader. #21

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment