Performance impact of moving User-Agent header to Client Hint headers

[ronancremin]

Many web sites take advantage server-side identification of device type based on the User-Agent header in order to tailor responses to the requesting device. This technique is often used to tailor sites for classes of devices such as mobile, tablet and desktop. Responsive design allows for some client-side optimisations but server-side identification remains the only way to modify markup before it's sent to the client, where most gains can be made.

A recent test suggests that over 80% of the top 50 websites employ this technique, a number that's in line with previous analyses.

The proposal suggests:

Chrome could perhaps send the following for mobile, regardless of the underlying device type:

Mozilla/5.0 (Linux; Android 9; Unspecified Device) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome

…

We can ratchet this deprecation over time, beginning by freezing the version numbers in the header, then removing platform and model information as developers migrate to the alternative mechanisms proposed below.

If widely adopted, this change would remove the ability for sites to adapt to device classes on the first HTTP response, since the requisite information would only follow in subsequent requests. The first HTTP response is typically the most important, especially in the case of a new visitor. Thus this change adds a performance impact for the most critical requests. What if my service is not working for some subset of devices e.g. it's broken a way such that there is no secondary request? How do I find which devices should I look into to fix it? RFC1945 and its successors foresaw this possibility and catered for it.

The User-Agent request-header field contains information about the
user agent originating the request. This is for statistical
the tracing of protocol violations, and automated recognition of user
agents for the sake of tailoring responses to avoid particular user
agent limitations.

Furthermore, some websites also employ logic based on the User-Agent that is not at the application layer e.g. proxy servers and firewalls, in which case the second request may be too late.

[jonarnes]

Fully support Ronan here. I'm not against User-Agent client hints, but messing with the User-Agent string will break a lot of features that we have grown accustomed to over the years. The User-Agent string is, among other headers, important in content negotiation scenarios. Delaying content negotiation to subsequent requests will degrade both user experience and performance.

[foolip]

@yoavweiss, @mikewest tells me you might have thoughts on avoiding the extra roundtrip for CH generally, or maybe mitigating the impact. Can you share? :)

[foolip]

@ronancremin, the Alexa 50 UA string-based content adaptation research is really cool, is it your work? Do you know what the criteria used was, did any difference in HTML size count as adaptation, or were the sites compared by manual inspection?

FWIW, the results seem very plausible, in the top 50 I'd expect a lot of sophisticated sites that do adaptation.

[yoavweiss]

I wrote a bit about the tradeoffs here.

I have some ideas on how we could enable "very first request" negotiation using Client Hints (e.g. push some sort of an opt-in into DNS records, or to QUIC/TLS connection negotiation mechanisms). At the same time, our main focus right now is on standardizing the core Client Hints mechanism rather than those extra opt-in ones.

In the specific context of the User-Agent string, I already replied on the email thread that I believe kicked off this issue, but I don't think that current practices justify keeping the User-Agent header in its current state forever, just because "we have grown accustomed" to that.

I think it would be more productive to tease out which bits are really required for content negotiation (e.g. mobile vs. desktop form factors), understand if exposing them would be a reasonable compromise in terms of user privacy vs. required functionality/performance, and then figure out based on that what we can and cannot expose.

[foolip]

Thanks @yoavweiss! Some reactions:

I don't think that current practices justify keeping the User-Agent header in its current state forever, just because "we have grown accustomed" to that

Sure, it's just a question of what paths are blocked entirely by web compat, and which of the open paths will involve the least web developers pain.

I think it would be more productive to tease out which bits are really required for content negotiation

Do you mean required as in "freezing/removing these bits would break too much", "these bits cannot wait for a second roundtrip", or something else?

Since the idea is to eventually freeze the UA string, sites will have to adapt, and the ease of adapting is very important in addition to the performance impact. Even for some rarely used bits of the UA string there will be some legacy server that uses it, and it will have to be possible to put something in front of it that translates the new negotiation headers into old UA strings. Does that make sense?

[yoavweiss]

Do you mean required as in "freezing/removing these bits would break too much", "these bits cannot wait for a second roundtrip", or something else?

"required" as in "use-cases that are needed by a large majority of sites"

Since the idea is to eventually freeze the UA string, sites will have to adapt, and the ease of adapting is very important in addition to the performance impact. Even for some rarely used bits of the UA string there will be some legacy server that uses it, and it will have to be possible to put something in front of it that translates the new negotiation headers into old UA strings. Does that make sense?

It does, but if those rarely used bits have to wait an RTT, that may be a price we'd be willing to pay.
With that said, discussions elsewhere make me optimistic that we'd be able to mint an H2 SETTINGS based opt-in that can significantly reduce instances where that extra RTT is required.

[foolip]

Right, the most easily deployed workaround would almost certainly add an RTT. If the H2 SETTINGS approach works out then that seems fine.

What I'm still unclear about is if the server-side usage of each bit of information actually matters to the design here. Is it that if something is widely used enough, then that would block it from being frozen at all, even in the presence of a simple and RTT-free workaround?

[ronancremin]

In November 2017 Apple announced that it was freezing Safari's user-agent string with a release of Safari Technology Preview. The reaction from the community was swift and sharp, ultimately leading to a reversal of the position a few months later.

The feedback to Apple from the community around that time might inform this issue also. In summary, the rich user-agent information was being used for:

• Notifying people when they're using old browsers (for security)
• JavaScript tweaking. Netflix said: "This dooms Safari to an suboptimal loading and performance vs all other browsers."
• Working around known bugs
• Debugging & reproducing problems. How else can you know what browser version you are dealing with if you can't get code to run?
• Detecting undetectable features. "There are many bugs that cannot be feature detected in reasonable ways. For example, there are versions of safari where CSS custom variables would not update correctly and required a forced layout. Using the version number in the user agent was the only way to know."
• Content adaptation on first request
• OS detection for customer service purposes
• Analytics

You can read much of this in the Twitter thread where the announcement was made.

My sense is that this proposal will run into similar issues. The user-agent string is perhaps more widely used in the first request than people realise.

[dgoldstein0]

What is the privacy benefit if every website opts in to receiving the finer grained (device, os, minor version numbers, etc) info? The main benefit I see to the overall proposal is reducing the entropy sent over http and thus reducing the ability to passive surveillance to use user agent as a piece of a fingerprinting mechanism; which makes this another incentive to adopt https. Beyond that though... I'm not seeing the value?

I'm all for privacy, I just was expecting the proposal to require some form of user agreement (like opting into notifications or other similar rich web app features) to share finer grained info, and the value seems a lot smaller since that's not the proposed solution; and given the performance concerns listed above and in #3, I'm not sure an opt-in model would work for those use cases either.

[yoavweiss]

@dgoldstein0 - your questions don't seem directly related to this thread. I've answered them below, but f further discussion is required, please open a separate issue.

What is the privacy benefit if every website opts in to receiving the finer grained (device, os, minor version numbers, etc) info?

I don't believe every website will opt in to receive finer grained info, similar to the way not every website is accessing fingerprintable information browsers expose today.
For the ones that do, there will be privacy benefits knowing that these sites actively access that information, rather than having all website receive it passively.

I just was expecting the proposal to require some form of user agreement (like opting into notifications or other similar rich web app features) to share finer grained info

There are no current permissions to access fingerprinting information the different browsers expose, and it would be very tricky to explain to users what we are asking them to consent to here. Turning this information from passive to active is a clear step in the right direction, and will also enable browsers to take measures against sites that are accessing a lot of active fingerprinting vectors.

[yoavweiss]

What I'm still unclear about is if the server-side usage of each bit of information actually matters to the design here. Is it that if something is widely used enough, then that would block it from being frozen at all, even in the presence of a simple and RTT-free workaround?

@foolip - apologies but missed your question many months ago...

It's a tradeoff between use and exposed entropy. If something is not widely used, the performance hit of a very-first-view opt-in will not be huge. If something is widely used but exposes many bits, we may prefer not to expose it despite its wide usage. But if something is widely used and exposes very few bits, that seems like something we can afford to expose (from a privacy perspective) in order to prevent the performance costs of the opt-in.

[yoavweiss]

• Notifying people when they're using old browsers (for security)
• JavaScript tweaking. Netflix said: "This dooms Safari to an suboptimal loading and performance vs all other browsers."
• Working around known bugs
• Debugging & reproducing problems. How else can you know what browser version you are dealing with if you can't get code to run?
• Detecting undetectable features. "There are many bugs that cannot be feature detected in reasonable ways. For example, there are versions of safari where CSS custom variables would not update correctly and required a forced layout. Using the version number in the user agent was the only way to know."

All that seems feasible using the UA info (brand and major/feature version) that we already plan to expose.

• Content adaptation on first request

That varies wildly by what you mean by "content adaptation". It seems like a common case for that would be "mobile vs. desktop". We've been discussing adding the "mobileness" bit to the hints that are sent by default, which may help there.

• OS detection for customer service purposes

That doesn't seem like something that required for the very-first-hit on a site's landing page

• Analytics

UA info will still be available by default for server-side. Anything else, will require the site to opt-in, but tbh, I'm not sure that e.g. OS analytics are worth touching those extra bits of user information.

[foolip]

That calculus makes sense to me if the performance hit is unavoidable, and it sounds like it is then.

[yoavweiss]

Since the majority of use-cases are well covered by exposing the UA brand and meaningful version by default (and perhaps the mobileness bit), the performance hit on the very-first-view would be limited to a small number of use-cases.

That is not to mean that the potential performance hit is being ignored. I opened WICG/client-hints-infrastructure#16 to track the design of a mechanism that will enable us to avoid it.

But, I don't think that we should block Client Hints in general and UA-CH in particular on such a mechanism. I'm therefore closing this issue.