Xray-core: More robust browser header masquerading (chrome, firefox, edge)

[PoneyClairDeLune]

As demonstrated in #5800, as soon as there is any kind of WAF rule trying to fingerprint headers from the client, merely having UA strings match will instead trigger them. This PR aims at fixing that.

• The request initiators no longer just try to use user agent strings, but supply the missing headers from the real browsers adhering to the conventions as well.
• The User-Agent header field now contains special values, namely chrome, firefox, edge, golang and an empty string.
  • Leaving the header unset or having it set to chrome will apply headers from Google Chrome.
  • firefox to set the headers to the current Firefox ESR release.
  • edge to set the headers to Microsoft Edge.
  • golang to retain the default headers from Go net/http.
  • An empty string to disable the UA header altogether.

This PR does not, however, fix the potential issue in Chrome version generation.

Original content:

Details

As demonstrated in #5800, as soon as there is any kind of WAF rule trying to fingerprint headers from the client, merely having UA strings match will instead trigger them. This PR aims at fixing that by bringing in the missing headers following the browser conventions.

However, this PR is by no means complete for the following reasons:

• The JS runtime within the browser is fundamentally incapable of initiating real gRPC requests, as such having browsers initiating compliant gRPC requests is impossible to encounter in the real world.
• Having the headers emulate Chrome ones, even when the uTLS fingerprint is set to emulate browsers other than Chrome, is ridiculous. If header masquerading is to be done correctly, an empty UA should cause the default headers to follow the one set in uTLS. Or even better, make it configurable when set the User-Agent string to following values.
  • An empty string or !utls causes the default headers to follow the intended browser set in uTLS.
  • !chrome, !firefox, !edge, !safari, etc will set the default headers accordingly.

It's merely a suggestion, but please talk to someone having experience in in-browser fingerprinting before pushing changes like this.

Edit: There's also this additional comment regarding potential pitfalls on Chromium version generation. It's also equally important for GUI clients and sharable links to include a field specifying user agent strings, as this is going to be applied by default, likely causing disruptions.

[PoneyClairDeLune]

Added default headers for both Chrome and Firefox. I don't have either Edge or Safari for their headers, but I'd be quite delighted if anyone's capable of adding them.

[PoneyClairDeLune]

I've gone forward and fully implemented the brand GREASE algorithm from the Chromium project. Header masquerading should now be pretty close to a real Chromium-based browser.

[PoneyClairDeLune]

Apart from gRPC, user agent fields in other web-based transports already had null values allowed yesterday. I did add special values in the UA field in gRPC, but it's entirely not recommended.

The question currently is, would GUI clients like v2rayN(G), OneXray etc actually allow setting the UA field without writing raw config files? And the lack of such field in the sharable link standard would also be encouraging the incoherence problem, especially when the uTLS fingerprint is not set to chrome.

Edit: Oh, and I forgot to mention, but I just added !edge to the arsenal.

[PoneyClairDeLune]

So... Is this pull request good enough to get merged?

[RPRX]

本来问题不大的，改出问题了

1. 全局默认 Chrome UA 本来就是因 gRPC 的问题而提上日程，被你改没了，还是改回来吧，有问题的话再说
2. 目前的代码看起来没有区分用户是否指定 UA 为空（至少 XHTTP 支持分享这个），gRPC 除外
3. 和 fingerprint 的配置方式对齐一下，比如 chrome 就填 chrome，不需要 !，go 改为 golang，虽然指纹配置名是 unsafe

[PoneyClairDeLune]

The current code doesn't seem to differentiate between whether the user specifies an empty User Agent (at least XHTTP supports sharing this), except for gRPC.

It does in an implicit way. Current code detects if the header hasn't been set so that Chrome can be used as default, and empty strings used to prevent UA from being set entirely will be set one step before the special value handling, so nothing is needed for that specific use case. No idea how to deal with null UA for gRPC though.

Hopefully the exclamation mark removal wouldn't get Go to scream at me again.

[Fangliding]

本来问题不大的，改出问题了

1. 全局默认 Chrome UA 本来就是因 gRPC 的问题而提上日程，被你改没了，还是改回来吧，有问题的话再说
2. 目前的代码看起来没有区分用户是否指定 UA 为空（至少 XHTTP 支持分享这个），gRPC 除外
3. 和 fingerprint 的配置方式对齐一下，比如 chrome 就填 chrome，不需要 !，go 改为 golang，虽然指纹配置名是 unsafe

主要是从真实性的角度来看chrome不可能去发出一个grpc请求 这样的流量绝对是诡异的 只是从面向CF编程的角度这样可以解决问题

[Fangliding]

UseDefaultHeadersWith和ApplyDefaultHeaders应该没必要拆开 Use 就是判断一下用户是否输入UA是否存在以及根据里面的预设值选择header组合调用ApplyDefaultHeaders 对用户没法自定义的外部请求UA本来就是空的就是走的第一条自动进chrome分支 都可以用一个UseDefaultHeadersWith

[RPRX]

毕竟 gRPC 传输层从一开始就是面向 cf 编程，而为了讨好 GFW 还得弄 Chrome CH 指纹，所以

if len(header.Values("User-Agent")) < 1 { 这个判断方式还是有问题，用户指定为空的话也会命中 @PoneyClairDeLune

[Fangliding]

空的话不应该 len 出来是1进下面的switch最后哪个case都没中就正常返回了吗（不动header）

[RPRX]

原来这个是取成员数而不是取值吗，那没事了

按照 @Fangliding 说的合并一下函数吧

[PoneyClairDeLune]

There is still a problem with this criterion. If the user specifies an empty string, it will still be satisfied.

@RPRX Empty strings will not satisfy that criterion, as that will make the total length of the value slice to be 1, with the empty string as the sole member of that slice.

There should be no need to separate UseDefaultHeadersWith and ApplyDefaultHeaders.

@Fangliding I split the two methods because it's not just the transports that are applying the headers, places like the DoH resolver and the observatory are also using that. UseDefaultHeadersWith hence is solely used to handle the transports, while ApplyDefaultHeaders is used everywhere else. Not so sure about merging the two methods together is a good idea.

[Fangliding]

DOH 的 len(header.Values("User-Agent")) 肯定是0的 又没有人去set它 我只是觉得保留两个含义相近命名的导出函数容易造成混淆

[RPRX]

确实是，我 review 的时候也觉得有点混乱，还是合并下吧

[PoneyClairDeLune]

Every single instance trying to apply the masqueraded headers now go through utils.HandleTransportUASettings(), and applyMasqueradedHeaders() is now internal to get around spaghetti code. Guess this would be good enough?

[RPRX]

@PoneyClairDeLune 看一下 review，然后没别的要改的话我就合了

[PoneyClairDeLune]

@RPRX Should be all good now once the checks are passed.

[RPRX]

合并中，@PoneyClairDeLune 更新一下 PR 正文描述

[PoneyClairDeLune]

The description of the PR was updated.