Skip to content
This repository has been archived by the owner on Sep 6, 2019. It is now read-only.

Restricting the user agent for Chromium does not work #825

Closed
BlueMax opened this issue Nov 25, 2013 · 41 comments
Closed

Restricting the user agent for Chromium does not work #825

BlueMax opened this issue Nov 25, 2013 · 41 comments

Comments

@BlueMax
Copy link

BlueMax commented Nov 25, 2013

CM11 (24.11.2013)
Galaxy SIII - I9300
XPrivacy 1.10.27
Xposed 2.4 beta 1

@M66B
Copy link
Owner

M66B commented Nov 26, 2013

@BlueMax can you please confirm if this version can restrict the user agent on KitKat: http://d-h.st/vHJ

@BlueMax
Copy link
Author

BlueMax commented Nov 26, 2013

Yep, gimme a minute...

@BlueMax
Copy link
Author

BlueMax commented Nov 26, 2013

Doesn't work... Also, the user-agent permissions are not present anymore. There are only two right now. There used to be 6 or so...

browser

@M66B
Copy link
Owner

M66B commented Nov 26, 2013

About the restrictions: that is correct, getSettings has replace the previous present functions.

Did you reboot your device after installing the test version of XPrivacy?
Can you enable XPrivacy logging (settings menu) and capture a logcat like this:

adb shell "logcat | grep XWeb"

@M66B M66B reopened this Nov 26, 2013
@M66B
Copy link
Owner

M66B commented Nov 26, 2013

The screenshot shows getSettings was actually called.

@BlueMax
Copy link
Author

BlueMax commented Nov 26, 2013

Yep, i did reboot.

I/XPrivacy/XWebView( 1927): android: hooked android.webkit.WebView.getSettings/view (1)
I/XPrivacy/XWebView( 6438): Hooking com.android.webview.chromium.ContentSettingsAdapter
I/XPrivacy/XWebView( 6438): Hooking getUserAgentString
I/XPrivacy/XWebView( 6438): Hooking setUserAgentString
I/XPrivacy/XWebView( 6438): Queue usage data=10027/view/getSettings=true size=4
I/XPrivacy/XWebView( 6438): get 10027/getSettings view=restricted (file)
I/XPrivacy/XWebView( 6438): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6438): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6438): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6438): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6438): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6438): Queue usage data=10027/view/getSettings=true size=7
I/XPrivacy/XWebView( 6438): get 10027/getSettings view=restricted (file)

useragent

usagedata

@M66B
Copy link
Owner

M66B commented Nov 26, 2013

@BlueMax can you please try if this version works: http://d-h.st/EvR

@BlueMax
Copy link
Author

BlueMax commented Nov 26, 2013

Still no go...
Also, new permission 'WebView.constructor' has no orange triangle.

I/XPrivacy/XWebView( 1928): android: hooked android.webkit.WebView.constructor/view (4)
I/XPrivacy/XWebView( 1928): android: hooked android.webkit.WebView.getSettings/view (1)
I/XPrivacy/XWebView( 6489): Queue usage data=10027/view/WebView.constructor=true size=4
I/XPrivacy/XWebView( 6489): get 10027/WebView.constructor view=restricted (file)
I/XPrivacy/XWebView( 6489): Hooking com.android.webview.chromium.ContentSettingsAdapter
I/XPrivacy/XWebView( 6489): Hooking getUserAgentString
I/XPrivacy/XWebView( 6489): Hooking setUserAgentString
I/XPrivacy/XWebView( 6489): Queue usage data=10027/view/getSettings=true size=5
I/XPrivacy/XWebView( 6489): get 10027/getSettings view=restricted (file)
I/XPrivacy/XWebView( 6489): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6489): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6489): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6489): get 10027/WebView.constructor view=restricted (cached)
I/XPrivacy/XWebView( 6489): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6489): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6489): Queue usage data=10027/view/getSettings=true size=8
I/XPrivacy/XWebView( 6489): get 10027/getSettings view=restricted (file)
I/XPrivacy/XWebView( 6651): Queue usage data=10027/view/WebView.constructor=true size=4
I/XPrivacy/XWebView( 6651): get 10027/WebView.constructor view=restricted (file)
I/XPrivacy/XWebView( 6651): Hooking com.android.webview.chromium.ContentSettingsAdapter
I/XPrivacy/XWebView( 6651): Hooking getUserAgentString
I/XPrivacy/XWebView( 6651): Hooking setUserAgentString
I/XPrivacy/XWebView( 6651): Queue usage data=10027/view/getSettings=true size=5
I/XPrivacy/XWebView( 6651): get 10027/getSettings view=restricted (file)
I/XPrivacy/XWebView( 6651): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6651): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6651): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6651): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6651): get 10027/WebView.constructor view=restricted (cached)
I/XPrivacy/XWebView( 6651): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6651): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6651): Queue usage data=10027/view/getSettings=true size=8
I/XPrivacy/XWebView( 6651): get 10027/getSettings view=restricted (file)

@BlueMax
Copy link
Author

BlueMax commented Nov 26, 2013

Wait, wait, wait.... It works. something is cached. If i change the user-agent (switch to desktop version) it shows faked data.

I will try previous version v1.10.28.1.

@M66B
Copy link
Owner

M66B commented Nov 26, 2013

Could you capture a log like this:

adb shell "logcat | grep XPrivacy"

start just before you run the browser, but reboot first, so I can see the initial hooking too.

@M66B M66B closed this as completed Nov 26, 2013
@BlueMax
Copy link
Author

BlueMax commented Nov 26, 2013

It just works when i switch to 'desktop' user-agent. And now even the 'WebView.constructor' permission is lit. But when i switch back to normal/mobile user-agent its back to unfaked state.

Correction: It just works for one time (right after switching user agent in browser - no matter what user agent). Reloading the page resets to original state. Switching the user-agent in the browser gives another one-time fake.

@M66B
Copy link
Owner

M66B commented Nov 26, 2013

Can you provide me with a logcat for the switch?

@BlueMax
Copy link
Author

BlueMax commented Nov 26, 2013

Sure...
http://pastebin.com/vfEGruYJ

What has been done:

  1. logcat started
  2. Browser loaded page with original user agent
  3. Agent switched manually in browser
  4. Page gets automatically reloaded (shows faked user agent=ggg)
  5. logcat aborted

M66B pushed a commit that referenced this issue Nov 26, 2013
@M66B
Copy link
Owner

M66B commented Nov 26, 2013

Another test version: http://d-h.st/rES

(sorry, I cannot test myself, since I don't have a device that can run KitKat yet)

If it not works, please make a new logcat:

adb shell "logcat | grep XWeb"

@BlueMax
Copy link
Author

BlueMax commented Nov 26, 2013

Indeed.. :)
Even the switch doesn't work. Its crashing somewhere...

I/XPrivacy/XWebView( 1927): android: hooked android.webkit.WebView.constructor/view (4)
I/XPrivacy/XWebView( 1927): android: hooked android.webkit.WebView.getSettings/view (1)
I/XPrivacy/XWebView( 6563): Queue usage data=10027/view/WebView.constructor=true size=4
I/XPrivacy/XWebView( 6563): get 10027/WebView.constructor view=restricted (file)
I/XPrivacy/XWebView( 6563): Hooking com.android.webview.chromium.ContentSettingsAdapter
W/System.err( 6563): at biz.bokhorst.xprivacy.XWebView.after(XWebView.java:72)
W/System.err( 6563): at biz.bokhorst.xprivacy.XWebView.after(XWebView.java:60)
I/Xposed ( 6563): at biz.bokhorst.xprivacy.XWebView.after(XWebView.java:72)
I/Xposed ( 6563): at biz.bokhorst.xprivacy.XWebView.after(XWebView.java:60)
I/XPrivacy/XWebView( 6563): get 10027/WebView.constructor view=restricted (cached)
I/XPrivacy/XWebView( 6563): Queue usage data=10027/view/WebView.constructor=true size=7
I/XPrivacy/XWebView( 6563): get 10027/WebView.constructor view=restricted (file)

M66B pushed a commit that referenced this issue Nov 26, 2013
M66B pushed a commit that referenced this issue Nov 26, 2013
@M66B
Copy link
Owner

M66B commented Nov 26, 2013

Another test version: http://d-h.st/Pll

@M66B M66B reopened this Nov 26, 2013
@BlueMax
Copy link
Author

BlueMax commented Nov 26, 2013

Still only for one time...

I/XPrivacy/XWebView( 6757): Queue usage data=10027/view/WebView.constructor=true size=1
I/XPrivacy/XWebView( 6757): get 10027/WebView.constructor view=restricted (file)
I/XPrivacy/XWebView( 6757): Hooking com.android.webview.chromium.ContentSettingsAdapter
I/XPrivacy/XWebView( 6757): Hooking setUserAgent
I/XPrivacy/XWebView( 6757): Hooking setUserAgentString
I/XPrivacy/XWebView( 6757): Queue usage data=10027/view/getSettings=true size=2
I/XPrivacy/XWebView( 6757): get 10027/getSettings view=restricted (file)
I/XPrivacy/XWebView( 6757): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6757): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6757): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6757): get 10027/WebView.constructor view=restricted (cached)
I/XPrivacy/XWebView( 6757): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6757): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6757): get 10027/getSettings view=restricted (cached)
I/XPrivacy/XWebView( 6757): Queue usage data=10027/view/getSettings=true size=1
I/XPrivacy/XWebView( 6757): get 10027/getSettings view=restricted (file)
I/XPrivacy/XWebView( 6757): get 10027/getSettings view=restricted (cached)

@M66B
Copy link
Owner

M66B commented Nov 26, 2013

:-(

M66B pushed a commit that referenced this issue Nov 26, 2013
M66B pushed a commit that referenced this issue Nov 26, 2013
@M66B
Copy link
Owner

M66B commented Nov 26, 2013

Unfortunately I am out of ideas how to fix this at the moment.
I will think about it, maybe I will get a good idea.

@gdmzyejian
Copy link

Also does not work in version 4.12

@M66B
Copy link
Owner

M66B commented Dec 1, 2013

@gdmzyejian :
I am using CM10 = Android 4.1.2 and it works perfectly for me.
Please provide a logcat.

@BlueMax
Copy link
Author

BlueMax commented Dec 1, 2013

On KitKat its the following scenario:
If i start the browser with https://www.startpage.com/m, search for 'user agent' and go to www.useragentstring.com (5th link, creates new tab) the user agent is original/untouched. But when i copy the URL (www.useragentstring.com), restart the browser and paste the URL directly into the browser the user agent string is faked. Also, most sites i enter by bookmarks are faked properly.

Its probably somewhat related to the new tabs. I created several stack traces, forced all Set-/GetUserAgent, parent/child methods and linked lists/hashtables with constants but there's still an alternative runtime path. I couldn't find a proper target.

The browser uses an internal counter for custom user agents (overriding system static string from framework-res.apk). If that value is '0' system static settings are in effect. I traced it back and forth into several system parts browser.apk/framework-res.apk/WebviewChromium.jar and also stumbled upon native methods but whatever i touched the original user agent always came through. The shared/static system settings are kinda hard to trace offline. They use exceptions and abstract methods to sync the settings and i don't have a proper debug chain installed.

My theory is that the browser omits filling the user agent value on purpose so that the system fills it up with static system values. But i couldn't test it yet...

@M66B
Copy link
Owner

M66B commented Dec 1, 2013

@BlueMax good work!

Maybe getUserAgent(String) needs to be hooked to fix this.

Did you find useful class/method names to look into?
Chromium is open source, so we can look things up.

@BlueMax
Copy link
Author

BlueMax commented Dec 1, 2013

Didn't you already hooked getUserAgentString? It's present in the pastebin log above.
I haven't found any new methods i haven't had verified already being ineffective as well. I will poke around some more later.

M66B pushed a commit that referenced this issue Dec 2, 2013
M66B pushed a commit that referenced this issue Dec 2, 2013
@M66B
Copy link
Owner

M66B commented Dec 2, 2013

@BlueMax could you please try if this version fixes your problem: http://d-h.st/4Wc

Please capture a log (whether it works or not).

@M66B
Copy link
Owner

M66B commented Dec 2, 2013

Please try this newer version: http://d-h.st/IIe

@BlueMax
Copy link
Author

BlueMax commented Dec 2, 2013

Still no go...

Log:
Startpage.com (probably faked) -> useragentstring.com (unfaked)
browser restart -> useragentstring.com (faked)

I/XPrivacy/XWebView( 1928): android: hooked android.webkit.WebView.constructor/view (4)
I/XPrivacy/XWebView( 1928): android: hooked android.webkit.WebView.getSettings/null (1)
I/XPrivacy/XWebView( 6541): Queue usage data=10027/view/WebView.constructor=true size=3
I/XPrivacy/XWebView( 6541): get 10027/WebView.constructor view=restricted (file)
I/XPrivacy/XWebSettings( 6541): com.android.browser: hooked android.webkit.WebSettings.getDefaultUserAgent/view (1)
I/XPrivacy/XWebSettings( 6541): com.android.browser: hooked com.android.webview.chromium.ContentSettingsAdapter.getUserAgent/view (1)
I/XPrivacy/XWebSettings( 6541): com.android.browser: hooked com.android.webview.chromium.ContentSettingsAdapter.getUserAgentString/view (1)
I/XPrivacy/XWebSettings( 6541): com.android.browser: hooked com.android.webview.chromium.ContentSettingsAdapter.setUserAgent/view (1)
I/XPrivacy/XWebSettings( 6541): com.android.browser: hooked com.android.webview.chromium.ContentSettingsAdapter.setUserAgentString/view (1)
I/XPrivacy/XWebSettings( 6541): Queue usage data=10027/view/setUserAgentString=true size=4
I/XPrivacy/XWebSettings( 6541): get 10027/setUserAgentString view=restricted (file)
I/XPrivacy/XWebSettings( 6541): get 10027/setUserAgentString view=restricted (cached)
I/XPrivacy/XWebSettings( 6541): get 10027/setUserAgentString view=restricted (cached)
I/XPrivacy/XWebView( 6541): Queue usage data=10027/view/WebView.constructor=true size=7
I/XPrivacy/XWebView( 6541): get 10027/WebView.constructor view=restricted (file)
I/XPrivacy/XWebSettings( 6541): Queue usage data=10027/view/setUserAgentString=true size=7
I/XPrivacy/XWebSettings( 6541): get 10027/setUserAgentString view=restricted (file)
I/XPrivacy/XWebSettings( 6541): get 10027/setUserAgentString view=restricted (cached)
I/XPrivacy/XWebView( 6541): Queue usage data=10027/view/WebView.constructor=true size=2
I/XPrivacy/XWebView( 6541): get 10027/WebView.constructor view=restricted (file)
I/XPrivacy/XWebSettings( 6541): Queue usage data=10027/view/setUserAgentString=true size=3
I/XPrivacy/XWebSettings( 6541): get 10027/setUserAgentString view=restricted (file)
I/XPrivacy/XWebSettings( 6541): get 10027/setUserAgentString view=restricted (cached)
I/XPrivacy/XWebSettings( 6541): Queue usage data=10027/view/setUserAgentString=true size=3
I/XPrivacy/XWebSettings( 6541): get 10027/setUserAgentString view=restricted (file)

@M66B
Copy link
Owner

M66B commented Dec 2, 2013

:-(

@BlueMax
Copy link
Author

BlueMax commented Dec 2, 2013

I've traced the string some more. The one in framework.res (strings.xml) seems to be obsolete/outdated:
Mozilla/5.0 (Linux; U; Android %s) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 %sSafari/534.30

The real user agent string is:
Mozilla/5.0 (Linux; Android 4.4; GT-I9300 Build/KRT16S) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/30.0.0.0 Mobile Safari/537.36

Note the 534.30 <> 537.36. Probably legacy string from old engine not needed in new chromium engine anymore.

I just started analyzing the native chromium libraries (20MB monster). It seems it does contain the real user-agent string mask (Mozilla/5.0 (%s) AppleWebKit/%d.%d (KHTML, like Gecko) %s Safari/%d.%d). I've hooked some native methods already leading to nothing (same partial leak) but there must be another bit/flag/switch/setting/whatever that leads to different runtime path (ignoring all setUserAgentString methods).

I will cross reference the string mask to see whats involved.

@M66B
Copy link
Owner

M66B commented Dec 2, 2013

@BlueMax again thanks for looking into this!

I guess the old Android browser is present next to Chromium, explaining two user agent strings.

@BlueMax
Copy link
Author

BlueMax commented Dec 2, 2013

They just started implementing chromium. Several things are still buggy like saving passwords, incognito mode and some dialogs/TextViews (like the one that pops up when you want to change the theme on XDA forum). Logcat shows crash due to "Not implemented yet". :-)

@BlueMax
Copy link
Author

BlueMax commented Dec 4, 2013

Can you try hooking Landroid.webkit.WebView.loadUrl? Its an 'abstract method' that leads to Chromium for KitKat and into Webkit(?) for others. If this works all other user-agent related hooks for all systems should be obsolete.

The class holds two methods. Both are used in Android Browser.
.method public loadUrl(Ljava/lang/String;)V
.method public loadUrl(Ljava/lang/String;Ljava/util/Map;)V

1st argument is URL, 2nd is additional http header parameters including user-agent.

XPrivacy could force-jump 1st method into 2nd one and 2nd method could be hooked with (static) faked 2nd argument holding the user-agent.

If that's not possible, unreasonable or doesn't work we can still tackle Chromium directly by enforcing something like the below runtime path as a last resort. Its probably not as "code evolution resistant" as the loadUrl method above though.

java.lang.Throwable: stack dump
java.lang.Thread.dumpStack(Thread.java:489)
com.android.org.chromium.content.browser.LoadUrlParams.setOverrideUserAgent(LoadUrlParams.java:204)
com.android.org.chromium.android_webview.AwContents.loadUrl(AwContents.java:879)
com.android.webview.chromium.WebViewChromium.loadUrlOnUiThread(WebViewChromium.java:629)
com.android.webview.chromium.WebViewChromium.loadUrl(WebViewChromium.java:524)
android.webkit.WebView.loadUrl(WebView.java:785)
com.android.browser.Tab.loadUrl(Tab.java:1747)

setOverrideUserAgent is integer, held '2' as i logged and is only called when fake was successful. Probably involved in enabling honoring setUserAgentString internally. Somewhere there i expect the target. Haven't looked all too thoroughly into this yet. I hope we don't need it...

M66B pushed a commit that referenced this issue Dec 4, 2013
M66B pushed a commit that referenced this issue Dec 4, 2013
@M66B
Copy link
Owner

M66B commented Dec 4, 2013

Test version with loadUrl hooked: http://d-h.st/j4E

@BlueMax
Copy link
Author

BlueMax commented Dec 4, 2013

Still no go...
The problematic case is when a webpage opens a new tab/view by itself. If a page will be opened explicitly by 'open in new tab' the tab is stable/faked under every condition. But when it's been opened automatically by pressing a link its unfaked. Going into URL bar just pressing -enter- fakes the page for one shot. Reloading the page falls back to unfaked. Reloading also has none of the hooked methods involved (at least none show up with grep XWeb).

These two 'open new tab' scenarios show different runtime paths. The stable one shows 'setUserAgentString' and 'LoadUrl'. The unstable one only has 'setUserAgentString'.

I've disabled hooking 'LoadUrl' and it doesn't show any difference so its probably not of any use.

I will inject more stack dumps and watch variables to analyze the two scenarios. There's still enough left i haven't looked into...
UA_OVERRIDE_FALSE:I
UA_OVERRIDE_INHERIT:I
UA_OVERRIDE_TRUE:I

getUserAgentLocked
nativeUpdateUserAgentLocked
nativeGetDefaultUserAgent
setUserAgentOverride
nativeSetUseDesktopUserAgent
nativeGetUseDesktopUserAgent

@M66B
Copy link
Owner

M66B commented Dec 4, 2013

Thanks for testing.
With your persistence I believe we can fix this problem!

@gdmzyejian
Copy link

In the Android 4.12, MIUIv5, MIUI browser can fake UA, but others do not work.

@gdmzyejian
Copy link

Xprivicy 1.10.37
MIUI V5 (4.12)
Xposed 2.4.1

@BlueMax
Copy link
Author

BlueMax commented Dec 7, 2013

Its a bug in the chromium engine (native code). The chromium internal child tabs do not properly inherit parent properties delivered from Android/Java code. The built-in custom user-agents (Android browser) are affected as well. Most user actions ("open in new tab") result in a new (parent) WebView instance (thus unaffected) compared to clicked links that create new tabs automatically (as child). Unless we can hook native code i'm out of ideas at the moment.

@M66B
Copy link
Owner

M66B commented Dec 7, 2013

@BlueMax thanks for your research!

I guess we have to wait until this Chromium bug is fixed.

@BlueMax
Copy link
Author

BlueMax commented Dec 8, 2013

Just for reference:

  • The primary source (method) for the browser getting the system user-agent from the engine is 'nativeGetDefaultUserAgent'. It feeds all UA variables but hooking involves some risks since the browser compares old/new user-agent in some cases and prevents (if both equally) calling methods involved in overriding UA intentionally (thus native code will never be contacted about an intended change/override resulting in untouched/unfaked UA).
  • loadUrl is somewhat more reliable than setUserAgentString on child tabs. Its the only hook that remains stable on child tabs when an URL is entered manually afterwards (startpage.com->search for 'user agent'->pick useragentstring.com (opens in child tab)->Press enter in URL bar on same address->reload page). 'SetUserAgentString' brakes this behavior if hooked as well (child tab URL reload only stable for one load).
  • There is at least one app ("Codecheck") that doesn't like loadUrl. It loads the page but immediately blanks it out. If there are more side-effects like this 'loadUrl' might be a 'dangerous' candidate (unless there's something wrong with the hook implementation).

@M66B
Copy link
Owner

M66B commented Dec 9, 2013

Thanks again for your research!

  • I only want to hook native/private methods as a last resort, because these hooks can break easily in a future releases
  • For now I think it is good to keep both hooks (setUserAgentString and loadUrl)

@M66B M66B closed this as completed Mar 8, 2014
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants