okhttp/4.9.0 user agent is blocked in potential bot crawling lists

[e-minguez]

Home Assistant Android version:
2021.1.1-full

Android version:
10

Phone model:
OP5T

Home Assistant version:
2021.1.4

Last working Home Assistant release (if known):

Description of problem:
The user agent used by the android app (okhttp/4.9.0) seems to be pretty generic and it is being blocked in some blacklists (see JayBizzle/Crawler-Detect#437)

I guess it would be better to modify the user-agent with something less generic to avoid being blocked.

Traceback (if applicable, to get the logs you may refer to: https://companion.home-assistant.io/docs/troubleshooting/faqs/#android-crash-logs):

n/a

Screenshot of problem:

Additional information:

bunkerized-nginx uses these lists, and it blocks access to home-assistant when trying to log in as:

2021/02/16 12:55:07 [warn] 24951#24951: *35 [lua] access_by_lua(main-lua.conf:166):80: [BLOCK] User-Agent okhttp/4.9.0 is blacklisted, client: 10.0.2.100, server: www.example.com, request: "POST /auth/token HTTP/2.0", host: "www.example.com"

[dshokouhi]

What's wrong with just removing it from blocked list? It is up to every user who uses a block list to ensure it blocks what they want and to modify it for their ow needs. Not to mention there can be other things dependent on the user agent string such as the HA frontend to understand the android app is talking to it. I don't think this is something the app should be fixing.

[e-minguez]

What's wrong with just removing it from blocked list? It is up to every user who uses a block list to ensure it blocks what they want and to modify it for their ow needs. Not to mention there can be other things dependent on the user agent string such as the HA frontend to understand the android app is talking to it. I don't think this is something the app should be fixing.

There is nothing wrong with it but I guess it would be better if instead using a generic user agent, the app uses its own u-a so you can identify it properly (something like home-assistant-android <arch>-<version>)

[dshokouhi]

So when you say its blocked are you seeing the frontend load at all? What parts of the app work and doesn't work? Do sensors update?

[e-minguez]

So when you say its blocked are you seeing the frontend load at all? What parts of the app work and doesn't work? Do sensors update?

Let me elaborate my issue.

I have my home-assistant instance running in a container. To expose it, I'm using bunkerized-nginx, which can block requests based on different methods, including 'bad' user-agents. In order to block those user agents, it uses public lists of known bots and crawlers, such as JayBizzle/Crawler-Detect that includes okhttp

After disabling the user-agent filtering, I can access my home-assistant instance, but I think it would be nice to customize the user-agent in the android application so it can be identified itself instead of a vague 'okhttp'.

I've requested also to remove 'okhttp' from the list but I still think the 'best practice' would be to customize the user-agent.

My two cents.

[anarion80]

Oh boy! I was struggling for days to get HA on Android to connect to my Home Assistant remotely, and in the end it was this issue.
I have OPNsense with Nginx plugin set up on my router and I use that as a reverse proxy for my HA. Everything works fine when I connect remotely from Chrome or any other browser, but HA Android app is unable to authenticate (get token I think). After checking dozens of similar issues everywhere, I found this log on my router:

192.168.2.195 - - [24/Feb/2021:00:00:00 +0100] "POST /api/webhook/49ef04f73e868c444226a3fded8475bd6d440d3a160b78115d4b1606ccaf4a8e HTTP/2.0" 403 44 "-" "okhttp/4.9.0" "-"

in /var/log/nginx/permanentban.access.log

After digging some more, turns out you can't edit the blacklist on OPNsense, but you can disable it. So after enabling "Disable Bot Protection" under my HTTP Server entry in OPNSense/Nginx settings - everything started working as expected.

So while I understand there is a workaround, it would be great it HA app user-agent could be updated to something not present on various block lists.

[sondreen]

I seem to be struggling with the same issue, though I can't solve the issue as I see no mention of user-agent-filtering or bot protection on the Nginx Web UI, also can't find any config-files I could change parameters on. I've posted a thread on the forums and any help would be highly appreciated, thank you.

[e-minguez]

It seems okhttp does indeed support changing the user-agent via interceptors https://square.github.io/okhttp/interceptors/#application-interceptors

[dshokouhi]

So I had tried to test those above changes however I do not have an environment setup to see the headers in the logs nor do they print to logcat. So I could attempt to make changes and submit a PR if you guys are willing to test the debug APK that gets generated. Would you guys be ok with that? Debug APKs are fine to install side by side the production app and can be safely removed once done testing.

[anarion80]

Sure, I can test.

[dshokouhi]

@anarion80 ok the debug APK is ready for testing, you can find it here in the actions tab: https://github.com/home-assistant/android/actions/runs/640306067

Please comment on the linked PR and if possible show the output you are getting :)

[anarion80]

With the debug app the connection process goes through with my Bot Protection enabled and doesn't get banned, the new user-agent is visible:

192.168.2.195 - - [10/Mar/2021:20:12:17 +0100] "POST /auth/login_flow/a755064164e341abaf4b3d17a7a4c72a HTTP/2.0" 200 189 "https://redacted/auth/authorize?response_type=code&client_id=https://home-assistant.io/android&redirect_uri=homeassistant://auth-callback" "Mozilla/5.0 (Linux; Android 11; Nokia 8.3 5G Build/RKQ1.200928.002; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/88.0.4324.181 Mobile Safari/537.36" "-" 192.168.2.195 - - [10/Mar/2021:20:12:17 +0100] "POST /auth/token HTTP/2.0" 200 337 "-" "HomeAssistant/Android LOCAL-1" "-" 192.168.2.195 - - [10/Mar/2021:20:12:29 +0100] "GET /api/discovery_info HTTP/2.0" 200 202 "-" "HomeAssistant/Android LOCAL-1" "-" 192.168.2.195 - - [10/Mar/2021:20:12:29 +0100] "POST /api/mobile_app/registrations HTTP/2.0" 201 128 "-" "HomeAssistant/Android LOCAL-1" "-" 192.168.2.195 - - [10/Mar/2021:20:12:29 +0100] "GET /?external_auth=1 HTTP/2.0" 200 1388 "-" "Mozilla/5.0 (Linux; Android 11; Nokia 8.3 5G Build/RKQ1.200928.002; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/88.0.4324.181 Mobile Safari/537.36" "-"

At the same time, with the same router setting, the original HA app gets banned and cannot find the server.

After logging in with the debug app during Registering Application, there is an error popping up:
Firebase Error: By skipping/ignoring this error you will be unable to receive any push notifications.
Not sure if this is wrong or even related.

[dshokouhi]

The error is safe to ignore as notifications do not work in the debug build. Thank you for confirming the user agent change and that is working for your setup.