Add custom header support

[ericmedina024]

Summary

Adds support for adding a list of headers to add to requests made to a HA server. I tried my best to find all the places the headers need to be added, but I'm sure I missed some. Please let me know any additional places I need to address.

This pull request is a work in progress. The main functionality works, but the UI for adding/removing headers is pretty wonky functionally still.

Screenshots

Link to pull request in Documentation repository

Documentation: home-assistant/companion.home-assistant#
None yet

Any other notes

Thanks to @Meister1977 for his PR #3510 as it helped me figure out where to start digging. I saw his pull request was tagged help wanted, but our implementations ended up being so different I thought it made more sense to make my own PR.

I have little to no experience with Android dev and Kotlin, so please feel free to point out where I can improve the code!

[home-assistant]

Hi @ericmedina024

It seems you haven't yet signed a CLA. Please do so here.

Once you do that we will be able to review and accept this pull request.

Thanks!

[marazmarci]

Please note that it's only possible to add custom headers to the OkHttpClient's requests, but not to the WebView's requests!

For an explanation, see my comment on #2650.

[ericmedina024]

Please note that it's only possible to add custom headers to the OkHttpClient's requests, but not to the WebView's requests!

For an explanation, see my comment on #2650.

This is definitely a big limitation. However, for Cloudflare Zero Trust specifically it is not that big of an issue as I can add the headers on the first request and from that point on Cloudflare sends an auth cookie which will persist for future WebView requests. That said, I think it still may be possible. HA already overrides shouldOverrideUrlLoading here, and the WebResourceRequest arg has a headers field which is mutable. I haven't tested yet, but I'm thinking (hoping) it's possible to just add the headers there.

[ericmedina024]

I haven't tested yet, but I'm thinking (hoping) it's possible to just add the headers there.

It seems this is not possible. I modified my code to add the headers to the WebResourceRequest, then set up burp suite as my emulator's proxy and watched the traffic. Unfortunately, resource requests still did not have the custom headers. That said, they do still succeed as the cloudflare cookies have already been set at this point. Apparently, if you return null any modifications made to the WebResourceRequest are ignored. With this limitation in mind, adding full custom header support is likely either impossible or too hacky to be worth it.

What I can do is take this PR in a more Cloudflare Zero Trust specific direction. Basically, I'd remove the "custom header" wording and replace it with options for the service account id and secret. Since we can ensure that the headers are present the first time the WebView loads, this would work well for that. @jpelgrom I see you were active in this thread - is there interest in merging this if it's Cloudflare Zero Trust specific? I'm not sure what the HA stance might be on such a company specific feature.

Another alternative to consider is just adding these limitations to the documentation. I could specify in the UI and in the docs PR that these are only guaranteed to be sent on the FIRST request but may not be sent on future requests.

[marazmarci]

HA already overrides shouldOverrideUrlLoading here, and the WebResourceRequest arg has a headers field which is mutable. I haven't tested yet, but I'm thinking (hoping) it's possible to just add the headers there.

Here's the key part from my comment on #2650 that I linked above:

Technical details:
See WebViewClient::shouldInterceptRequest. If you want to modify the WebView's HTTP requests, you have to make the HTTP request yourself from the application code, with eg. using an OkHttpClient, but the WebViewClient API won't give you the request body, which makes it practically useless for POST requests, because those almost always have a body. Not to mention WebSocket requests, as they're 100% not interceptable, and there's currently no Android API for them.

What this means, is that WebView practically doesn't support setting custom headers. However, you can set the User-Agent and the cookies, that's all.

And also, mTLS client certificates can be used, which is a better (more secure) alternative for Service Tokens. The HA Android app already supports client certificates, but it's a bit complicated to set it up. (and Cloudflare's free plan includes support for client certificates)

[ericmedina024]

and Cloudflare's free plan includes support for client certificates

do you have a link to the docs on this? I was looking into it and it seems it is only available on the enterprise plan for zero trust. You can add a firewall rule for it on the free plan, but that won't help if you want the website available with other access configurations as well.

[jpelgrom]

@jpelgrom I see you were active in this thread - is there interest in merging this if it's Cloudflare Zero Trust specific? I'm not sure what the HA stance might be on such a company specific feature.

I might be a frequent contributor but my opinion isn't "the HA stance" (if there even is one).

Personally speaking: to merge something like this you'd need to meet expectations and it should be maintainable.

• Saying it's custom headers for all requests -> incorrect because it doesn't handle requests in the webview (which may be required for it to work).
• Saying it's Cloudflare Zero Trust support -> we discovered the combination of headers + cookies appears to work right now, most of the time, but will it continue to work in the future (your config changes, Cloudflare changes how the headers + cookies interact, what about cookies and changes outside a browser context)? And what about authentication layer x/y/z which use a similar setup with headers + cookies, should we add presets for those?

In the end, I think the fact that this is a hack that ended up working but also the very specific requirements make this hard to maintain and accept. Systems like these simply won't work well with Home Assistant's current auth system + native apps, which isn't something for the app to change.

Like previously mentioned, client certificates are a good alternative and also available for free with Cloudflare :) Documentation basic or documentation zero trust or for basic go to your domain > SSL/TLS > Client Certificates, there are two buttons there to create a certificate and create a rule which is all you need.

[ericmedina024]

Thanks for the reply. I agree that it is not worth moving forward with this feature.

Like previously mentioned, client certificates are a good alternative and also available for free with Cloudflare :) Documentation basic or documentation zero trust or for basic go to your domain > SSL/TLS > Client Certificates, there are two buttons there to create a certificate and create a rule which is all you need.

This is actually a different offering than the Zero Trust mTLS stuff. It allows most of the same behavior, but the Zero Trust version is different as you can configure it more flexibly as part of your access policies instead of using a WAF rule. With a WAF rule, you are forced to either deny or allow the traffic and can't offer an alternative auth scheme when the mTLS cert is missing. That said, you guys gave some great alternatives here so I ended up coming up with a scheme that works for me. Here is what I ended up configured in case it is useful to anyone else.

I have three subdomains (which are all just zero trust tunnels to my HA instance) set up as follows:

• ha.ericmedina024.com

This is what I would use if I wanted to access my HA instance from a device I don't own. It is configured via access policies to require access code auth for all requests.

• ha-mtls.ericmedina024.com

This is what I use for the app as I have the mTLS cert installed on my phone (under "VPN & app user cert" in settings). It is configured via a WAF rule to block any traffic that doesn't present a valid mTLS cert.

• ha-ga.ericmedina024.com

This is what I use to allow the Google Assistant integration to work. In my google action, I have the fulfillment URL set to https://ha-ga.ericmedina024.com/api/google_assistant?key=xyz where xyz is a secure "password". I then have a WAF rule set up to ensure that the only accessible path on this subdomain is /api/google_assistant and that the key query parameter is set correctly.

[jeleniain]

* ha-mtls.ericmedina024.com

This is what I use for the app as I have the mTLS cert installed on my phone (under "VPN & app user cert" in settings). It is configured via a WAF rule to block any traffic that doesn't present a valid mTLS cert.

Thanks a bunch for the comment!! I was lurking in these threads for a while trying to find a solution. I am currently a bit stuck on the Cloudflare side and was hoping you could provide some more details on how to configure this. I tried around in Cloudflare and tried to use the Cloudflare documentation, chatgpt and youtube but am stuck :)

Any help would make my day.
Thanks!

Edit:
I think I am bit closer to a solution, maybe it helps someone or if you can see if I made a mistake:

1. Within Cloudflare > click on your domain > SSL/ TLS > Client Certificates > Create Certificate > Here I used the standard settings (RSA Key and 10 Years validity)
2. Create mTLS Rules which brings you to the WAF settings > Custom rules > Create rule
3. Choose a fancy name > Field: Hostname > Operator: is in > Value: mysubdomain.domain.com > add a 2nd line of rules > Field: Client Certificate Verified > equals not. -> The expression looks like this: (http.host in {"mysubdomain.domain.com"} and not cf.tls_client_auth.cert_verified) > Then take action: Block > Safe

So far this seems to block everything, now I just need to figure if it grants access once the certificate is deployed :)

[ericmedina024]

* ha-mtls.ericmedina024.com

This is what I use for the app as I have the mTLS cert installed on my phone (under "VPN & app user cert" in settings). It is configured via a WAF rule to block any traffic that doesn't present a valid mTLS cert.

Thanks a bunch for the comment!! I was lurking in these threads for a while trying to find a solution. I am currently a bit stuck on the Cloudflare side and was hoping you could provide some more details on how to configure this. I tried around in Cloudflare and tried to use the Cloudflare documentation, chatgpt and youtube but am stuck :)

Any help would make my day. Thanks!

Edit: I think I am bit closer to a solution, maybe it helps someone or if you can see if I made a mistake:

1. Within Cloudflare > click on your domain > SSL/ TLS > Client Certificates > Create Certificate > Here I used the standard settings (RSA Key and 10 Years validity)
2. Create mTLS Rules which brings you to the WAF settings > Custom rules > Create rule
3. Choose a fancy name > Field: Hostname > Operator: is in > Value: mysubdomain.domain.com > add a 2nd line of rules > Field: Client Certificate Verified > equals not. -> The expression looks like this: (http.host in {"mysubdomain.domain.com"} and not cf.tls_client_auth.cert_verified) > Then take action: Block > Safe

So far this seems to block everything, now I just need to figure if it grants access once the certificate is deployed :)

Sounds like you got it. That is how I set it up as well. Here's a picture of the rule I set up in case it is helpful:

[jeleniain]

Thanks! The only difference for my setting is that I used the "is in" operator, have not looked up the difference just yet :)
It took me a bit to install the cert on my phone so for future reference:
Once I had the certificate and key downloaded/ saved from CF as a .pem file, I installed OpenSSL, navigated to the location of the 2 files and combined them in a .p12 file with:
openssl pkcs12 -export -out outfilepassword.p12 -inkey keyfile.pem -in certfile.pem
However, when I try to install it on my Android phone, I get prompted over and over to insert the credentials for the certificate. I tried both setting credentials and without any credentials, neither worked, just the endless password prompt.

I than installed the certificate on my Windows machine, and then exported it as a .pfx file.
There is probably better / easier ways to do this but once I changed the format, I could install it on the phone.

Next, it turns out the Firefox Android app can not use those certificates (me coming down from my Firefox is here to save the day horse..) however Chrome can. All I had to do is to wipe all chrome cookies (no clue how to find a specific one on the Android app (sometimes its just...) and once that was done I could access the domain.

The HA app also prompted me to use the cert and for now everything seems to be working fine.

Very nice!!!
Thank you all :)

[kingamajick]

• ha-mtls.ericmedina024.com

This is what I use for the app as I have the mTLS cert installed on my phone (under "VPN & app user cert" in settings). It is configured via a WAF rule to block any traffic that doesn't present a valid mTLS cert.

Hi, thanks for putting this together, looks like it could be an acceptable work around for me at least. I had one question, I assume the ha-mtls domain. How did you disable authentication for this domain? The cloudflared zero trust doesn't have an option to run a endpoint without auth as far as I can see?

[jeleniain]

@kingamajick you don't have to create an access application for the tunnel. I solely have the tunnel configured and than the firewall rule.

[kingamajick]

@jeleniain thanks, removed that and it all works as expected :D

[jeleniain]

Nice! I also had some troubles installing the certificate on my Android phone. Turned out to be a format issue of the certificate where it caused an endless loop when my phone tried to apply it.
Let me know if you have troubles with that. otherwise I have been using this solution for about a month now and it works flawless. Super nice solution and I feel so much better to have my HA behind 2fa and a certificate :)

[some-guy-23]

@jeleniain / @ericmedina024 / @kingamajick trying to walk through this now and running into issue getting the Cloudflare mTLS certificate into the correct format to install on Android. Any step-by-step instruction on how you achieved this is greatly appreciated!

Edit / Update - I was able to run the following OpenSSL command on the .pem certificate file, and private key (.key) file from Cloudflare -

openssl pkcs12 -export -out certificate.pfx -inkey private.key -in certificate.pem

Then I could import the .pfx file with 'Install a Certificate' -> 'VPN & app user certificate' in Android, prompts for a name. Then at that point was able to go back to HA mobile app, and once remote it will prompt for that certificate file.

[jeleniain]

@some-guy-23 happy you resolved it. I cant remember how I converted it but also concerted it to a .pfx file to be able to install it. Only issue I ran into after is that for some reason Firefox could not handle the mtls cert so I had to use chrome for it instead. No clue why that is but seems to be a FF limitation.