Oauth for google redirect_uri error expecting HTTPS but sending HTTP

[gvillafanetapia]

Apache Airflow version:
Version: v2.1.0 (The problem still exists in v2.1.2)
Git Version: .release:2.1.0+304e174674ff6921cb7ed79c0158949b50eff8fe

Apache Airflow Provider versions : apache-airflow-providers-google==4.0.0

Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:28:09Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.13-eks-8df270", GitCommit:"8df2700a72a2598fa3a67c05126fa158fd839620", GitTreeState:"clean", BuildDate:"2021-07-31T01:36:57Z", GoVersion:"go1.15.14", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: AWS
• OS: Debian GNU/Linux 10 (buster)
• Kernel: Linux 5.4.105-48.177.amzn2.x86_64
• Install tools: Helm version.BuildInfo{Version:"v3.4.2", GitCommit:"23dd3af5e19a02d4f4baa5b2f242645a1a3af629", GitTreeState:"clean", GoVersion:"go1.14.13"}

What happened:

On configuring Google Oauth login, when selecting an account to continue with we get a message from Google saying:

The redirect URI in the request, http://****/oauth-authorized/google, does not match the ones authorized for the OAuth client. To update the authorized redirect URIs, visit: https://console.developers.google.com/apis/credentials/oauthclient/****.apps.googleusercontent.com?project=*******

When visiting google cloud console there is no way of configuring a redirect_uri with HTTP (HTTPS is enforced now), and there is no way in Airflow to change the redirect_uri sent in the request.

What you expected to happen:

I expect Airflow to send in the oauth request my configured BASE_URL with HTTPS as the redirect_uri, or to use the redirect_uri configured in the OAUTH_PROVIDERS object variable set in webserver_config.py

How to reproduce it:

Get oauth credentials from google cloud console. Check that it won't let you set a redirect_uri with HTTP.

Install Airflow v2.1.* with the following webserver_config.py:

import os
      from airflow import configuration as conf
      from flask_appbuilder.security.manager import AUTH_OAUTH
      basedir = os.path.abspath(os.path.dirname(__file__))
 
      SQLALCHEMY_DATABASE_URI = conf.get('core', 'SQL_ALCHEMY_CONN')
      CSRF_ENABLED = True
      AUTH_TYPE = AUTH_OAUTH
 
      # registration configs
      AUTH_USER_REGISTRATION = True  # allow users who are not already in the FAB DB
      AUTH_USER_REGISTRATION_ROLE = "Viewer"  # this role will be given in addition to any AUTH_ROLES_MAPPING
 
      GOOGLE_KEY = os.getenv('AIRFLOW_GOOGLE_CLIENT_ID', 'GOOGLE_KEY_NOT_SET')
      GOOGLE_SECRET = os.getenv('AIRFLOW_GOOGLE_CLIENT_SECRET', 'GOOGLE_SECRET_NOT_SET')
 
      # the list of providers which the user can choose from
      OAUTH_PROVIDERS = [{
          'name': 'google',
          'whitelist': ['@*****'],
          'token_key':'access_token',
          'icon':'fa-google',
          'remote_app': {
              'client_id': '*******.apps.googleusercontent.com',
              'client_secret': '*******',
              'api_base_url': 'https://www.googleapis.com/oauth2/v2/',
              'client_kwargs': {
                  'scope': 'email profile'
              },
              'request_token_url': None,
              'access_token_url': 'https://accounts.google.com/o/oauth2/token',
              'authorize_url': 'https://accounts.google.com/o/oauth2/auth'
          }
      }]

and these settings;

AIRFLOW__WEBSERVER__BASE_URL: "https://******" # SOME URI with HTTPS
AIRFLOW__WEBSERVER__AUTHENTICATE: "True"
AIRFLOW__WEBSERVER__AUTH_BACKEND: "airflow.contrib.auth.backends.google_auth"
AIRFLOW__WEBSERVER__RBAC: "True"

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[feluelle]

I can confirm this. Spent already a couple of hours or days trying to figure this out.

But tbh I think the bug is somewhere upstream. My guess is FAB.

I already tried a dozen things. In my case we have Airflow running behind a proxy, but the proxy fix did not solve this specific issue.
To better debug it, I used ksniff as we have Airflow running in K8s and I wanted to investigate the network traffic.
I used the following command to retrieve all http(s) get and post requests:

kubectl sniff <pod> -n airflow -p -f 'tcp dst port 80 or tcp dst port 443 and tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420 or tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x504F5354 or tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x48545450 or tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x3C21444F' -o -

then I also piped this to tshark for better visualisation and I could only find http calls not a single https call being made.

I don't know what is going on there. Appreciate help as well! :)

[juanmanuelcarreira]

Hello!!!
We have exactly the same issue!!!

[jedcunningham]

I believe @feluelle is on the right track with this being a FAB issue.

Found this, which looks like it's the same issue: dpgaspar/Flask-AppBuilder#1666

You all should add any helpful context there. This is most likely a "wont-fix" from Airflow's side.

[potiuk]

I am going to collaborate with @dpgaspar soon (maybe even this week) on fixing some other - related - FAB issues, so I will add that to the list.

[gvillafanetapia]

Thank you for taking notice of this!

In the meantime, I found a workaround that got us up and running: setting AIRFLOW__WEBSERVER__ENABLE_PROXY_FIX to True.

[sensei100]

We solved this by adding @gvillafanetapia's workaround in the airflow.cfg file under the webserver section. This solution also works for Airflow 1.10.15.

[webserver]
enable_proxy_fix = True

[zachliu]

i hereby confirm the workaround works on 2.2.0 🎉

also, evil google's documentation is not clear on this part (probably a 🐛 on google's end):

In the GCP console, under OAuth consent screen, when the Publishing status is In production, we can still put http://localhost:8080/oauth-authorized/google under the Authorized redirect URIs without triggering the red error message Invalid Redirect. However, it doesn't work unless the app is in Testing status.

[potiuk]

Cool. BTW. Maybe you can submit a change to Google about the docs ? They have a great system (developed by the team I worked at in Cracow years ago) that you can visually mark errors on the page with docs and suggest a change. It will not open PR with the change, but i think there are some people (or bots now) looking at the feedback. I think you can find 'provide feedback for this page' or similar link on each page at the bottom :)

[zachliu]

yeah, i knew that system but Google seems to adopt the Facebook way "move fast and break things" so their documentation is always behind 😿 anyway, i already submitted the feedback to Google 🤞

[vparmeland]

I encountered the same issue for my Gitlab OAUTH :/ (My Airflow service is behind an AWS ALB)
Does anyone know a workaround ?

[potiuk]

@nk74 - ask in Flask App Builder forums. Here are docs about authentication in Flask App Builder https://flask-appbuilder.readthedocs.io/en/latest/security.html

[potiuk]

Converting it into discussion - as this is FAB issue not an Airflow one.

[eugenelisevich]

Faced the same issue (http instead of https in redirect url), try setting AIRFLOW__WEBSERVER__ENABLE_PROXY_FIX to True, but still get http, any suggestions?

[potiuk]

Did you ask in FAB as suggested above? This is the best suggestion you can get.

[eugenelisevich]

I missed to add annotations for the ingress in chart
ingress.kubernetes.io/custom-request-headers: x-forwarded-proto:https||x-forwarded-ssl:on||x-forwarded-protocol:ssl ingress.kubernetes.io/ssl-proxy-headers: "X-Forwarded-Proto: https"

[akshay201]

Hi, we are running ariflow version: 2.1.4.
Tried adding AIRFLOW__WEBSERVER__ENABLE_PROXY_FIX to True and also updated airflow.cfg to include the config in webserver section. Also tried changing ing annotations, still it is redirecting to http and not https. Any suggestions would be helpful. Thanks in advance.

[akshay201]

Any information will be helpful. Thanks.
@eugenelisevich @potiuk

[potiuk]

No idea - but there are suggestions up there - it's really a deployment issue, not airflow. And my suggestion is always the same. If you want to deploy airflow in production, make sure that you put authenticatin proxy in fromt of Airflow - this is how you professionally secure applications. Authentication, SSL termination is better done with dedicated proxy rather than with FAB.