SAML Login Fails With Invalid Response

[newlandk]

ISSUE TYPE

• Bug Report

COMPONENT NAME

• UI
• API

SUMMARY

When logging into AWX with SAML configured the final redirect appears to be to an incorrect port that is of the container and not the web interface.

ENVIRONMENT

• AWX version: 1.0.2.370
• AWX install method: official image
• Ansible version: 2.4.1
• Operating System: Centos 7.6
• Web Browser: Chrome/Safari

STEPS TO REPRODUCE

Configure AWX with a SAML Identity Provider. The SAML portion of the authentication appears to work up to the point where a redirect occurs to an incorrect port and protocol (that appears to be the port and protocol on the docker container) and not that which the web/API is hosted from.

It appears this may be a regression from the transition from social-app-django to social-auth-core and social-auth-app-django.

There has been some activity here that seems related: #119 but it looks like there have been changes in the authentication back-end since it was closed and the issue covers multiple challenges.

EXPECTED RESULTS

Redirected to the URL that includes the hostname and port of the web/API logged in.

ACTUAL RESULTS

An error after successfully logging into the IDp.

Authentication failed: SAML login failed: ['invalid_response'] (The response was received at http://awx.test.com:8052/sso/complete/saml/ instead of https://awx.test.com/sso/complete/saml/).

[newlandk]

I'm not too familiar with social-core or Python, but I've done my best to understand it.
It looks like social-app-django may be using the container to determine port and protocol since it is the source of the SAML request

• request_port
• request_is_secure

[newlandk]

It turns out the issue faced was proxy related and I had to modify the settings.py configuration to allow the use of the X-Forwarded-Port. After this django provides the correct port to social-auth for the complete URL.

Good reference if you run into the same issue:
Tower Proxy Documentation

• the doc has details regarding the configuration you need to set for AWX to use additional headers provided by the proxy server.
• You may need to configure your proxy server to provide the port as it is was provided by default and simply being https does not assume port 443
• Using the official docker images I mounted the settings.py as a volume with the following lines added
  • USE_X_FORWARDED_PORT = True
  • USE_X_FORWARDED_HOST = True

This issue was especially confusing since the there are multiple places in the UI that state the full URL of the server including protocol and social-auth determines these values from django instead.

[BenJuan26]

Can you explain a little about how you mounted settings.py as a volume? I tried changing awx/installer/roles/image_build/files/settings.py but even after rebuilding all containers, my version of the settings was not located at /etc/tower/settings.py inside the container.

I'm getting the same error you've provided here.

[newlandk]

@BenJuan26 we just mount a modified settings.py outside the container and mount it in the container as a volume for the time being.

When you run your docker-compose add arguments similar to this for awx_web:

services:
  awx_web:
    volumes:
      - /src/settings.py:/etc/awx/settings.py

More information here on volumes/mounts: https://docs.docker.com/storage/volumes/#start-a-container-with-a-volume

[sudomateo]

So the way I fixed this to work correctly with SAML was by building the container images locally and pushing them to the remote AWX server. This alleviates the need to mount a volume.

First I commented out the following lines in installer/inventory:

#dockerhub_base=ansible
#dockerhub_version=latest

This allows you to build the container image locally instead of pulling from ansible's dockerhub images.

Next I added the following lines to installer/roles/image_build/files/settings.py

USE_X_FORWARDED_PORT = True
USE_X_FORWARDED_HOST = True

This is what gets injected into the container images for the web and task containers.

Then I configured the nginx reverse proxy on the AWX remote host. My config looks like this:

server {
    listen 80;
    server_name awx.example.com;

    location / {
        return 301 https://awx.example.com$request_uri;
    }
}
server {  
    listen 443 ssl;
    server_name awx.example.com;

    ssl_certificate cert.pem;
    ssl_certificate_key key.pem;
    ssl_protocols TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_ciphers "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA"; 
    ssl_ecdh_curve secp384r1;
    ssl_session_cache shared:SSL:10m;
    ssl_session_tickets off;
    ssl_stapling on;
    ssl_stapling_verify on;

    add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
    add_header X-Content-Type-Options nosniff;


    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port 443;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

The important line here is proxy_set_header X-Forwarded-Port 443;. This forwards the correct port along for AWX to use with the USE_X_FORWARDED_HOST = True setting above.

Lastly, I added the HTTP_X_FORWARDED_FOR header in Settings > System > Remote Host Headers

I used https://gist.github.com/defionscode/fc21488e44d73cdd919f81ee1b43e204 as a reference for how my SAML config should look.

[jeunii]

Hello @sudomateo
I encountered a similar issue. Actually I have the exact same issue. My AWX setup is deployed in a kubernetes cluster. And rather than nginx as my Ingress, I have HA Proxy.
Would you by any chance know what would be the equivalent haproxy config ?

[newlandk]

I believe you would have configuration options like this under your frontend to overcome the challenges we faced noted above.
http-request set-header X-Forwarded-Proto https if https
http-request set-header X-Forwarded-Proto http if !https
http-request set-header X-Forwarded-Port 443
option forwardfor

Good luck!

[sudomateo]

@jeunii I'm sorry I do not have an exact HAProxy configuration handy but what @newlandk commented is a good start. As long as the headers are properly set it should be fine.

[svrraja]

How to fix the issue if its running in kubernetes, How to configure the NGINX reverse proxy

[naren-ambati]

@sudomateo I'm facing the same issue, But the changes I did for enabling tls is completely different of what you used in nginx.conf.

#user awx;

worker_processes  1;

pid        /tmp/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log /dev/stdout main;

    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }

    sendfile        on;
    #tcp_nopush     on;
    #gzip  on;

    upstream uwsgi {
        server 127.0.0.1:8050;
        }

    upstream daphne {
        server 127.0.0.1:8051;
    }

    
    server {
                listen 8052 default_server;
        
        # If you have a domain name, this is where to add it
        server_name awx.project.qa;
        keepalive_timeout 65;
        ssl_certificate           /etc/nginx/certs/awx.project.qa.chain.pem;
        ssl_certificate_key       /etc/nginx/certs/awx.project.qa.key;
        ssl on;
        ssl_session_cache  builtin:1000  shared:SSL:10m;
        ssl_protocols TLSv1.2;
        ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
        ssl_prefer_server_ciphers on;
        access_log            /var/log/nginx/awx.access.log;
        ssl_session_tickets off;
        ssl_stapling on;
        ssl_stapling_verify on;

        # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
        add_header Strict-Transport-Security max-age=15768000;
        add_header Content-Security-Policy "default-src 'self'; connect-src 'self' ws: wss:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' cdn.pendo.io; img-src 'self' data:; report-uri /csp-violation/";
        add_header X-Content-Security-Policy "default-src 'self'; connect-src 'self' ws: wss:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' cdn.pendo.io; img-src 'self' data:; report-uri /csp-violation/";

        # Protect against click-jacking https://www.owasp.org/index.php/Testing_for_Clickjacking_(OTG-CLIENT-009)
        add_header X-Frame-Options "DENY";

        location /nginx_status {
          stub_status on;
          access_log off;
          allow 127.0.0.1;
          deny all;
        }

        location /static/ {
            alias /var/lib/awx/public/static/;
        }

        location /favicon.ico { alias /var/lib/awx/public/static/favicon.ico; }

        location /websocket {
            # Pass request to the upstream alias
            proxy_pass http://daphne;
            # Require http version 1.1 to allow for upgrade requests
            proxy_http_version 1.1;
            # We want proxy_buffering off for proxying to websockets.
            proxy_buffering off;
            # http://en.wikipedia.org/wiki/X-Forwarded-For
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            # enable this if you use HTTPS:
            proxy_set_header X-Forwarded-Proto https;
            # pass the Host: header from the client for the sake of redirects
            proxy_set_header Host $http_host;
            # We've set the Host header, so we don't need Nginx to muddle
            # about with redirects
            proxy_redirect off;
            # Depending on the request value, set the Upgrade and
            # connection headers
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
        }

        location / {
            # Add trailing / if missing
            rewrite ^(.*)$http_host(.*[^/])$ $1$http_host$2/ permanent;
            uwsgi_read_timeout 120s;
            uwsgi_pass uwsgi;
            include /etc/nginx/uwsgi_params;

      }
    }
}

I'm trying to make use of your nginx configuration, but my awx_web is not working and I don't see any errors. Do you have any turn around so that I can use your configuration in the above. Any help is appriciated, Thanks.

[loitho]

I'm putting my reply here too.
This applies if you're running the AWX_WEB with docker-compose with the default port configuration with SSL

If the Nginx that forward its requests to the Django server (the uwsgi_pass uwsgi;) isn't running on a standard port (like here, it's running in the 8053) you can either :

• Change the configuration so that Nginx listens on the 443 port (and modify the docker compose accordingly) That's the "easy" way but I tried to understand how to make it work with the default configuration.

• Or you can leave the default port configuration, and just modify the nginx.conf that's in the "docker_compose_dir" option
  Simply add the following line to the "location /" part :

uwsgi_param HTTP_X_FORWARDED_PORT 443;

And off you go. The error :

The response was received at https://XXXXX:8053/sso/complete/saml/ instead of https://XXXXX/sso/complete/saml/

Should be gone.
I'm attaching my nginx.conf so that you can check it out (it's the AWS 9.0.1 default file, just with the line addition)

#user awx;

worker_processes  1;

pid        /tmp/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    server_tokens off;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log /dev/stdout main;

    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }

    sendfile        on;
    #tcp_nopush     on;
    #gzip  on;

    upstream uwsgi {
        server 127.0.0.1:8050;
        }

    upstream daphne {
        server 127.0.0.1:8051;
    }

        server {
        listen 8052 default_server;
        server_name _;

        # Redirect all HTTP links to the matching HTTPS page
        return 301 https://$host$request_uri;
    }

    server {
#        listen 443 ssl;
        listen 8053 ssl;

        ssl_certificate /etc/nginx/awxweb.pem;
        ssl_certificate_key /etc/nginx/awxweb.pem;

        # If you have a domain name, this is where to add it
        server_name _;
        keepalive_timeout 65;

        # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
        add_header Strict-Transport-Security max-age=15768000;
        add_header Content-Security-Policy "default-src 'self'; connect-src 'self' ws: wss:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' *.pendo.io; img-src 'self' *.pendo.io data:; report-uri /csp-violation/";
        add_header X-Content-Security-Policy "default-src 'self'; connect-src 'self' ws: wss:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' *.pendo.io; img-src 'self' *.pendo.io data:; report-uri /csp-violation/";

        # Protect against click-jacking https://www.owasp.org/index.php/Testing_for_Clickjacking_(OTG-CLIENT-009)
        add_header X-Frame-Options "DENY";

        location /nginx_status {
          stub_status on;
          access_log off;
          allow 127.0.0.1;
          deny all;
        }

        location /static/ {
            alias /var/lib/awx/public/static/;
        }

        location /favicon.ico { alias /var/lib/awx/public/static/favicon.ico; }

        location /websocket {
            # Pass request to the upstream alias
            proxy_pass http://daphne;
            # Require http version 1.1 to allow for upgrade requests
            proxy_http_version 1.1;
            # We want proxy_buffering off for proxying to websockets.
            proxy_buffering off;
            # http://en.wikipedia.org/wiki/X-Forwarded-For
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            # enable this if you use HTTPS:
            proxy_set_header X-Forwarded-Proto https;
            # pass the Host: header from the client for the sake of redirects
            proxy_set_header Host $http_host;
            # We've set the Host header, so we don't need Nginx to muddle
            # about with redirects
            proxy_redirect off;
            # Depending on the request value, set the Upgrade and
            # connection headers
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
        }

        location / {
            # Add trailing / if missing
            rewrite ^(.*)$http_host(.*[^/])$ $1$http_host$2/ permanent;
            uwsgi_read_timeout 120s;
            uwsgi_pass uwsgi;
            include /etc/nginx/uwsgi_params;

            proxy_set_header X-Forwarded-Port 443;
           # that's the important bit
            uwsgi_param HTTP_X_FORWARDED_PORT 443;
        }
    }
}

I haven't seen much ADFS SAML configuration available and since I had quite some trouble setting it up, I'm also adding my ADFS configuration for "SAML ENABLED IDENTITY PROVIDERS"

{
 "saml_ms_adfs": {
  "attr_last_name": "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname",
  "entity_id": "http://signin.server.com/adfs/services/trust",
  "x509cert": "<redacted>",
  "url": "https://signin.server.com/adfs/ls/",
  "attr_username": "http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname",
  "attr_email": "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress",
  "attr_first_name": "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname",
  "attr_user_permanent_id": "http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname"
 }
}

Big thanks to this thread : pallets/werkzeug#1465 (comment) for the enlightenment

[ryanpetrello]

For those following along in this thread, @loitho has contributed a patch which should alleviate this issue (which I'm merging shortly):

#5577

[peacengell]

Hello All,

I found a fix, tested and working. Credit goes to [ https://medium.com/@_jonas/traefik-kubernetes-ingress-and-x-forwarded-headers-82194d319b0e ] This kind man, thank you.

If you using k3s or Kubernetes and Traefik, you can edit this file

Traefix is setup using Helm Chart.
The file path is /var/lib/rancher/k3s/server/manifests/traefik.yaml

Edit it and add the three line with #< with it.

valuesContent: |-
    podAnnotations:
      prometheus.io/port: "8082"
      prometheus.io/scrape: "true"
    providers:
      kubernetesIngress:
        publishedService:
          enabled: true
    additionalArguments: #<
      - "--entryPoints.web.proxyProtocol.insecure" #<
      - "--entryPoints.web.forwardedHeaders.insecure" #<

Helm will redeploy your traefix pod, once restarted give it a 10 min.

Then check.