Issues with ELK behind Google Auth Proxy (double reverse proxies)

[matthewsaeedsc]

Hi,

Thank you very much for providing the Google Auth Proxy code here on github. I have things working pretty well for simple services, but it looks like things get a lot more complicated when you are trying to place a Google Auth Proxy in front of a service that already functions as a Reverse Proxy.

In my example, I created a new Elasticsearch, Fluentd and Kibana stack (EFK). This is very similar to ELK for those that do not know. Well, EFK actually uses Nginx as a reverse proxy so you can visit myexample.com via port 80 and get content back from Elasticsearch which runs on 9200.

Does anyone have experience adding a Google Auth Proxy in front of another service like ELK/EFK that already uses nginx as a reverse proxy? I'm not sure of the exact wording, but this sounds like "double reverse" proxying to me? What is the recommended way ahead for this? I can get the Auth Proxy to function (with SSL), however when the Kibana dashboard loads after successful oAuth you are required to click "load unsafe scripts" in order to get real EFK content....

I'd like to have the Google Auth Proxy configured with HTTPS and my EFK stack configured with HTTP if that makes sense.

Any help would be appreciated, thanks!

Matthew

[LukeHandle]

How about [nginx] > [GAP] > [Kibana]? Nginx will then be the SSL termination and that should fix your unsafe script error? Each reverse proxy doesn't really care of you send them to another proxy, it just adds more jumps and processing for each connection.

[ploxiln]

There have been a few options recently added to google-auth-proxy to make its proxying more flexible, but the way to do whatever you want (without re-implementing nginx in google-auth-proxy) is NGINX -> GAP -> NGINX -> stuff. That's a single NGINX listening on (at least) two ports - one for the initial client connections, and another for google-auth-proxy to use as its upstream.

The first NGINX listen port can terminate ssl, and proxy some un-authed requests directly to various upstreams (or serve files directly) based on various rules nginx supports, and proxy all other requests to google-auth-proxy.

The second NGINX listen port receives authed requests from google-auth-proxy, and can proxy them to various upstreams (or serve files directly) based on various rules nginx supports.

[LukeHandle]

The question is, what is nginx actually doing for you here? If it is just so you don't have 2 processes listening on :80 and you can have a nice host just follow the implementation in the Readme.md and that should be fine.

(Put nginx at the front and then have GAP and finally whatever backend!)

[matthewsaeedsc]

Hi everyone, thanks so much for the multiple responses! Really appreciate it.

The standard Elasticsearch + Fluentd + Kibana (EFK) stack does not offer Encryption or authorization, so that is why the Google Auth Proxy makes a perfect fit to place in front of EFK. If you start from scratch with a very basic design it would look something like this (Copied from Digital Ocean's guide: https://www.digitalocean.com/community/tutorials/elasticsearch-fluentd-and-kibana-open-source-log-search-and-visualization so basically adding https://github.com/bitly/google_auth_proxy in front of this EFK tutorial as a means of authorization, authentication and encryption with SSL ):

General Flow:
User on internet --> types https://efk.example.com -> Google Auth Proxy to ensure they are an authorized user --> Proxy redirects user to Kibana --> Kibana serves data from localhost9200 (elasticsearch). However most EFK/ELK installs already rely on Nginx Reverse proxy to gather Elasticsearch data from localhost 9200. So effectively the Reverse proxy is being placed in front of another reverse proxy. Also: ES, GAP, Kibana, Fluentd and nginx proxy are all on the same host.

Google Auth Config:

include_recipe "google_auth_proxy"
google_auth_proxy_install "my_app" do
client_id "00000000000000"
client_secret "00000000000000"
google_apps_domains ["example.org"]
cookie_domain "efk.example.org"
redirect_url "https://example.org/oauth2/callback"
listen_address "127.0.0.1:4180"
upstreams ["http://127.0.0.1:8081/"]
end

Nginx Configs:
Here is an example nginx.config to illustrate my point:

server {
listen 443 default ssl;
server_name efk.example.org;
ssl_certificate ;
ssl_certificate_key ;

access_log /var/log/nginx/kibana.log;
error_log /var/log/nginx/website.com.error.log;
location / {
proxy_pass http://127.0.0.1:4180;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Scheme $scheme;
proxy_connect_timeout 1;
proxy_send_timeout 30;
proxy_read_timeout 30;
}
}

server {

listen 8081;
server_name efk.example.org;

access_log /var/log/nginx/kibana.log;
error_log /var/log/nginx/website.com.error.log debug;

location / {
root /usr/share/nginx/kibana3;
index index.html index.htm;
}

location ~ ^/aliases$ {
proxy_pass http://localhost:9200;
proxy_read_timeout 90;
}
location ~ ^/./_aliases$ {
proxy_pass http://localhost:9200;
proxy_read_timeout 90;
}
location ~ ^/nodes$ {
proxy_pass http://localhost:9200;
proxy_read_timeout 90;
}
location ~ ^/./_search$ {
proxy_pass http://localhost:9200;
proxy_read_timeout 90;
}
location ~ ^/.*/_mapping {
proxy_pass http://localhost:9200;
proxy_read_timeout 90;
}

Password protected end points

location ~ ^/kibana-int/dashboard/.$ {
proxy_pass http://localhost:9200;
proxy_read_timeout 90;
limit_except GET {
proxy_pass http://localhost:9200;
auth_basic "Restricted";
auth_basic_user_file /etc/nginx/conf.d/kibana.myhost.org.htpasswd;
}
}
location ~ ^/kibana-int/temp.$ {
proxy_pass http://localhost:9200;
proxy_read_timeout 90;
limit_except GET {
proxy_pass http://localhost:9200;
auth_basic "Restricted";
auth_basic_user_file /etc/nginx/conf.d/kibana.myhost.org.htpasswd;
}
}
}

[matthewsaeedsc]

Hm any idea?

"Double Reverse SSL Proxy with Nginx"... impossible? Or just insanely frustrating...

[kcampos]

I just went through this exact debug process and got it working. Here's how:

1. Incoming 443 Nginx vhost -> proxy_pass ALL traffic to GAP
2. Configure GAP upstream to another non-ssl nginx vhost (port 8080 for example)
3. It's on the 8080 vhost where you'd do all your ELK URL filtering as usual and proxy to elasticsearch.

This works great for me. If you try to do the URL filtering before GAP you'll run into all sorts of problems.

[jehiah]

thanks for the clear description on how to set this up @kcampos