3.0.2776 hangs at loading behind nginx proxy

[evanrich]

Ombi build Version:

V 3.0.2776

Update Branch:

Open Beta (Don't see an option)

Media Sever:

Plex

Media Server Version:

Operating System:

Docker

Ombi Applicable Logs (from /logs/ directory or the Admin page):

(Logs go here. Don't remove the ' tags for showing your logs correctly. Please make sure you remove any personal information from the logs)

Problem Description:

I had Ombi v3 working just fine for weeks running in a docker container, auto update was set to on, and over the last two days, I noticed it no longer worked behind nginx. I found the following link https://github.com/tidusjar/Ombi/wiki/Reverse-Proxy-Examples Where it calls out:

 location /ombi {		
     return 301 $scheme://$host/ombi/;		
}
location /ombi/ {
    proxy_pass http://127.0.0.1:5000;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Host $server_name;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Ssl on;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_read_timeout  90;
    proxy_redirect http://127.0.0.1:5000 https://$host;
}
if ($http_referer ~* /ombi/) {
        rewrite ^/dist/(.*) $scheme://$host/ombi/dist/$1 permanent;
}

But this doesn't seem to work. Here is my nginx conf:

server {
        listen      *:443 ssl;
        server_name plexrequest.me;
        access_log /etc/nginx/logs/nginx-access.log;

        ssl_certificate        /etc/nginx/ssl/plexrequest.me.pem;
        ssl_certificate_key    /etc/nginx/ssl/plexrequest.me.key;
        ssl_verify_client      off;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_prefer_server_ciphers on;
        ssl_ciphers ECDH+AESGCM:ECDH+AES256:ECDH+AES128:DHE+AES128:!ADH:!AECDH:!MD5;

        location / {
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-Host $server_name;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Ssl on;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_read_timeout  90;
            proxy_pass http://192.168.1.18:3580;
        }
        if ($http_referer ~* /) {
            rewrite ^/dist/(.*) $scheme://$host/dist/$1 permanent;
        }
    }

When viewed via the url "https://plexrequest.me" on my desktop or mobile device, I just get a "Loading..." page. I did add the http_referer part, which wasn't there originally, but that doesn't seem to work. I modified the referer section, as I'm not running on one site with a sub page like the example. When I view the page loading in developer tools in chrome, it seems to be not able to load main.js and vendor.js.

Reproduction Steps:

Please include any steps to reproduce the issue, this the request that is causing the problem etc.

[tidusjar]

Can you please show what's in the browsers console. Can you also check if it works locally without the proxy.

[evanrich]

Edit, just realized i missed one line...here's updated and still not working config:

 server {
      listen 80;
      return 301 https://$server_name$request_uri;
    }

        server {
        listen      443 ssl;

        server_name plexrequest.me;
        access_log /etc/nginx/logs/nginx-access.log;

        ssl_certificate        /etc/nginx/ssl/plexrequest.me.pem;
        ssl_certificate_key    /etc/nginx/ssl/plexrequest.me.key;
        ssl_verify_client      off;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_prefer_server_ciphers on;
        ssl_ciphers ECDH+AESGCM:ECDH+AES256:ECDH+AES128:DHE+AES128:!ADH:!AECDH:!MD5;

        location / {
            proxy_pass http://192.168.1.18:3580;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-Host $server_name;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Ssl on;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_read_timeout  90;
            proxy_redirect http://192.168.1.18:3580 https://$host;
        }
        if ($http_referer ~* /) {
            rewrite ^/dist/(.*) $scheme://$host/dist/$1 permanent;
        }
    }

[tidusjar]

Please see my above comment, also try in some sort of incognito mode in your browser to make sure nothing is cached.

[louis-lau]

@evanrich You're not using a baseurl so you don't need the rewrite, just delete that. Also delete return 301 https://$server_name$request_uri;. I don't run a nginx proxy myself but I think those aren't needed in this scenario. Try it out! :)

[JimboMonkey1234]

Hit a similar sounding problem myself, took a look in the nginx error.log and found things like:
2018/01/30 20:52:48 [crit] 3928#3928: *165336 open() "/var/lib/nginx/proxy/3/04/0000000043" failed (13: Permission denied) while reading upstream, client: XX.XX.XXX.XXX, server: HOSTNAME, request: "GET /ombi/dist/vendor.js?v=q2CbbVGoKT10XOOP463VR4Z6rIBnV9m7MvbWSzqmpQ4 HTTP/2.0", upstream: "http://[::1]:5000/ombi/dist/vendor.js?v=q2CbbVGoKT10XOOP463VR4Z6rIBnV9m7MvbWSzqmpQ4", host: "HOSTNAME", referrer: "https://HOSTNAME/ombi/requests"

For some reason it was only failing calls from iOS safari.

Anyway, I fixed it by deleting the /var/lib/nginx/proxy directory.

I've been experimenting with changing the nginx user, so that probably explains my problem.

@evanrich, I'd check your nginx error logs.

[evanrich]

@tidusjar

here's the browser+debug

when i do a reload, nothing shows up in nginx error log, but this is in access

172.68.189.125 - - [30/Jan/2018:23:07:03 -0800] "GET / HTTP/1.1" 200 1233 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
172.68.189.125 - - [30/Jan/2018:23:07:06 -0800] "GET / HTTP/1.1" 200 1233 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
172.68.189.125 - - [30/Jan/2018:23:07:12 -0800] "GET / HTTP/1.1" 200 1233 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"

[evanrich]

@JimboMonkey1234 thanks i just now noticed after trying on my android device the following:

bind() to 0.0.0.0:443 failed (98: Address in use)
2018/01/30 23:26:24 [emerg] 434#434: bind() to 0.0.0.0:80 failed (98: Address in use)
2018/01/30 23:26:24 [crit] 285#285: *118 open() "/var/tmp/nginx/proxy/8/01/0000000018" failed (13: Permission denied) while reading upstream, client: 66.249.84.4, server: plexrequest.me, request: "GET /dist/vendor.js?v=d_4NuxOjKEK0pboSL0VarZ08LgNgLHDpenZxeT7voJo HTTP/1.1", upstream: "http://192.168.1.18:3580/dist/vendor.js?v=d_4NuxOjKEK0pboSL0VarZ08LgNgLHDpenZxeT7voJo", host: "plexrequest.me", referrer: "http://plexrequest.me/"
2018/01/30 23:26:25 [crit] 285#285: *114 open() "/var/tmp/nginx/proxy/9/01/0000000019" failed (13: Permission denied) while reading upstream, client: 66.249.84.2, server: plexrequest.me, request: "GET /dist/vendor.css?v=2GCDoRZwuCMsB5hCBgLZvUpd336ZXIhrNwnyCorabRo HTTP/1.1", upstream: "http://192.168.1.18:3580/dist/vendor.css?v=2GCDoRZwuCMsB5hCBgLZvUpd336ZXIhrNwnyCorabRo", host: "plexrequest.me", referrer: "http://plexrequest.me/"
2018/01/30 23:26:25 [crit] 285#285: *121 open() "/var/tmp/nginx/proxy/0/02/0000000020" failed (13: Permission denied) while reading upstream, client: 192.168.2.11, server: plexrequest.me, request: "GET /dist/vendor.js?v=d_4NuxOjKEK0pboSL0VarZ08LgNgLHDpenZxeT7voJo HTTP/1.1", upstream: "http://192.168.1.18:3580/dist/vendor.js?v=d_4NuxOjKEK0pboSL0VarZ08LgNgLHDpenZxeT7voJo", host: "plexrequest.me", referrer: "http://plexrequest.me/"
2018/01/30 23:26:25 [crit] 285#285: *123 open() "/var/tmp/nginx/proxy/1/02/0000000021" failed (13: Permission denied) while reading upstream, client: 66.249.84.2, server: plexrequest.me, request: "GET /dist/main.js?v=CKbJxIji4vnrpGzKW4ELW2IkyJTvPf99mITB-2gDOlU HTTP/1.1", upstream: "http://192.168.1.18:3580/dist/main.js?v=CKbJxIji4vnrpGzKW4ELW2IkyJTvPf99mITB-2gDOlU", host: "plexrequest.me", referrer: "http://plexrequest.me/"
2018/01/30 23:26:24 [emerg] 434#434: bind() to 0.0.0.0:443 failed (98: Address in use)
2018/01/30 23:26:24 [emerg] 434#434: bind() to 0.0.0.0:80 failed (98: Address in use)
2018/01/30 23:26:25 [crit] 285#285: *125 open() "/var/tmp/nginx/proxy/2/02/0000000022" failed (13: Permission denied) while reading upstream, client: 192.168.2.11, server: plexrequest.me, request: "GET /dist/main.js?v=CKbJxIji4vnrpGzKW4ELW2IkyJTvPf99mITB-2gDOlU HTTP/1.1", upstream: "http://192.168.1.18:3580/dist/main.js?v=CKbJxIji4vnrpGzKW4ELW2IkyJTvPf99mITB-2gDOlU", host: "plexrequest.me", referrer: "http://plexrequest.me/"

this was after dicking with cloudflare for a bit, but i checked in /var/tmp/nginx/proxy, and it's empty..tried making the dir 777, restarting nginx docker, still no dice. Strangely enough I can get to the site now via http, but https is still having issues. This was working up until recently, so I'm not sure what's changed....I'm going to try tomorrow to bypass cloudflare, seeing if http works 100%, (though it seems to now through cloudflare), then try generating a self signed cert, testing that locally, then re-issuing cloudflare to see where it breaks. I will also try removing the other proxy blocks i have in nginx to rule them out, although nothing has changed from when it worked to now...but we'll see.

[evanrich]

might have solved it with @JimboMonkey1234's help... the path didn't exist, but i did a combination of the following in the nginx-docker container:

rm-rf /var/tmp/nginx/proxy (symlinked to /var/lib/nginx/tmp/proxy)
chmod -R nobody:nobody /var/lib/nginx (from dockerfile/nginx#4

It's sorta working now, going to try placing cloudflare back in the mix.

edit with cloudflare back in, still having weird issues...this bugs me since it was working without any intervention before... getting new errors now... Will update this when i have more time to look into it this weekend.

[evanrich]

Going to mark this resolved. I changed the nginx user from "nobody" (per nginx-docker) to root, site at least loads now...im tired of messing with this. Since it's just a home instance, and the container still uses docker-user for host filesystem, potential risk is low.

Thanks.