XHR request for starting Job returns internal IP as URL instead of configured FQDN

[Sushi1324]

Peering Manager version

1.7.3

Python version

3.9

Steps to Reproduce

I have peering manager setup behind an Nginx reverse proxy (on a secondary node) that I use to add TLS. These are the proxy options I have configured in my nginx configuration in case they are relevant:

    location / {

        proxy_pass_request_headers on;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://100.100.118.85;
        proxy_redirect default;
    }

Essentially all of the site works as expected, except when trying to make any API calls that require pulling the results of a started background job. For example, after configuring a router and adding the correct NAPALM configuration, trying to use either the "Ping" or "Poll Sessions" buttons fails. When accessing the site directly without going through the proxy, these buttons work fine.

Expected Behavior

After clicking "Ping", it should call this chunk of code in the HTML page generated by the view:

$("#id_router_ping").click(function () {
    $.ajax({
      method: "get",
      url: "/api/peering/routers/1/test-napalm-connection/",
    }).done(function(r) {
      PeeringManager.setWorkingButton($('#id_router_ping'));
      PeeringManager.pollJobResult(r, pingDoneHandler, pingFailHandler);
    });
  });

The request to "/api/peering/routers/1/test-napalm-connection/" successfully passes through the reverse proxy and returns. The Javascript Object that gets passed into the Ajax handler "r" has a value "url" that should be set to "/api/extras/job-results/{{ job-id }}/". That url then gets used to monitor the status of the Job to determine if it finished successfully.

Observed Behavior

Rather than the request to "/api/peering/routers/1/test-napalm-connection/" returning an object with the url "/api/extras/job-results/{{ job-id }}/", it returns with the url "http://100.100.118.85/api/extras/job-results/{{ job-id }}/". Since I'm connecting via reverse proxy, the browser sees "http" and blocks the traffic, plus it fails to connect to the internal IP anyways when specifically allowed.

It appears that the Peering Manager API tries to resolve an absolute URL when returning Job Result objects, rather than a relative URL like most of the site is designed around.

This might be something I misconfigured, but I don't see anywhere in the database or config where it could be pulling the internal address from, so I am guess Django is resolving that from the interface it is listening on. I tried finding in the code where the url was being generated, but I am not a developer and could use some help for what I imagine is a fairly simple fix if I understand what is happening correctly.

[gmazoyer]

My config is similar but I don't face this specific issue.

Relevant config snippets are:

    location /static/ {
        alias /opt/peering-manager/static/;
    }

    # reverse proxy
    location / {
        proxy_pass http://localhost:8001;
        include    /etc/nginx/conf.d/proxy.conf;
    }

/etc/nginx/conf.d/proxy.conf contains:

proxy_http_version                 1.1;
proxy_cache_bypass                 $http_upgrade;

# Proxy headers
proxy_set_header Upgrade           $http_upgrade;
proxy_set_header Connection        "upgrade";
proxy_set_header Host              $host;
proxy_set_header X-Real-IP         $remote_addr;
proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host  $host;
proxy_set_header X-Forwarded-Port  $server_port;

# Proxy timeouts
proxy_connect_timeout              60s;
proxy_send_timeout                 60s;
proxy_read_timeout                 60s;

[Sushi1324]

I should have clarified that I am running the reverse proxy on a second virtual machine, not directly on the peering manager box (I have several sites that all land on this reverse proxy). I just tested it with the same proxy configuration that you have and saw no change.

When debugging this issue, I used this Javascript code snippet to check what url the API call returns and it always returns an absolute URL with the internal IP of the peering manager node.

$.ajax({
      method: "get",
      url: "/api/peering/routers/1/test-napalm-connection/",
    }).done(function(r) {
      console.log(r.url);
    });

Sample output: http://100.100.118.85/api/extras/job-results/171/

Does your test setup return an absolute URL when calling that snippit? If the API just returned a relative url (/api/extras/job-results/171/), then this shouldn't break when behind a separate proxy. That's how all of the other API calls are made, which is how everything else works fine behind the proxy.

[gmazoyer]

I see, that might be Django not looking at the X-Forwarded-Proto header then. Can you try adding the following lines in the peeting_manager/configuration.py and restart the WSGI process?

USE_X_FORWARDED_HOST = True
SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")

It supposed to be set by default but depends on the version you are using.

[Sushi1324]

I added those and saw no change in behavior unfortunately.

[gmazoyer]

That's weird. In my case the generated URL does use https:// and not http:// even though my nginx to gunicorn protocol is HTTP. What version of Peering Manager are you currently using?

[Sushi1324]

That suggestion did give me an idea though that helped me figure out what was going on. I am still using nginx as a proxy directly on the peering manager node as the documentation describes here: https://peering-manager.readthedocs.io/en/stable/setup/4b-nginx/, so the traffic is actually getting proxied twice for non static objects.

I removed the set_header options from the proxy directly on the node, which was resetting the headers to what the proxy was using to connect, which of course was the internal IP and port 80. I also had the server_name set to "_", but I had to change that to my actual domain name. My nginx configuration now looks like this on my peering manager node:

server {
	listen 80;

	server_name { domain_name };

	location / {
		proxy_pass http://127.0.0.1:8001;
	}

	location /static {
		alias /data/peering-manager/static;
	}

}

Once I reloaded that configuration everything seems to have started working as expected. I still wonder if it might be simple for the API to return a relative URL that would've helped with this, but that might be an incorrect assumption.

I appreciate all the help!