Nextcloud:apache behind NGINX reverse proxy not using X-Forwarded-... headers

[maanloper]

I have been trying day and night, but cannot find a solution to the following problem:
Nextcloud refuses to use the X-Forwarded-... headers (-For, -Proto & -Host) when run behind an NGINX reverse proxy (also in docker).

My setup:
Host: Ubuntu 20.04 LTS
Docker & docker-compose both latest (fresh install)
compose.yml (also have nextcloud-cron and redis, but not shown for length):

services:
 nginx-proxy:
  image: nginx:alpine
  container_name: nginx-proxy
  restart: always
  volumes:
   - ${PTH}/nginx-volumes/conf.d:/etc/nginx/conf.d
   - ${PTH}/nginx-volumes/vhost.d:/etc/nginx/vhost.d
   - ${PTH}/nginx-volumes/html:/usr/share/nginx/html
   - ${PTH}/nginx-volumes/certs:/etc/nginx/certs:ro
  networks:
   - proxy
   - nextcloud
  ports:
   - "80:80"
   - "443:443"

 nextcloud:
  build:
   context: ${PTH}/dockerfiles
   dockerfile: nextcloud
  container_name: nextcloud
  restart: always
  volumes:
   - ${PTH}/nextcloud-volumes/html:/var/www/html
   - /nextcloud-data:/var/www/html/data
  environment:
   - VIRTUAL_HOST=www.example.org
   - LETSENCRYPT_HOST=www.example.org
   - APACHE_DISABLE_REWRITE_IP=1
  networks:
   - nextcloud

 nextcloud-db:
  image: mariadb
  container_name: nextcloud-db
  restart: always
  command: --transaction-isolation=READ-COMMITTED
  networks:
   - nextcloud

Note APACHE_DISABLE_REWRITE_IP=1 as per https://github.com/nextcloud/docker#using-the-apache-image-behind-a-reverse-proxy-and-auto-configure-server-host-and-protocol

In nginx.conf the following headers are set:

proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

And the nextcloud config.php relevant lines:

<?php
$CONFIG = array (
  'htaccess.RewriteBase' => '/',
  'trusted_proxies' =>
  array (
    0 => 'nginx-proxy',
  ),
  'trusted_domains' =>
  array (
    0 => 'localhost',
    1 => 'www.example.org',
  ),
  'overwrite.cli.url' => 'https://www.example.org',
);

What I have checked to make sure the NGINX reverse proxy passes the correct headers:
I have replaced index.php in /var/www/html/nextcloud with the following content to view the headers received by Nextcloud (inspired by #800 (comment)):

<?php
print_r($_SERVER);
?>

Results in the correct HTTP_X_FORWARDED_... headers being passed to Nextcloud:

Array (
[HTTP_AUTHORIZATION] =>
[modHeadersAvailable] => true
[htaccessWorking] => true
[front_controller_active] => true
[PATH_INFO] =>
[HTTP_HOST] => www.example.org
[HTTP_X_FORWARDED_HOST] => www.example.org
[HTTP_X_REAL_IP] => 192.168.2.254
[HTTP_X_FORWARDED_FOR] => 192.168.2.254
[HTTP_X_FORWARDED_PROTO] => https
[HTTP_CONNECTION] => close
[HTTP_SEC_CH_UA] => " Not A;Brand";v="99", "Chromium";v="90", "Google Chrome";v="90"
[HTTP_SEC_CH_UA_MOBILE] => ?0
[HTTP_UPGRADE_INSECURE_REQUESTS] => 1
[HTTP_USER_AGENT] => Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36
[HTTP_ACCEPT] => text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
[HTTP_SEC_FETCH_SITE] => none
[HTTP_SEC_FETCH_MODE] => navigate
[HTTP_SEC_FETCH_USER] => ?1
[HTTP_SEC_FETCH_DEST] => document
[HTTP_ACCEPT_ENCODING] => gzip, deflate, br
[HTTP_ACCEPT_LANGUAGE] => en-US,en;q=0.9,nl;q=0.8
[HTTP_COOKIE] => nc_sameSiteCookielax=true; nc_sameSiteCookiestrict=true; __Host-nc_sameSiteCookielax=true; __Host-nc_sameSiteCookiestrict=true; oc_sessionPassphrase=aVD6Y9e2bKwlcy9hO3sesvG2ZpAfwxxVIpl%2FKMnFG0tEC8QVdiCS5oaXsYS8WhQcEGUyejRJ8NXMo596GiKy6MJIdZ6fl8F2m7cWUhwDLqGttTCthNBSdBT5fka6%2B0IS; nc_username=Admin; ocbr5gtdx8ep=c746bfdcee0f25a0fbd742e049eb3cf1; nc_token=9ViR2EVpfAGL1ovx5vTIHqIUHXWOZvEd; nc_session_id=c746bfdcee0f25a0fbd742e049eb3cf1
[PATH] => /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[SERVER_SIGNATURE] =>
Apache Server at www.example.org Port 80
[SERVER_SOFTWARE] => Apache
[SERVER_NAME] => example.org
[SERVER_ADDR] => 192.168.160.5
[SERVER_PORT] => 80
[REMOTE_ADDR] => 192.168.160.4
[DOCUMENT_ROOT] => /var/www/html
[REQUEST_SCHEME] => http
[CONTEXT_PREFIX] =>
[CONTEXT_DOCUMENT_ROOT] => /var/www/html
[SERVER_ADMIN] => webmaster@localhost
[SCRIPT_FILENAME] => /var/www/html/index.php
[REMOTE_PORT] => 56342
[GATEWAY_INTERFACE] => CGI/1.1
[SERVER_PROTOCOL] => HTTP/1.1
[REQUEST_METHOD] => GET
[QUERY_STRING] =>
[REQUEST_URI] => /
[SCRIPT_NAME] => /index.php
[PHP_SELF] => /index.php
[REQUEST_TIME_FLOAT] => 1621199303.6339
[REQUEST_TIME] => 1621199303
[argv] => Array ( )
[argc] => 0 )

Now my question: why does Nextcloud not honor the headers as it should per it's config?
Any other container I run behind my NGINX proxy works like a charm with the headers. Just Nextcloud is being a, well, you know...

See https://github.com/nextcloud/server/blob/9de329a4c2327767d86bd7f594b232eb56af0d01/lib/private/AppFramework/Http/Request.php#L692? for the relevant server code that should auto-configure using the headers.

And before anyone mentions it:
Yes, I know I can set the OVERWRITEPROTOCOL=https environment variable, but that doesn't fix the problem of Nextcloud not using the X-Forwarded-For header to get the client IP. Only 'make-shift' solution so far is to remove APACHE_DISABLE_REWRITE_IP, and in remoteip.conf make some sketchy edits with RemoteIPInternalProxy etc, see https://help.nextcloud.com/t/how-to-get-the-real-ips-in-logs/83096/31

Sooooo, who can help me pinpoint the problem? Might be me doing something stupid, might be something wrong with the nextcloud:apache containers, might be something wrong entire different. I just do not know where to look anymore for a solution...

EDIT:

Sooo, I found that adding the IP address of the NGINX-container instead of nginx-proxy solves the problem:

  'trusted_proxies' =>
  array (
    0 => 'nginx-proxy',
  ),

Changed to:

  'trusted_proxies' =>
  array (
    0 => '192.168.160.4',
  ),

(192.168.160.4 is the IP of my nginx-proxy container, this changes every time you start the docker containers)

I do not understand why, since in other containers I can happily use nginx-proxy instead of an every varying IP-address whenever docker containers are down (e.g. for backups).

So the question would be: how can I configure the 'trusted_proxies' config that it dynamically points to the nginx-proxy container? Note that whitelisting the entire 192.XXX range is not a solution; due to hairpin routing all internal traffic is logged as the router IP address (ISP router, non-configurable :( ).

EDIT2:
Problem comes from:

protected function matchesTrustedProxy($trustedProxy, $remoteAddress) {
		$cidrre = '/^([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})\/([0-9]{1,2})$/';

		if (preg_match($cidrre, $trustedProxy, $match)) {
			$net = $match[1];
			$shiftbits = min(32, max(0, 32 - intval($match[2])));
			$netnum = ip2long($net) >> $shiftbits;
			$ipnum = ip2long($remoteAddress) >> $shiftbits;

			return $ipnum === $netnum;
		}

		return $trustedProxy === $remoteAddress;
	}

in https://github.com/nextcloud/server/blob/9de329a4c2327767d86bd7f594b232eb56af0d01/lib/private/AppFramework/Http/Request.php#L607.
It does not first resolve the trusted_proxy name, just CIDR comparison or literal IP-comparison.

[ghost]

You can find a solution here:
https://help.nextcloud.com/t/how-to-get-the-real-ips-in-logs/83096/29

The problem is the directive in remoteip.conf. This is a known problem of the current images it seems.

Adapt this for a custom Dockerfile using the apache image as base:

RUN sed -i -e 's/RemoteIPHeader X-Real-IP/RemoteIPHeader X-Forwarded-For/g' /etc/apache2/conf-enabled/remoteip.conf
RUN sed -i -e 's/RemoteIPTrustedProxy/#RemoteIPTrustedProxy/g' /etc/apache2/conf-enabled/remoteip.conf
# echo "RemoteIPInternalProxy $(env | grep TRUSTED_PROXIES | cut -d= -f2)" >> /etc/apache2/conf-enabled/remoteip.conf
RUN sed -i -e '/LogFormat/s/%h/%a/g' /etc/apache2/apache2.conf

Remember to set the TRUSTED_PROXIES.