Is it possible to automatically add matrix-nginx-proxy to an extra network with a specific alias?

[briankendall]

I'm using an external nginx forward proxy running in another docker container to handle routing and https among all of my web services. Consequently for matrix I have my vars.yml set up with:

matrix_ssl_retrieval_method: none
matrix_nginx_proxy_https_enabled: false
matrix_nginx_proxy_container_http_host_bind_port: ''
matrix_nginx_proxy_container_federation_host_bind_port: ''

What I'd like to do is add the matrix-nginx-proxy container and my external proxy's container to a different docker network than used by the rest of matrix, and assign matrix-nginx-proxy a specific alias on that network so that my proxy has a reliable hostname to use for it.

I know I can use matrix_docker_network to assign a network for all of the matrix related docker containers, and could use the network that my external proxy is using, but then I think that's going against proper separation of concerns. The external nginx container shouldn't be on the same network and therefore have access to every matrix docker container. It should only be able to communicate with matrix-nginx-proxy.

I can set this up manually after running start, and have confirmed that it works, but of course every time I restart matrix then I'd have to manually set it up again. It needs to be automated.

Is there any way to do this?

[spantaleev]

Perhaps matrix-nginx-proxy could remain as-is and you could have your other nginx proxy container on another Docker network. You can then connect your other nginx proxy to the matrix network using docker network connect.

Unfortunately, this exposes all Matrix services (not just matrix-nginx-proxy) to your other nginx proxy, so it's likely no good.

---

Maybe ideally, you'd have matrix-nginx-proxy on another network (e.g. matrix-public), have that container connected to the matrix network (via docker network connect ..), so that it can access all matrix services. Then your other nginx proxy could be on yet another network, which is connected to the matrix-public network. This way, your other nginx proxy would be able to access matrix-nginx-proxy, but none of the other services (which remain in the matrix network).

The problem with docker network connect is that it needs to be done after the container is created. It's tricky to set this up with systemd, I think.

[briankendall]

Maybe ideally, you'd have matrix-nginx-proxy on another network (e.g. matrix-public), have that container connected to the matrix network (via docker network connect ..), so that it can access all matrix services. Then your other nginx proxy could be on yet another network, which is connected to the matrix-public network.

This is precisely what I'd like to do. I already have a network set up for my front facing nginx proxy that allows it to access just the services it needs to. What I'm looking for is a way to have running start add matrix-nginx-proxy to that network after it's done starting all the various docker instances. As things work now, if I run stop and then start again, matrix-nginx-proxy loses any extra networks I've added it to.

[spantaleev]

I think you could try if modifying roles/matrix-nginx-proxy/templates/systemd/matrix-nginx-proxy.service.j2 in the following way works for you:

1. Changing the network like this:

---network={{ matrix_docker_network }}
+--network=matrix-public

This makes the matrix-nginx-proxy container part of a different network (matrix-public), which is distinct from the matrix network.

You can adjust the name of this matrix-public network as you see fit. In any case, make sure that the network exists (by running docker network create matrix-public manually).

2. Adding a line like this below the ExecStart statement:

ExecStartPost={{ matrix_host_command_docker }} network connect {{ matrix_docker_network }} matrix-nginx-proxy

This connects the matrix-nginx-proxy container to the matrix network where all other services are. This ExecStartPost command is supposed to run after ExecStart.

I wonder if running the docker network connect command immediately after the docker run command will work though. Or will Docker complain that the matrix-nginx-proxy container hasn't fully started yet or something. If so, we may need to use sleep somehow. Example:

ExecStartPost={{ matrix_host_command_sh }} -c '{{ matrix_host_command_sleep }} 3 && {{ matrix_host_command_docker }} network connect {{ matrix_docker_network }} matrix-nginx-proxy'

3. Re-running the playbook (ansible-playbook .... --tags=setup-nginx-proxy,start)

---

Let me know what you find and we can turn this into a PR!

[briankendall]

Option 2 with sleeping does seem to work! I set it up so that the matrix-nginx-proxy role has an additional variable named matrix_nginx_proxy_extra_network that defaults to an empty string. If it's not empty when running the playbook then it will add ExecStartPost so that it adds the network to matrix-nginx-proxy.

The one thing that worried me was that there could be an error starting matrix-nginx-proxy is it takes longer than 3 seconds for it to start up for some reason. So I did a bit of bash-fu to have it poll the container to see if it's up for up to 30 seconds before connecting it to the extra network. So what I'm proposing is that we add the following to matrix-nginx-proxy.service.j2, after the ExecStart=... line:

{% if matrix_nginx_proxy_extra_network != '' %}
ExecStartPost={{ matrix_host_command_sh }} -c 'attempt=0; while [ $attempt -le 29 ]; do attempt=$(( $attempt + 1 )); if [ "`docker inspect -f {{ '{{.State.Running}}' }} matrix-nginx-proxy 2> /dev/null`" = "true" ]; then break; fi; sleep 1; done; {{ matrix_host_command_docker }} network connect {{ matrix_nginx_proxy_extra_network }} matrix-nginx-proxy'
{% endif %}

This is the first time I've ever worked with ansible and jinja directly so I'm not sure if I made an error or formatted it in a daft way. But it does seem to work.

Let me know what you think, and if it looks good, I'll submit a PR.

[spantaleev]

Ouch, that bash sleep loop is a little scary.. But I guess it could work.

---

I think we should rethink the network situation some more (instead of just having matrix_nginx_proxy_extra_network) and redo things for both matrix-nginx-proxy and matrix-coturn as well (these both open ports directly).

I'm thinking we should introduce this change (2 separate matrix* networks) to the playbook for all users, and not merely as an optional thing added on top of the matrix-nginx-proxy role.

[briankendall]

Yes I'm not a fan of obtuse bash one-liners, but it felt like more of a correct solution than assuming the container will be up and running in three seconds. Written out in multiple lines makes it much more readable and less scary looking too.

Are you thinking that matrix-nginx-proxy and matrix-coturn will both always be added to a second network intended to be used with other containers, something like matrix-public?

[spantaleev]

You can now use matrix_nginx_proxy_container_additional_networks to connect matrix-nginx-proxy to another container network. I've made it use your bash loop.

Example:

matrix_nginx_proxy_container_additional_networks: [traefik]

---

Initially, I had created a matrix_nginx_proxy_container_network variable as well with this documentation:

# The container network (`--network` to use for matrix-nginx-proxy).
# This can be different than the network used for the other Matrix services.
# For background, see:
# - `matrix_nginx_proxy_container_network_to_connect`.
# - https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/1498
#
# Use this with caution.
#
# Using a different network may cause various breakage in certain configurations.
# For example, when Synapse works with workers enabled, bridges internally connect to Synapse
# via matrix-nginx-proxy, which ensures requests are forwarded to the correct worker process.
# A bridge that's part of another network (than matrix-nginx-proxy) may fail to reach matrix-nginx-proxy.

.. but decided against it. As that comment says, making matrix-nginx-proxy part of another network will break certain bridges, etc.

It's better to leave matrix-nginx-proxy part of the matrix network. To assist your other reverse-proxy (running in a container on another network), it's better to have matrix-nginx-proxy get connected to that other network. You can achieve this using matrix_nginx_proxy_container_additional_networks.

---

This "Synapse in workers mode is special and bridges need matrix-nginx-proxy's help to reach Synapse anyway" thing likely points out another problem with our setup.

Perhaps there should be a reverse-proxy that is part of the matrix-synapse role and that hides all the complexity of exposing Synapse to other services.

Right now the matrix-nginx-proxy is that mediator, doing one thing or another depending on matrix_synapse_workers_enabled and on whether matrix-corporal is enabled, but most of this complexity should probably live in the matrix-synapse role.

That role would ideally expose matrix-synapse:8008 for Client-Server APIs (and handle forwarding traffic to matrix-synapse-all-in-one or to various matrix-synapse-* processes) and matrix-synapse:8448 for Federation APIs (again, either making all requests go to matrix-synapse-all-in-one or to various federation worker processes).

Other roles would then only need to care about these 2 endpoints (matrix-synapse:8008 and matrix-synapse:8448).

The matrix-corporal complexity ("should a specific Client-Server request go to the homeserver software or to matrix-corporal?") probably needs to remain on the public side (in the matrix-nginx-proxy role).