Cannot finish setup-all after recent commit (completely moved to Traefik reverse-proxy) #3095

ppkhoa · 2024-01-15T19:55:30Z

Describe the bug
Following instructions from CHANGELOG.md, some add-ons services cannot be started, even though matrix_playbook_reverse_proxy_type: 'other-traefik-container' have been set, those add-ons kept looking for matrix-traefik.service and failed to start. In addition, there's an error/unknown variable in group_vars/matrix_servers: matrix_well_known_ident, this will cause the setup/upgrade to fail since it's not defined anywhere:

TASK [custom/matrix-static-files : Ensure matrix-static-files container network is created] ******************************
fatal: [matrix.<mydomain>]: FAILED! =>
  msg: |-
    The task includes an option with an undefined variable. The error was: {{ devture_traefik_container_network if matrix_playbook_reverse_proxy_type == 'playbook-managed-traefik' else matrix_well_known_ident }}: 'matrix_well_known_ident' is undefined

    The error appears to be in '/root/matrix-docker-ansible-deploy/roles/custom/matrix-static-files/tasks/install.yml': line 84, column 3, but maybe elsewhere in the file depending on the exact syntax problem.

    The offending line appears to be:
    - name: Ensure matrix-static-files container network is created
      ^ here

I also encountered issues at the end when services being started, matrix-synapse.service failed to start because it cannot resolve matrix-postgres to the matrix-postgres container. From my vars.yml below, if I uncomment the following lines, just setup-all will complete without any errors:

matrix_well_known_ident: 'traefik'
matrix_synapse_container_additional_networks: ["traefik", "matrix-postgres"] #uncomment the next line with only traefik network
matrix_playbook_reverse_proxy_container_network: 'traefik'

Also added this line so add-on services won't look for matrix-traefik.service (my guess is this was not the right way to do it, please correct me if I'm wrong, maybe change the matrix_addons_homeserver_systemd_services_list variable from group_vars/matrix_servers?)

matrix_playbook_internal_matrix_client_api_traefik_entrypoint_enabled: false

To Reproduce
My vars.yml file looks like this:

---
# The bare domain name which represents your Matrix identity.
# Matrix user ids for your server will be of the form (`@user:<matrix-domain>`).
#
# Note: this playbook does not touch the server referenced here.
# Installation happens on another server ("matrix.<matrix-domain>").
#
# If you've deployed using the wrong domain, you'll have to run the Uninstalling step,
# because you can't change the Domain after deployment.
#
# Example value: example.com
matrix_domain: <mydomain>

# The Matrix homeserver software to install.
# See `roles/matrix-base/defaults/main.yml` for valid options.
matrix_homeserver_implementation: synapse

# A secret used as a base, for generating various other secrets.
# You can put any string here, but generating a strong one is preferred (e.g. `pwgen -s 64 1`).
matrix_homeserver_generic_secret_key: 'secret'

# This is something which is provided to Let's Encrypt when retrieving SSL certificates for domains.
#
# In case SSL renewal fails at some point, you'll also get an email notification there.
#
# If you decide to use another method for managing SSL certificates (different than the default Let's Encrypt),
# you won't be required to define this variable (see `docs/configuring-playbook-ssl-certificates.md`).
#
# Example value: someone@example.com
matrix_playbook_ssl_enabled: true
matrix_playbook_internal_matrix_client_api_traefik_entrypoint_enabled: false
matrix_playbook_reverse_proxy_type: 'other-traefik-container'
#matrix_well_known_ident: 'traefik'
matrix_playbook_reverse_proxyable_services_additional_network: 'traefik'
matrix_playbook_reverse_proxy_hostname: 'traefik'
#matrix_playbook_reverse_proxy_container_network: 'traefik'
devture_traefik_certs_dumper_enabled: true
devture_traefik_certs_dumper_ssl_dir_path: "/opt/containers/traefik/data/"
devture_traefik_config_certificatesResolvers_acme_enabled: false
matrix_playbook_public_matrix_federation_api_traefik_entrypoint_port: 8449
matrix_playbook_public_matrix_federation_api_traefik_entrypoint_host_bind_port: 127.0.0.1:8449
matrix_playbook_public_matrix_federation_api_traefik_entrypoint_config_custom:
  forwardedHeaders:
    insecure: true
#matrix_synapse_container_additional_networks: ["traefik", "matrix-postgres"]
matrix_synapse_container_additional_networks: ["traefik"]
matrix_coturn_systemd_required_services_list: ['docker.service']
matrix_coturn_container_additional_volumes: |
  {{
    (
      [
       {
         'src': (matrix_ssl_config_dir_path + '/live/' + matrix_domain + '/fullchain.pem'),
         'dst': '/fullchain.pem',
         'options': 'ro',
       },
       {
         'src': (matrix_ssl_config_dir_path + '/live/' + matrix_domain + '/privkey.pem'),
         'dst': '/privkey.pem',
         'options': 'ro',
       },
      ] if matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] and matrix_coturn_tls_enabled else []
    )
    +
    (
      [
       {
         'src': (devture_traefik_certs_dumper_dumped_certificates_dir_path +  '/' + matrix_domain + '/certificate.crt'),
         'dst': '/certificate.crt',
         'options': 'ro',
       },
       {
         'src': (devture_traefik_certs_dumper_dumped_certificates_dir_path +  '/' + matrix_domain + '/privatekey.key'),
         'dst': '/privatekey.key',
         'options': 'ro',
       },
      ] if matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] and devture_traefik_certs_dumper_enabled and matrix_coturn_tls_enabled else []
    )
  }}
# A Postgres password to use for the superuser Postgres user (called `matrix` by default).
#
# The playbook creates additional Postgres users and databases (one for each enabled service)
# using this superuser account.
matrix_synapse_auto_compressor_enabled: true
matrix_synapse_auto_compressor_container_image_self_build: true
devture_postgres_connection_password: 'sdfsdfsdfsdf'
matrix_architecture: 'arm64'
matrix_static_files_container_labels_base_domain_enabled: true
matrix_static_files_file_index_html_template: |-
  <!doctype html>
  <meta charset="utf-8" />
  <html>
    <head>
        <title>Systems Status</title>
        <style type="text/css">
            body, html
            {
                margin: 0; padding: 0; height: 100%; overflow: hidden;
            }
            #content
            {
                position:absolute; left: 0; right: 0; bottom: 0; top: 0px;
            }
        </style>
    </head>
    <body>
      <div id="content">
            <iframe width="100%" height="100%" frameborder="0" src="https://khoa.betteruptime.com/"></iframe>
      </div>
      <div style="display:none;">proven233175</div>
    </body>
  </html>
matrix_dimension_hsts_preload_enabled: true
matrix_synapse_admin_hsts_preload_enabled: true
matrix_client_element_hsts_preload_enabled: true
matrix_sliding_sync_enabled: true
matrix_coturn_turn_external_ip_address: <myip>
matrix_synapse_admin_enabled: true
matrix_synapse_enable_registration: true
matrix_synapse_enable_registration_captcha: true
matrix_synapse_recaptcha_public_key: 'dsfsdfdsf'
matrix_synapse_recaptcha_private_key: 'sdfsdfsdf'
matrix_synapse_allow_public_rooms_over_federation: true
matrix_synapse_forget_rooms_on_leave: true
matrix_mautrix_gmessages_enabled: false
exim_relay_sender_address: "me@<mydomain>"
exim_relay_relay_use: true
exim_relay_relay_host_name: "smtp.sendgrid.net"
exim_relay_relay_host_port: 587
exim_relay_relay_auth: true
exim_relay_relay_auth_username: "apikey"
exim_relay_relay_auth_password: "secret"
matrix_synapse_ext_password_provider_shared_secret_auth_com_devture_shared_secret_auth_support_enabled: true
matrix_synapse_ext_password_provider_shared_secret_auth_enabled: true
matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret: 'secret'
matrix_mautrix_wsproxy_enabled: true
matrix_mautrix_imessage_appservice_token: 'secret'
matrix_mautrix_imessage_homeserver_token: 'secret'
matrix_mautrix_androidsms_appservice_token: 'secret'
matrix_mautrix_androidsms_homeserver_token: 'secret'
matrix_mautrix_wsproxy_syncproxy_shared_secret: 'secret'
matrix_mautrix_wsproxy_bind_port: true
matrix_mautrix_imessage_appservice_bot_username: imessage-bot
matrix_mautrix_wsproxy_syncproxy_database_username: 'matrix'
matrix_mautrix_wsproxy_syncproxy_database_password: 'somepassword'
matrix_mautrix_wsproxy_syncproxy_database_hostname: 'matrix-postgres'
matrix_mautrix_wsproxy_syncproxy_database_port: 5432
matrix_mautrix_wsproxy_syncproxy_database_name: 'matrix_mautrix_wsproxy_syncproxy'

matrix_synapse_federation_enabled: true
matrix_synapse_federation_domain_whitelist:
- matrix.org
- vector.im
matrix_mautrix_whatsapp_enabled: true
matrix_synapse_configuration_extension_yaml: |
  experimental_features:
    msc2716_enabled: true
matrix_mautrix_whatsapp_configuration_extension_yaml:
  bridge:
    encryption:
      allow: true
      default: true
    history_sync:
      backfill: true
    permissions:
      '@khoa:<mydomain>': admin
matrix_mautrix_telegram_enabled: true
matrix_mautrix_telegram_configuration_extension_yaml:
  bridge:
    encryption:
      allow: true
      default: true
    history_sync:
      backfill: true
    permissions:
      '@khoa:<mydomain>': admin
matrix_mautrix_telegram_api_id: nope
matrix_mautrix_telegram_api_hash: secret
matrix_mautrix_facebook_enabled: true
matrix_mautrix_facebook_configuration_extension_yaml: |
  bridge:
    encryption:
      allow: true
      default: true
    history_sync:
      backfill: true
    permissions:
      '@khoa:<mydomain>': admin
matrix_mautrix_discord_enabled: true
matrix_ma1sd_enabled: true
matrix_ma1sd_container_image_self_build: true
matrix_ma1sd_matrixorg_forwarding_enabled: true
matrix_ma1sd_configuration_extension_yaml: |
forward:
  servers:
    - 'matrix-org'
  threepid:
    medium:
      msisdn:
        connectors:
          twilio:
            account_sid: 'password'
            auth_token: 'password'
            number: '+1<number>'
matrix_dimension_enabled: true
matrix_dimension_admins: ['@khoa:<mydomain>', '@dimension:<mydomain>']
matrix_dimension_access_token: "password"
matrix_bot_matrix_reminder_bot_enabled: true

# Adjust this to whatever password you chose when registering the bot user
matrix_bot_matrix_reminder_bot_matrix_user_password: <password>
matrix_bot_matrix_reminder_bot_matrix_user_id_localpart: bot.reminder
# Adjust this to your timezone
matrix_bot_matrix_reminder_bot_reminders_timezone: Asia/Bangkok
matrix_mautrix_signal_enabled: true
matrix_mautrix_signal_configuration_extension_yaml: |
  bridge:
    permissions:
      '@khoa:<mydomain>': admin
matrix_bot_chatgpt_enabled: false

matrix_bot_chatgpt_openai_api_key: 'password'
matrix_bot_chatgpt_matrix_access_token: 'password'
matrix_client_cinny_enabled: false

Expected behavior
just setup-all completes without error and services continue to work without any issues.

Matrix Server:

OS: [e.g. Ubuntu 21.04]
Architecture [e.g. amd64, arm32, arm64]

Ansible:
If your problem appears to be with Ansible, tell us:

where you run Ansible -- on Matrix server itself, localhost
what version of Ansible you're running: ansible [core 2.13.4]

Additional context
I was already running Traefik on my own (not managed by playbook) and instructions on completely removing nginx reverse-proxy and migrating everything to Traefik did not work, requires extra steps to get it working, not sure if it's the right way to do it, but it's working for now and Matrix Federation Tester showed all green.

The text was updated successfully, but these errors were encountered:

spantaleev · 2024-01-15T20:04:35Z

Your error message seems to be cut off, but I imagine it says that matrix_well_known_ident is undefined.

If so, it should be fixed in 0b76573

spantaleev · 2024-01-15T20:14:00Z

Addon services (bridges, bots) are explicitly being connected to Traefik, because of Traefik's new role as an intermediary service through which addon services connect to the homeserver. This is described here.

The same section in the changelog entry also tells you that you can disable this behavior via matrix_playbook_internal_matrix_client_api_traefik_entrypoint_enabled: false (like you've done), but does not recommend this.

About Synapse failing to start, it'd be interesting to see what /etc/systemd/system/matrix-synapse.service looks like for you. Do you see it being connected to the matrix-postgres network without having to resort to changing matrix_synapse_container_additional_networks manually?

ppkhoa · 2024-01-15T20:22:22Z

Addon services (bridges, bots) are explicitly being connected to Traefik, because of Traefik's new role as an intermediary service through which addon services connect to the homeserver.

I already defined other-traefik-container and Traefik network name in vars.yml, I still have to set matrix_playbook_internal_matrix_client_api_traefik_entrypoint_enabled to false? I would expect the addon services to connect to the specified Traefik network and not look for matrix-traefik.service since Traefik being managed by me.

See error here:

TASK [galaxy/systemd_service_manager : Ensure systemd services are started] *************************************************************************************************************
changed: [matrix.<mydomain>] => (item={'name': 'matrix-postgres.service', 'priority': 500, 'groups': ['matrix', 'postgres']})
changed: [matrix.<mydomain>] => (item={'name': 'matrix-synapse.service', 'priority': 1000, 'groups': ['matrix', 'homeservers', 'synapse']})
changed: [matrix.<mydomain>] => (item={'name': 'matrix-client-element.service', 'priority': 2000, 'groups': ['matrix', 'clients', 'element', 'client-element']})
ok: [matrix.<mydomain>] => (item={'name': 'matrix-exim-relay.service', 'priority': 2000, 'groups': ['matrix', 'mailer', 'exim-relay']})
failed: [matrix.<mydomain>] (item={'name': 'matrix-ma1sd.service', 'priority': 2000, 'groups': ['matrix', 'ma1sd']}) => changed=false
  ansible_loop_var: item
  item:
    groups:
    - matrix
    - ma1sd
    name: matrix-ma1sd.service
    priority: 2000
  msg: |-
    Unable to start service matrix-ma1sd.service: Failed to start matrix-ma1sd.service: Unit matrix-traefik.service not found.
failed: [matrix.<mydomain>] (item={'name': 'matrix-mautrix-discord.service', 'priority': 2000, 'groups': ['matrix', 'bridges', 'mautrix-discord']}) => changed=false
  ansible_loop_var: item
  item:
    groups:
    - matrix
    - bridges
    - mautrix-discord
    name: matrix-mautrix-discord.service
    priority: 2000
  msg: |-
    Unable to start service matrix-mautrix-discord.service: Failed to start matrix-mautrix-discord.service: Unit matrix-traefik.service not found.

About Synapse failing to start, it'd be interesting to see what /etc/systemd/system/matrix-synapse.service looks like for you. Do you see it being connected to the matrix-postgres network without having to resort to changing matrix_synapse_container_additional_networks manually?

Nope, docker inspect matrix-synapse does not show matrix-postgres in Networks section (that's why I added matrix-postgres in additional networks in the first place). Running just setup-all again without matrix-postgres in matrix_synapse_container_additional_networks, will post result here shortly.

…ther-traefik-container" setups Related to #3095

spantaleev · 2024-01-15T20:30:16Z

Your problem seems to lie with matrix_addons_homeserver_systemd_services_list: "{{ ([devture_traefik_identifier + '.service']) if matrix_playbook_internal_matrix_client_api_traefik_entrypoint_enabled else matrix_homeserver_systemd_services_list }}" in group_vars/matrix_servers.

It assumed you're using the integrated Traefik webserver, which is wrong.. It should take a look at the reverse-proxy type variable. It's supposedly fixed in b1e08db.

If you're still experiencing trouble with it (despite that fix), you may try redefining it to point to your Traefik container's systemd service (e.g. ``matrix_addons_homeserver_systemd_services_list: [my-traefik.service]).. or.. to an empty list: matrix_addons_homeserver_systemd_services_list: []`.

ppkhoa · 2024-01-15T20:33:09Z

This is my /etc/systemd/system/matrix-synapse.service without matrix-postgres in additional network:

[Unit]
Description=Synapse server
Requires=docker.service
After=docker.service

Requires=matrix-postgres.service
After=matrix-postgres.service

Wants=matrix-coturn.service
Wants=matrix-exim-relay.service


DefaultDependencies=no

[Service]
Type=simple
Environment="HOME=/root"
ExecStartPre=-/usr/bin/env sh -c '/usr/bin/env docker stop --time=3 matrix-synapse 2>/dev/null || true'
ExecStartPre=-/usr/bin/env sh -c '/usr/bin/env docker rm matrix-synapse 2>/dev/null || true'

ExecStartPre=/usr/bin/env docker create \
                        --rm \
                        --name=matrix-synapse \
                        --log-driver=none \
                        --user=997:1002 \
                        --cap-drop=ALL \
                        --read-only \
                        --tmpfs=/tmp:rw,noexec,nosuid,size=2500m \
                        --network=matrix-homeserver \
                        --mount type=bind,src=/matrix/synapse/config,dst=/data,ro \
                        --mount type=bind,src=/matrix/synapse/storage,dst=/matrix-media-store-parent,bind-propagation=slave \
                        --label-file=/matrix/synapse/labels \
                        --mount type=bind,src=/matrix/synapse/ext/shared_secret_authenticator.py,dst=/usr/local/lib/python3.11/site-packages/shared_secret_authenticator.py,ro \
                        --mount type=bind,src=/matrix/mautrix-discord/config/registration.yaml,dst=/matrix-mautrix-discord-registration.yaml,ro \
                        --mount type=bind,src=/matrix/mautrix-facebook/config/registration.yaml,dst=/matrix-mautrix-facebook-registration.yaml,ro \
                        --mount type=bind,src=/matrix/mautrix-signal/config/registration.yaml,dst=/matrix-mautrix-signal-registration.yaml,ro \
                        --mount type=bind,src=/matrix/mautrix-telegram/config/registration.yaml,dst=/matrix-mautrix-telegram-registration.yaml,ro \
                        --mount type=bind,src=/matrix/mautrix-whatsapp/config/registration.yaml,dst=/matrix-mautrix-whatsapp-registration.yaml,ro \
                        --mount type=bind,src=/matrix/wsproxy/config/androidsms-registration.yaml,dst=/matrix-mautrix-androidsms-registration.yaml,ro \
                        --mount type=bind,src=/matrix/wsproxy/config/imessage-registration.yaml,dst=/matrix-mautrix-imessage-registration.yaml,ro \
                        ghcr.io/matrix-org/synapse:v1.98.0  \
                        run -m synapse.app.homeserver -c /data/homeserver.yaml

ExecStartPre=/usr/bin/env docker network connect traefik matrix-synapse

ExecStart=/usr/bin/env docker start --attach matrix-synapse

# Intentional delay, so that the homeserver can manage to start and various services
# that depend on it (After/Requires) may only start after the homeserver is up and running.
ExecStartPost=-/usr/bin/env sleep 5

ExecStop=-/usr/bin/env sh -c '/usr/bin/env docker stop --time=3 matrix-synapse 2>/dev/null || true'
ExecStop=-/usr/bin/env sh -c '/usr/bin/env docker rm matrix-synapse 2>/dev/null || true'

ExecReload=/usr/bin/env docker exec matrix-synapse /bin/sh -c 'kill -HUP 1'
Restart=always
RestartSec=30
SyslogIdentifier=matrix-synapse

[Install]
WantedBy=multi-user.target

Setup failed, in addition to matrix-traefik.service error from not setting matrix_playbook_internal_matrix_client_api_traefik_entrypoint_enabled to false

TASK [galaxy/systemd_service_manager : Ensure systemd services are started] *************************************************************************************************************
changed: [matrix.<mydomain>] => (item={'name': 'matrix-postgres.service', 'priority': 500, 'groups': ['matrix', 'postgres']})
failed: [matrix.<mydomain>] (item={'name': 'matrix-synapse.service', 'priority': 1000, 'groups': ['matrix', 'homeservers', 'synapse']}) => changed=false
  ansible_loop_var: item
  item:
    groups:
    - matrix
    - homeservers
    - synapse
    name: matrix-synapse.service
    priority: 1000
  msg: |-
    Unable to start service matrix-synapse.service: Job for matrix-synapse.service failed because the control process exited with error code.
    See "systemctl status matrix-synapse.service" and "journalctl -xeu matrix-synapse.service" for details.
changed: [matrix.<mydomain>] => (item={'name': 'matrix-client-element.service', 'priority': 2000, 'groups': ['matrix', 'clients', 'element', 'client-element']})
ok: [matrix.<mydomain>] => (item={'name': 'matrix-exim-relay.service', 'priority': 2000, 'groups': ['matrix', 'mailer', 'exim-relay']})
failed: [matrix.<mydomain>] (item={'name': 'matrix-ma1sd.service', 'priority': 2000, 'groups': ['matrix', 'ma1sd']}) => changed=false
  ansible_loop_var: item
  item:
    groups:
    - matrix
    - ma1sd
    name: matrix-ma1sd.service
    priority: 2000
  msg: |-
    Unable to start service matrix-ma1sd.service: Failed to start matrix-ma1sd.service: Unit matrix-traefik.service not found.

Output from journalctl -xeu matrix-synapse.service

2024-01-15 20:36:03,593 - root - 348 - WARNING - main - ***** STARTING SERVER *****
2024-01-15 20:36:03,593 - root - 349 - WARNING - main - Server /usr/local/lib/python3.11/site-packages/synapse/app/homeserver.py version 1.98.0
2024-01-15 20:36:03,597 - synapse.app._base - 213 - ERROR - main - Exception during startup
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/synapse/app/homeserver.py", line 353, in setup
    hs.setup()
  File "/usr/local/lib/python3.11/site-packages/synapse/server.py", line 340, in setup
    self.datastores = Databases(self.DATASTORE_CLASS, self)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/__init__.py", line 65, in __init__
    with make_conn(database_config, engine, "startup") as db_conn:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 160, in make_conn
    native_db_conn = engine.module.connect(**db_params)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.OperationalError: could not translate host name "matrix-postgres" to address: Temporary failure in name resolution
**********************************************************************************
 Error during initialisation:
     Traceback (most recent call last):
       File "/usr/local/lib/python3.11/site-packages/synapse/app/homeserver.py", line 353, in setup
         hs.setup()
       File "/usr/local/lib/python3.11/site-packages/synapse/server.py", line 340, in setup
         self.datastores = Databases(self.DATASTORE_CLASS, self)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/__init__.py", line 65, in __init__
         with make_conn(database_config, engine, "startup") as db_conn:
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 160, in make_conn
         native_db_conn = engine.module.connect(**db_params)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       File "/usr/local/lib/python3.11/site-packages/psycopg2/__init__.py", line 122, in connect
         conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     psycopg2.OperationalError: could not translate host name "matrix-postgres" to address: Temporary failure in name resolution


 There may be more information in the logs.
**********************************************************************************

ppkhoa · 2024-01-15T20:35:32Z

Commit b1e08db seems to fix the addon services kept looking matrix-traefik.service even when Traefik is not being managed by playbook issue. Thanks!

spantaleev · 2024-01-15T20:37:50Z

It's odd to see Synapse not being connected to neither the Postgres network (matrix-postgres), nor to the exim-relay network.

It seems like you haven't disabled either of these services, so Synapse should be connected to both by default.

Maybe there's something wrong here:

matrix-docker-ansible-deploy/group_vars/matrix_servers

Lines 3890 to 3903 in b1e08db

    
           matrix_synapse_container_additional_networks: | 
        
             {{ 
        
               ( 
        
                 ([matrix_playbook_reverse_proxyable_services_additional_network] if matrix_synapse_container_labels_traefik_enabled and matrix_playbook_reverse_proxyable_services_additional_network else []) 
        
                 + 
        
                 ([devture_postgres_container_network] if (devture_postgres_enabled and devture_postgres_container_network != matrix_synapse_container_network and matrix_synapse_database_host == devture_postgres_connection_hostname) else []) 
        
                 + 
        
                 ([redis_container_network] if matrix_synapse_redis_enabled and matrix_synapse_redis_host == redis_identifier else []) 
        
                 + 
        
                 ([exim_relay_container_network] if (exim_relay_enabled and matrix_synapse_email_enabled and matrix_synapse_email_smtp_host == exim_relay_identifier and matrix_synapse_container_network != exim_relay_container_network) else []) 
        
                 + 
        
                 ([matrix_ma1sd_container_network] if (matrix_ma1sd_enabled and matrix_synapse_account_threepid_delegates_msisdn == matrix_synapse_account_threepid_delegates_msisdn_mas1sd_url and matrix_synapse_container_network != matrix_ma1sd_container_network) else []) 
        
               ) | unique 
        
             }}

It may be helpful if you could add - debug: var="VARIABLE_NAME" statements somewhere in the playbook code for various of these variables (and for matrix_synapse_container_additional_networks itself) and see where our expectations are failing. Well, matrix_synapse_container_additional_networks would likely just contain traefik, given the resulting matrix-synapse.service file, but...

ppkhoa · 2024-01-15T20:43:58Z

It may be helpful if you could add - debug: var="VARIABLE_NAME" statements somewhere in the playbook code for various of these variables

Can you give examples on how to add debug statements and where? Do I add it in the roles, or group_vars/matrix_servers or my vars.yml?

ppkhoa · 2024-01-17T05:53:06Z

It's odd to see Synapse not being connected to neither the Postgres network (matrix-postgres), nor to the exim-relay network.

It seems like you haven't disabled either of these services, so Synapse should be connected to both by default.

Maybe there's something wrong here:

matrix-docker-ansible-deploy/group_vars/matrix_servers

Lines 3890 to 3903 in b1e08db

matrix_synapse_container_additional_networks: |

{{

(

([matrix_playbook_reverse_proxyable_services_additional_network] if matrix_synapse_container_labels_traefik_enabled and matrix_playbook_reverse_proxyable_services_additional_network else [])

+

([devture_postgres_container_network] if (devture_postgres_enabled and devture_postgres_container_network != matrix_synapse_container_network and matrix_synapse_database_host == devture_postgres_connection_hostname) else [])

+

([redis_container_network] if matrix_synapse_redis_enabled and matrix_synapse_redis_host == redis_identifier else [])

+

([exim_relay_container_network] if (exim_relay_enabled and matrix_synapse_email_enabled and matrix_synapse_email_smtp_host == exim_relay_identifier and matrix_synapse_container_network != exim_relay_container_network) else [])

+

([matrix_ma1sd_container_network] if (matrix_ma1sd_enabled and matrix_synapse_account_threepid_delegates_msisdn == matrix_synapse_account_threepid_delegates_msisdn_mas1sd_url and matrix_synapse_container_network != matrix_ma1sd_container_network) else [])

) | unique

}}

It may be helpful if you could add - debug: var="VARIABLE_NAME" statements somewhere in the playbook code for various of these variables (and for matrix_synapse_container_additional_networks itself) and see where our expectations are failing. Well, matrix_synapse_container_additional_networks would likely just contain traefik, given the resulting matrix-synapse.service file, but...

You're probably right, unless more details on how to debug the variables, I don't think I will spend more time running setup over and over again... I'm okay with adding 1 extra line to my vars.yml until playbook complains otherwise.

ppkhoa · 2024-01-17T06:14:50Z

Not sure if it's related, but additional network/my "other Traefik" network wasn't added to matrix-static-files container even after recent commit da1f570

Here's my current vars.yml:

---
# The bare domain name which represents your Matrix identity.
# Matrix user ids for your server will be of the form (`@user:<matrix-domain>`).
#
# Note: this playbook does not touch the server referenced here.
# Installation happens on another server ("matrix.<matrix-domain>").
#
# If you've deployed using the wrong domain, you'll have to run the Uninstalling step,
# because you can't change the Domain after deployment.
#
# Example value: example.com
matrix_domain: <mydomain>

# The Matrix homeserver software to install.
# See `roles/matrix-base/defaults/main.yml` for valid options.
matrix_homeserver_implementation: synapse

# A secret used as a base, for generating various other secrets.
# You can put any string here, but generating a strong one is preferred (e.g. `pwgen -s 64 1`).
matrix_homeserver_generic_secret_key: 'secret'

# This is something which is provided to Let's Encrypt when retrieving SSL certificates for domains.
#
# In case SSL renewal fails at some point, you'll also get an email notification there.
#
# If you decide to use another method for managing SSL certificates (different than the default Let's Encrypt),
# you won't be required to define this variable (see `docs/configuring-playbook-ssl-certificates.md`).
#
# Example value: someone@example.com
matrix_playbook_ssl_enabled: true
matrix_playbook_reverse_proxy_type: 'other-traefik-container'
matrix_playbook_reverse_proxyable_services_additional_network: 'traefik'
matrix_playbook_reverse_proxy_hostname: 'traefik'
matrix_playbook_reverse_proxy_container_network: 'traefik'
#matrix_static_files_container_additional_networks: ["traefik"]
matrix_playbook_reverse_proxyable_services_additional_network: ["traefik"]
matrix_playbook_reverse_proxyable_services_additional_network
devture_traefik_certs_dumper_enabled: true
devture_traefik_certs_dumper_ssl_dir_path: "/opt/containers/traefik/data/"
devture_traefik_config_certificatesResolvers_acme_enabled: false
matrix_playbook_public_matrix_federation_api_traefik_entrypoint_port: 8449
matrix_playbook_public_matrix_federation_api_traefik_entrypoint_host_bind_port: 127.0.0.1:8449
matrix_playbook_public_matrix_federation_api_traefik_entrypoint_config_custom:
  forwardedHeaders:
    insecure: true
matrix_synapse_container_additional_networks: ["traefik", "matrix-postgres"]

#Ignore the next 4 lines, just something to suppress Traefik complaints about missing ports...
matrix_mautrix_signal_container_labels_additional_labels: |
  traefik.http.services.matrix-mautrix-signal.loadbalancer.server.port=9254
matrix_mautrix_whatsapp_container_labels_additional_labels: |
  traefik.http.services.matrix-mautrix-whatsapp.loadbalancer.server.port=9255

matrix_coturn_systemd_required_services_list: ['docker.service']
matrix_coturn_container_additional_volumes: |
  {{
    (
      [
       {
         'src': (matrix_ssl_config_dir_path + '/live/' + matrix_domain + '/fullchain.pem'),
         'dst': '/fullchain.pem',
         'options': 'ro',
       },
       {
         'src': (matrix_ssl_config_dir_path + '/live/' + matrix_domain + '/privkey.pem'),
         'dst': '/privkey.pem',
         'options': 'ro',
       },
      ] if matrix_playbook_reverse_proxy_type in ['playbook-managed-nginx', 'other-nginx-non-container'] and matrix_coturn_tls_enabled else []
    )
    +
    (
      [
       {
         'src': (devture_traefik_certs_dumper_dumped_certificates_dir_path +  '/' + matrix_domain + '/certificate.crt'),
         'dst': '/certificate.crt',
         'options': 'ro',
       },
       {
         'src': (devture_traefik_certs_dumper_dumped_certificates_dir_path +  '/' + matrix_domain + '/privatekey.key'),
         'dst': '/privatekey.key',
         'options': 'ro',
       },
      ] if matrix_playbook_reverse_proxy_type in ['playbook-managed-traefik', 'other-traefik-container'] and devture_traefik_certs_dumper_enabled and matrix_coturn_tls_enabled else []
    )
  }}
# A Postgres password to use for the superuser Postgres user (called `matrix` by default).
#
# The playbook creates additional Postgres users and databases (one for each enabled service)
# using this superuser account.
matrix_synapse_auto_compressor_enabled: true
matrix_synapse_auto_compressor_container_image_self_build: true
devture_postgres_connection_password: 'sdfsdfsdfsdf'
matrix_architecture: 'arm64'
matrix_static_files_container_labels_base_domain_enabled: true
matrix_static_files_file_index_html_template: |-
  <!doctype html>
  <meta charset="utf-8" />
  <html>
    <head>
        <title>Systems Status</title>
        <style type="text/css">
            body, html
            {
                margin: 0; padding: 0; height: 100%; overflow: hidden;
            }
            #content
            {
                position:absolute; left: 0; right: 0; bottom: 0; top: 0px;
            }
        </style>
    </head>
    <body>
      <div id="content">
            <iframe width="100%" height="100%" frameborder="0" src="https://khoa.betteruptime.com/"></iframe>
      </div>
      <div style="display:none;">proven233175</div>
    </body>
  </html>
matrix_dimension_hsts_preload_enabled: true
matrix_synapse_admin_hsts_preload_enabled: true
matrix_client_element_hsts_preload_enabled: true
matrix_sliding_sync_enabled: true
matrix_coturn_turn_external_ip_address: <myip>
matrix_synapse_admin_enabled: true
matrix_synapse_enable_registration: true
matrix_synapse_enable_registration_captcha: true
matrix_synapse_recaptcha_public_key: 'dsfsdfdsf'
matrix_synapse_recaptcha_private_key: 'sdfsdfsdf'
matrix_synapse_allow_public_rooms_over_federation: true
matrix_synapse_forget_rooms_on_leave: true
matrix_mautrix_gmessages_enabled: false
exim_relay_sender_address: "me@<mydomain>"
exim_relay_relay_use: true
exim_relay_relay_host_name: "smtp.sendgrid.net"
exim_relay_relay_host_port: 587
exim_relay_relay_auth: true
exim_relay_relay_auth_username: "apikey"
exim_relay_relay_auth_password: "secret"
matrix_synapse_ext_password_provider_shared_secret_auth_com_devture_shared_secret_auth_support_enabled: true
matrix_synapse_ext_password_provider_shared_secret_auth_enabled: true
matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret: 'secret'
matrix_mautrix_wsproxy_enabled: true
matrix_mautrix_imessage_appservice_token: 'secret'
matrix_mautrix_imessage_homeserver_token: 'secret'
matrix_mautrix_androidsms_appservice_token: 'secret'
matrix_mautrix_androidsms_homeserver_token: 'secret'
matrix_mautrix_wsproxy_syncproxy_shared_secret: 'secret'
matrix_mautrix_wsproxy_bind_port: true
matrix_mautrix_imessage_appservice_bot_username: imessage-bot
matrix_mautrix_wsproxy_syncproxy_database_username: 'matrix'
matrix_mautrix_wsproxy_syncproxy_database_password: 'somepassword'
matrix_mautrix_wsproxy_syncproxy_database_hostname: 'matrix-postgres'
matrix_mautrix_wsproxy_syncproxy_database_port: 5432
matrix_mautrix_wsproxy_syncproxy_database_name: 'matrix_mautrix_wsproxy_syncproxy'

matrix_synapse_federation_enabled: true
matrix_synapse_federation_domain_whitelist:
- matrix.org
- vector.im
matrix_mautrix_whatsapp_enabled: true
matrix_synapse_configuration_extension_yaml: |
  experimental_features:
    msc2716_enabled: true
matrix_mautrix_whatsapp_configuration_extension_yaml:
  bridge:
    encryption:
      allow: true
      default: true
    history_sync:
      backfill: true
    permissions:
      '@khoa:<mydomain>': admin
matrix_mautrix_telegram_enabled: true
matrix_mautrix_telegram_configuration_extension_yaml:
  bridge:
    encryption:
      allow: true
      default: true
    history_sync:
      backfill: true
    permissions:
      '@khoa:<mydomain>': admin
matrix_mautrix_telegram_api_id: nope
matrix_mautrix_telegram_api_hash: secret
matrix_mautrix_facebook_enabled: true
matrix_mautrix_facebook_configuration_extension_yaml: |
  bridge:
    encryption:
      allow: true
      default: true
    history_sync:
      backfill: true
    permissions:
      '@khoa:<mydomain>': admin
matrix_mautrix_discord_enabled: true
matrix_ma1sd_enabled: true
matrix_ma1sd_container_image_self_build: true
matrix_ma1sd_matrixorg_forwarding_enabled: true
matrix_ma1sd_configuration_extension_yaml: |
forward:
  servers:
    - 'matrix-org'
  threepid:
    medium:
      msisdn:
        connectors:
          twilio:
            account_sid: 'password'
            auth_token: 'password'
            number: '+1<number>'
matrix_dimension_enabled: true
matrix_dimension_admins: ['@khoa:<mydomain>', '@dimension:<mydomain>']
matrix_dimension_access_token: "password"
matrix_bot_matrix_reminder_bot_enabled: true

# Adjust this to whatever password you chose when registering the bot user
matrix_bot_matrix_reminder_bot_matrix_user_password: <password>
matrix_bot_matrix_reminder_bot_matrix_user_id_localpart: bot.reminder
# Adjust this to your timezone
matrix_bot_matrix_reminder_bot_reminders_timezone: Asia/Bangkok
matrix_mautrix_signal_enabled: true
matrix_mautrix_signal_configuration_extension_yaml: |
  bridge:
    permissions:
      '@khoa:<mydomain>': admin
matrix_bot_chatgpt_enabled: false

matrix_bot_chatgpt_openai_api_key: 'password'
matrix_bot_chatgpt_matrix_access_token: 'password'
matrix_client_cinny_enabled: false

This is the resulting matrix-static-files.service:

[Unit]
Description=matrix-static-files server
Requires=docker.service
After=docker.service
DefaultDependencies=no

[Service]
Type=simple
Environment="HOME=/root"
ExecStartPre=-/usr/bin/env sh -c '/usr/bin/env docker stop --time=3 matrix-static-files 2>/dev/null || true'
ExecStartPre=-/usr/bin/env sh -c '/usr/bin/env docker rm matrix-static-files 2>/dev/null || true'

ExecStartPre=/usr/bin/env docker create \
                        --rm \
                        --name=matrix-static-files \
                        --log-driver=none \
                        --user=997:1002 \
                        --cap-drop=ALL \
                        --read-only \
                        --network=matrix-static-files \
                        --env-file=/matrix/static-files/env \
                        --label-file=/matrix/static-files/labels \
                        --mount type=bind,src=/matrix/static-files/public,dst=/public,ro \
                        --mount type=bind,src=/matrix/static-files/config,dst=/config,ro \
                        docker.io/joseluisq/static-web-server:2.24.2-alpine


ExecStart=/usr/bin/env docker start --attach matrix-static-files

ExecStop=-/usr/bin/env sh -c '/usr/bin/env docker stop --time=3 matrix-static-files 2>/dev/null || true'
ExecStop=-/usr/bin/env sh -c '/usr/bin/env docker rm matrix-static-files 2>/dev/null || true'
Restart=always
RestartSec=30
SyslogIdentifier=matrix-static-files

[Install]
WantedBy=multi-user.target

spantaleev · 2024-01-17T06:54:28Z

Someone had reported this in our Matrix room, so I've tried to fix this via da1f570 but it looks like it wasn't of much help.

Perhaps 17859ec is a new and better fix. Let me know!

ppkhoa · 2024-01-17T07:20:48Z

Latest commit mentioned above seems to be working for matrix-static-files.

My vars.yml is now this (excerpt, since most weren't changed):

...
matrix_playbook_ssl_enabled: true
matrix_playbook_reverse_proxy_type: 'other-traefik-container'
matrix_playbook_reverse_proxy_hostname: 'traefik'
matrix_playbook_reverse_proxy_container_network: 'traefik'

#no longer needed
#matrix_static_files_container_additional_networks: ['traefik']

matrix_playbook_reverse_proxyable_services_additional_network: 'traefik'
devture_traefik_certs_dumper_enabled: true
devture_traefik_certs_dumper_ssl_dir_path: "/opt/containers/traefik/data/"
devture_traefik_config_certificatesResolvers_acme_enabled: false
matrix_playbook_public_matrix_federation_api_traefik_entrypoint_port: 8449
matrix_playbook_public_matrix_federation_api_traefik_entrypoint_host_bind_port: 127.0.0.1:8449
matrix_playbook_public_matrix_federation_api_traefik_entrypoint_config_custom:
  forwardedHeaders:
    insecure: true
matrix_synapse_container_additional_networks: ['traefik', 'matrix-postgres', 'matrix-exim-relay'] # <-----------still have to do this manually
...

matrix-static-files.service:

[Unit]
Description=matrix-static-files server
Requires=docker.service
After=docker.service
DefaultDependencies=no

[Service]
Type=simple
Environment="HOME=/root"
ExecStartPre=-/usr/bin/env sh -c '/usr/bin/env docker stop --time=3 matrix-static-files 2>/dev/null || true'
ExecStartPre=-/usr/bin/env sh -c '/usr/bin/env docker rm matrix-static-files 2>/dev/null || true'

ExecStartPre=/usr/bin/env docker create \
                        --rm \
                        --name=matrix-static-files \
                        --log-driver=none \
                        --user=997:1002 \
                        --cap-drop=ALL \
                        --read-only \
                        --network=traefik \ <----------- this is correct
                        --env-file=/matrix/static-files/env \
                        --label-file=/matrix/static-files/labels \
                        --mount type=bind,src=/matrix/static-files/public,dst=/public,ro \
                        --mount type=bind,src=/matrix/static-files/config,dst=/config,ro \
                        docker.io/joseluisq/static-web-server:2.24.2-alpine


ExecStart=/usr/bin/env docker start --attach matrix-static-files

ExecStop=-/usr/bin/env sh -c '/usr/bin/env docker stop --time=3 matrix-static-files 2>/dev/null || true'
ExecStop=-/usr/bin/env sh -c '/usr/bin/env docker rm matrix-static-files 2>/dev/null || true'
Restart=always
RestartSec=30
SyslogIdentifier=matrix-static-files

[Install]
WantedBy=multi-user.target

spantaleev · 2024-01-17T07:33:20Z

Glad to hear it helped!

It'd be great to figure out why you still need to explicitly add matrix-postgres to matrix_synapse_container_additional_networks

ppkhoa · 2024-01-17T09:41:43Z

If you can tell me where to add the code to print out these variables, maybe we can find out why it's not adding the network:

devture_postgres_container_network
devture_postgres_enabled
devture_postgres_container_network
matrix_synapse_container_network
matrix_synapse_database_host
devture_postgres_connection_hostname

spantaleev · 2024-01-17T10:03:07Z

roles/custom/matrix-synapse/tasks/init.yml would be a good place to add various - debug: var="VARIABLE_NAME" lines, followed by a - fail: msg="Stop here" line.

ppkhoa · 2024-01-17T10:19:28Z

Looks like everything is correct there:

TASK [custom/matrix-synapse : debug] ****************************************
ok: [matrix.<mydomain>] =>
  devture_postgres_container_network: matrix-postgres

TASK [custom/matrix-synapse : debug] *****************************************
ok: [matrix.<mydomain>] =>
  'devture_postgres_enabled ': true

TASK [custom/matrix-synapse : debug] *****************************************
ok: [matrix.<mydomain>] =>
  devture_postgres_container_network: matrix-postgres

TASK [custom/matrix-synapse : debug] ****************************************
ok: [matrix.<mydomain>] =>
  matrix_synapse_container_network: matrix-homeserver

TASK [custom/matrix-synapse : debug] *******************************************
ok: [matrix.<mydomain>] =>
  matrix_synapse_database_host: matrix-postgres

TASK [custom/matrix-synapse : debug] *******************************************
ok: [matrix.<mydomain>] =>
  devture_postgres_connection_hostname: matrix-postgres

So this statement ([devture_postgres_container_network] if (devture_postgres_enabled and devture_postgres_container_network != matrix_synapse_container_network and matrix_synapse_database_host == devture_postgres_connection_hostname) else []) should be true (which is correct) and devture_postgres_container_network should be added to matrix-synapse container network but it wasn't...

ppkhoa · 2024-01-17T15:13:45Z

Closing this issue since it's not happening to anyone else. Just need to add 1 line to vars.yml for now.

…ther-traefik-container" setups Related to spantaleev#3095

ppkhoa changed the title ~~Cannot completely move to Traefik reverse-proxy~~ Cannot finish setup-all after recent commit (completely moved to Traefik reverse-proxy) Jan 15, 2024

spantaleev added a commit that referenced this issue Jan 15, 2024

Fix incorrect assumption for matrix_playbook_reverse_proxy_type == "o…

b1e08db

…ther-traefik-container" setups Related to #3095

ppkhoa closed this as completed Jan 17, 2024

ppkhoa reopened this Jan 17, 2024

ppkhoa closed this as completed Jan 17, 2024

ewonchang mentioned this issue Feb 7, 2024

Cannot run --tags=start after a recent commit #3161

Closed

KarolosLykos pushed a commit to KarolosLykos/matrix-docker-ansible-deploy that referenced this issue Mar 5, 2024

Fix incorrect assumption for matrix_playbook_reverse_proxy_type == "o…

fc603bb

…ther-traefik-container" setups Related to spantaleev#3095

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot finish setup-all after recent commit (completely moved to Traefik reverse-proxy) #3095

Cannot finish setup-all after recent commit (completely moved to Traefik reverse-proxy) #3095

ppkhoa commented Jan 15, 2024 •

edited

Loading

spantaleev commented Jan 15, 2024

spantaleev commented Jan 15, 2024

ppkhoa commented Jan 15, 2024 •

edited

Loading

spantaleev commented Jan 15, 2024

ppkhoa commented Jan 15, 2024 •

edited

Loading

ppkhoa commented Jan 15, 2024 •

edited

Loading

spantaleev commented Jan 15, 2024

ppkhoa commented Jan 15, 2024 •

edited

Loading

ppkhoa commented Jan 17, 2024 •

edited

Loading

ppkhoa commented Jan 17, 2024 •

edited

Loading

spantaleev commented Jan 17, 2024

ppkhoa commented Jan 17, 2024

spantaleev commented Jan 17, 2024

ppkhoa commented Jan 17, 2024

spantaleev commented Jan 17, 2024

ppkhoa commented Jan 17, 2024 •

edited

Loading

ppkhoa commented Jan 17, 2024

Cannot finish setup-all after recent commit (completely moved to Traefik reverse-proxy) #3095

Cannot finish setup-all after recent commit (completely moved to Traefik reverse-proxy) #3095

Comments

ppkhoa commented Jan 15, 2024 • edited Loading

spantaleev commented Jan 15, 2024

spantaleev commented Jan 15, 2024

ppkhoa commented Jan 15, 2024 • edited Loading

spantaleev commented Jan 15, 2024

ppkhoa commented Jan 15, 2024 • edited Loading

ppkhoa commented Jan 15, 2024 • edited Loading

spantaleev commented Jan 15, 2024

ppkhoa commented Jan 15, 2024 • edited Loading

ppkhoa commented Jan 17, 2024 • edited Loading

ppkhoa commented Jan 17, 2024 • edited Loading

spantaleev commented Jan 17, 2024

ppkhoa commented Jan 17, 2024

spantaleev commented Jan 17, 2024

ppkhoa commented Jan 17, 2024

spantaleev commented Jan 17, 2024

ppkhoa commented Jan 17, 2024 • edited Loading

ppkhoa commented Jan 17, 2024

ppkhoa commented Jan 15, 2024 •

edited

Loading

ppkhoa commented Jan 15, 2024 •

edited

Loading

ppkhoa commented Jan 15, 2024 •

edited

Loading

ppkhoa commented Jan 15, 2024 •

edited

Loading

ppkhoa commented Jan 15, 2024 •

edited

Loading

ppkhoa commented Jan 17, 2024 •

edited

Loading

ppkhoa commented Jan 17, 2024 •

edited

Loading

ppkhoa commented Jan 17, 2024 •

edited

Loading