Merge branch 'master' into HarHarLinks/hookshot-encryption

.github/renovate.json

@@ -10,5 +10,15 @@
 				"# renovate: datasource=(?<datasource>[a-z-.]+?) depName=(?<depName>[^\\s]+?)(?: (?:lookupName|packageName)=(?<packageName>[^\\s]+?))?(?: versioning=(?<versioning>[a-z-0-9]+?))?\\s+[A-Za-z0-9_]+?(?:_version|_tag)\\s*:\\s*[\"']?(?<currentValue>.+?)[\"']?\\s"
 			]
 		}
+	],
+	"packageRules": [
+		{
+			"matchSourceUrlPrefixes": [
+				"https://github.com/devture/com.devture.ansible.role",
+				"https://gitlab.com/etke.cc/roles",
+				"https://github.com/mother-of-all-self-hosting"
+			],
+			"ignoreUnstable": false
+		}
 	]
 }

.github/workflows/matrix.yml

@@ -13,7 +13,7 @@ jobs:
       - name: Check out
         uses: actions/checkout@v4
       - name: Run yamllint
-        uses: frenck/action-yamllint@v1.4.1
+        uses: frenck/action-yamllint@v1.4.2
   ansible-lint:
     name: ansible-lint
     runs-on: ubuntu-latest

README.md

@@ -17,7 +17,7 @@ We run all services in [Docker](https://www.docker.com/) containers (see [the co
 
 This Ansible playbook tries to make self-hosting and maintaining a Matrix server fairly easy. Still, running any service smoothly requires knowledge, time and effort.
 
-If you like the [FOSS](https://en.wikipedia.org/wiki/Free_and_open-source_software) spirit of this Ansible playbook, but prefer to put the responsibility on someone else, you can also [get a managed Matrix server from etke.cc](https://etke.cc/) - a service built on top of this Ansible playbook, which can help you run a Matrix server with ease.
+If you like the [FOSS](https://en.wikipedia.org/wiki/Free_and_open-source_software) spirit of this Ansible playbook, but prefer to put the responsibility on someone else, you can also [get a managed Matrix server from etke.cc](https://etke.cc?utm_source=github&utm_medium=readme&utm_campaign=mdad) - a service built on top of this Ansible playbook, which can help you run a Matrix server with ease.
 
 If you like learning and experimentation, but would rather reduce future maintenance effort, you can even go for a hybrid approach - self-hosting manually using this Ansible playbook at first and then transferring server maintenance to etke.cc at a later time.
 
@@ -197,14 +197,6 @@ When updating the playbook, refer to [the changelog](CHANGELOG.md) to catch up w
 
 ## Related
 
-You may also be interested in these other Ansible playbooks:
+You may also be interested in [mash-playbook](https://github.com/mother-of-all-self-hosting/mash-playbook) - another Ansible playbook for self-hosting non-Matrix services (see its [List of supported services](https://github.com/mother-of-all-self-hosting/mash-playbook/blob/main/docs/supported-services.md)).
 
-- [gitea-docker-ansible-deploy](https://github.com/spantaleev/gitea-docker-ansible-deploy) - for deploying a [Gitea](https://gitea.io/) git version-control server
-
-- [nextcloud-docker-ansible-deploy](https://github.com/spantaleev/nextcloud-docker-ansible-deploy) - for deploying a [Nextcloud](https://nextcloud.com/) server
-
-- [peertube-docker-ansible-deploy](https://github.com/spantaleev/peertube-docker-ansible-deploy) - for deploying a [PeerTube](https://joinpeertube.org/) video-platform server
-
-- [vaultwarden-docker-ansible-deploy](https://github.com/spantaleev/vaultwarden-docker-ansible-deploy) - for deploying a [Vaultwarden](https://github.com/dani-garcia/vaultwarden) password manager server (unofficial [Bitwarden](https://bitwarden.com/) compatible server)
-
-They're all making use of Traefik as their reverse-proxy, so it should be easy to host all these services on the same server. Follow the `docs/configuring-playbook-interoperability.md` documentation in each playbook.
+mash-playbook also makes use of [Traefik](./docs/configuring-playbook-traefik.md) as its reverse-proxy, so with minor [interoperability adjustments](https://github.com/mother-of-all-self-hosting/mash-playbook/blob/main/docs/interoperability.md), you can make matrix-docker-ansible-deploy and mash-playbook co-exist and host Matrix and non-Matrix services on the same server.

docs/configuring-playbook-bridge-beeper-linkedin.md

@@ -32,14 +32,10 @@ You may wish to look at `roles/custom/matrix-bridge-beeper-linkedin/templates/co
 
 ## Set up Double Puppeting
 
-If you'd like to use [Double Puppeting](https://docs.mau.fi/bridges/general/double-puppeting.html) (hint: you most likely do), you have 2 ways of going about it.
-
-### Method 1: automatically, by enabling Shared Secret Auth
+If you'd like to use [Double Puppeting](https://docs.mau.fi/bridges/general/double-puppeting.html) (hint: you most likely do), you have to enable Shared Secred Auth.
 
 The bridge will automatically perform Double Puppeting if you enable [Shared Secret Auth](configuring-playbook-shared-secret-auth.md) for this playbook.
 
-This is the recommended way of setting up Double Puppeting, as it's easier to accomplish, works for all your users automatically, and has less of a chance of breaking in the future.
-
 
 ## Usage
 

docs/configuring-playbook-bridge-mautrix-whatsapp.md

@@ -21,8 +21,8 @@ By default, only admins are allowed to set themselves as relay users. To allow a
 matrix_mautrix_whatsapp_bridge_relay_admin_only: false
 ```
 
-If you want to activate the relay bot in a room, use `!whatsapp set-relay`.
-Use `!whatsapp unset-relay` to deactivate.
+If you want to activate the relay bot in a room, use `!wa set-relay`.
+Use `!wa unset-relay` to deactivate.
 
 ## Enable backfilling history
 This requires a server with MSC2716 support, which is currently an experimental feature in synapse.

docs/configuring-playbook-matrix-media-repo.md

@@ -1,14 +1,20 @@
 # Setting up matrix-media-repo (optional)
 
-[matrix-media-repo](https://docs.t2bot.io/matrix-media-repo/) is a highly customizable multi-domain media repository for Matrix. Intended for medium to large environments consisting of several homeservers, this media repo de-duplicates media (including remote media) while being fully compliant with the specification.
+[matrix-media-repo](https://docs.t2bot.io/matrix-media-repo/) (often abbreviated "MMR") is a highly customizable multi-domain media repository for Matrix. Intended for medium to large environments consisting of several homeservers, this media repo de-duplicates media (including remote media) while being fully compliant with the specification.
 
 Smaller/individual homeservers can still make use of this project's features, though it may be difficult to set up or have higher than expected resource consumption. Please do your research before deploying this as this project may not be useful for your environment.
 
 For a simpler alternative (which allows you to offload your media repository storage to S3, etc.), you can [configure S3 storage](configuring-playbook-s3.md) instead of setting up matrix-media-repo.
 
+| **Table of Contents**                                                                       |
+| :------------------------------------------------------------------------------------------ |
+| [Quickstart](#quickstart)                                                                   |
+| [Additional configuration options](#configuring-the-media-repo)                             |
+| [Importing data from an existing media store](#importing-data-from-an-existing-media-store) |
+
 ## Quickstart
 
-Add the following configuration to your `inventory/host_vars/matrix.DOMAIN/vars.yml` file:
+Add the following configuration to your `inventory/host_vars/matrix.DOMAIN/vars.yml` file and [re-run the installation process](./installing.md) for the playbook:
 
 ```yaml
 matrix_media_repo_enabled: true
@@ -37,8 +43,9 @@ matrix_media_repo_database_max_connections: 25
 matrix_media_repo_database_max_idle_connections: 5
 
 # These users have full access to the administrative functions of the media repository.
-# See https://github.com/turt2live/matrix-media-repo/blob/release-v1.2.8/docs/admin.md for information on what these people can do. They must belong to one of the
-# configured homeservers above.
+# See https://github.com/turt2live/matrix-media-repo/blob/release-v1.2.8/docs/admin.md for 
+# information on what these people can do. They must belong to one of the configured 
+# homeservers above.
 matrix_media_repo_admins:
   admins: []
 # admins:
@@ -102,5 +109,56 @@ matrix_media_repo_datastores:
 
 ```
 
-Full list of configuration options with documentation can be found in `roles/custom/matrix-media-repo/templates/defaults/main.yml`
+Full list of configuration options with documentation can be found in [`roles/custom/matrix-media-repo/defaults/main.yml`](https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/roles/custom/matrix-media-repo/defaults/main.yml)
+
+## Importing data from an existing media store
+
+If you want to add this repo to an existing homeserver managed by the playbook, you will need to import existing media into MMR's database or you will lose access to older media while it is active. MMR versions up to `v1.3.3` only support importing from Synapse, but newer versions (at time of writing: only `latest`) also support importing from Dendrite.
+
+**Before importing**: ensure you have an initial matrix-media-repo deployment by following the [quickstart](#quickstart) guide above
+
+Depending on the homeserver implementation yu're using (Synapse, Dendrite), you'll need to use a different import tool (part of matrix-media-repo) and point it to the homeserver's database.
+
+### Importing data from the Synapse media store
+
+To import the Synapse media store, you're supposed to invoke the `import_synapse` tool which is part of the matrix-media-repo container image. Your Synapse database is called `synapse` by default, unless you've changed it by modifying `matrix_synapse_database_database`.
+
+This guide here is adapted from the [upstream documentation about the import_synapse script](https://github.com/turt2live/matrix-media-repo#importing-media-from-synapse).
+
+Run the following command on the server (after replacing `devture_postgres_connection_password` in it with the value found in your `vars.yml` file):
+
+```sh
+docker exec -it matrix-media-repo \
+    /usr/local/bin/import_synapse \
+        -dbName synapse \
+        -dbHost matrix-postgres \
+        -dbPort 5432 \
+        -dbUsername matrix \
+        -dbPassword devture_postgres_connection_password
+```
+
+Enter `1` for the Machine ID when prompted (you are not doing any horizontal scaling) unless you know what you're doing.
+
+This should output a `msg="Import completed"` when finished successfully!
+
+### Importing data from the Dendrite media store
+
+If you're using the [Dendrite](configuring-playbook-dendrite.md) homeserver instead of the default for this playbook (Synapse), follow this importing guide here.
+
+To import the Dendrite media store, you're supposed to invoke the `import_dendrite` tool which is part of the matrix-media-repo container image. Your Dendrite database is called `dendrite_mediaapi` by default, unless you've changed it by modifying `matrix_dendrite_media_api_database`.
+
+Run the following command on the server (after replacing `devture_postgres_connection_password` in it with the value found in your `vars.yml` file):
+
+```sh
+docker exec -it matrix-media-repo \
+    /usr/local/bin/import_dendrite \
+        -dbName dendrite_mediaapi \
+        -dbHost matrix-postgres \
+        -dbPort 5432 \
+        -dbUsername matrix \
+        -dbPassword devture_postgres_connection_password
+```
+
+Enter `1` for the Machine ID when prompted (you are not doing any horizontal scaling) unless you know what you're doing.
 
+This should output a `msg="Import completed"` when finished successfully!

docs/configuring-playbook-sliding-sync-proxy.md

@@ -8,7 +8,7 @@ See the project's [documentation](https://github.com/matrix-org/sliding-sync) to
 
 Element X iOS is [available on TestFlight](https://testflight.apple.com/join/uZbeZCOi).
 
-Element X Android requires manual compilation to get it working with a non-`matrix.org` homeseserver. It's also less feature-complete than the iOS version.
+Element X Android is [available on the Github Releases page](https://github.com/vector-im/element-x-android/releases).
 
 **NOTE**: The Sliding Sync proxy **only works with the Traefik reverse-proxy**. If you have an old server installation (from the time `matrix-nginx-proxy` was our default reverse-proxy - `matrix_playbook_reverse_proxy_type: playbook-managed-nginx`), you won't be able to use Sliding Sync.
 

docs/configuring-playbook-ssl-certificates.md

@@ -68,21 +68,21 @@ aux_file_definitions:
   # uploading a file from the computer where Ansible is running.
   - dest: "{{ devture_traefik_ssl_dir_path }}/privkey.pem"
     src: /path/on/your/Ansible/computer/to/privkey.pem
-	# Alternatively, comment out `src` above and uncomment the lines below to provide the certificate content inline.
-	# Note the indentation level.
-	# content: |
-	#   FILE CONTENT
-	#   HERE
+    # Alternatively, comment out `src` above and uncomment the lines below to provide the certificate content inline.
+    # Note the indentation level.
+    # content: |
+    #   FILE CONTENT
+    #   HERE
 
   # Create the cert.pem file on the server
   # uploading a file from the computer where Ansible is running.
   - dest: "{{ devture_traefik_ssl_dir_path }}/cert.pem"
     src: /path/on/your/Ansible/computer/to/cert.pem
-	# Alternatively, comment out `src` above and uncomment the lines below to provide the certificate content inline.
-	# Note the indentation level.
-	# content: |
-	#   FILE CONTENT
-	#   HERE
+    # Alternatively, comment out `src` above and uncomment the lines below to provide the certificate content inline.
+    # Note the indentation level.
+    # content: |
+    #   FILE CONTENT
+    #   HERE
 
   # Create the custom Traefik configuration.
   # The `/ssl/..` paths below are in-container paths, not paths on the host (/`matrix/traefik/ssl/..`). Do not change them!

group_vars/matrix_servers

@@ -2101,7 +2101,7 @@ backup_borg_gid: "{{ matrix_user_gid }}"
 
 backup_borg_container_network: "{{ devture_postgres_container_network if devture_postgres_enabled else backup_borg_identifier }}"
 
-backup_borg_postgresql_version_detection_devture_postgres_role_name: "{{ 'galaxy/com.devture.ansible.role.postgres' if devture_postgres_enabled else '' }}"
+backup_borg_postgresql_version_detection_devture_postgres_role_name: "{{ 'galaxy/postgres' if devture_postgres_enabled else '' }}"
 
 backup_borg_container_image_self_build: "{{ matrix_architecture not in ['amd64', 'arm32', 'arm64'] }}"
 
@@ -3272,6 +3272,7 @@ devture_postgres_backup_connection_username: "{{ devture_postgres_connection_use
 devture_postgres_backup_connection_password: "{{ devture_postgres_connection_password if devture_postgres_enabled else '' }}"
 
 devture_postgres_backup_postgres_data_path: "{{ devture_postgres_data_path if devture_postgres_enabled else '' }}"
+devture_postgres_backup_postgres_role_include_name: galaxy/postgres
 
 devture_postgres_backup_databases: "{{ devture_postgres_managed_databases | map(attribute='name') if devture_postgres_enabled else [] }}"
 
@@ -4303,9 +4304,6 @@ matrix_user_creator_users_auto: |
 #
 ######################################################################
 
-## FIXME: Needs to be updated when there is a proper release by upstream.
-matrix_user_verification_service_docker_image: "{{ matrix_user_verification_service_docker_image_name_prefix }}matrixdotorg/matrix-user-verification-service@sha256:d2aabc984dd69d258c91900c36928972d7aaef19d776caa3cd6a0fbc0e307270"
-
 matrix_user_verification_service_enabled: false
 matrix_user_verification_service_systemd_required_services_list: |
   {{
@@ -4399,7 +4397,7 @@ devture_traefik_additional_domains_to_obtain_certificates_for: "{{ matrix_ssl_ad
 
 devture_traefik_config_providers_docker_endpoint: "{{ devture_container_socket_proxy_endpoint if devture_container_socket_proxy_enabled else 'unix:///var/run/docker.sock' }}"
 
-devture_traefik_container_additional_networks: |
+devture_traefik_container_additional_networks_auto: |
   {{
     ([devture_container_socket_proxy_container_network] if devture_container_socket_proxy_enabled else [])
   }}

requirements.yml

@@ -1,54 +1,71 @@
 ---
 
 - src: git+https://github.com/mother-of-all-self-hosting/ansible-role-aux.git
-  version: v1.0.0-1
+  version: v1.0.0-3
   name: auxiliary
 - src: git+https://gitlab.com/etke.cc/roles/backup_borg.git
-  version: v1.2.6-1.8.4-0
+  version: v1.2.7-1.8.5-2
+  name: backup_borg
 - src: git+https://github.com/devture/com.devture.ansible.role.container_socket_proxy.git
-  version: v0.1.1-2
+  version: v0.1.1-3
+  name: container_socket_proxy
+- src: git+https://github.com/geerlingguy/ansible-role-docker
+  version: 7.0.2
+  name: docker
 - src: git+https://github.com/devture/com.devture.ansible.role.docker_sdk_for_python.git
   version: 129c8590e106b83e6f4c259649a613c6279e937a
+  name: docker_sdk_for_python
+- src: git+https://gitlab.com/etke.cc/roles/etherpad.git
+  version: v1.9.5-1
+  name: etherpad
+- src: git+https://gitlab.com/etke.cc/roles/grafana.git
+  version: v10.2.2-1
+  name: grafana
+- src: git+https://github.com/mother-of-all-self-hosting/ansible-role-jitsi.git
+  version: v9111-1
+  name: jitsi
+- src: git+https://gitlab.com/etke.cc/roles/ntfy.git
+  version: v2.8.0-1
+  name: ntfy
 - src: git+https://github.com/devture/com.devture.ansible.role.playbook_help.git
   version: c1f40e82b4d6b072b6f0e885239322bdaaaf554f
+  name: playbook_help
 - src: git+https://github.com/devture/com.devture.ansible.role.playbook_runtime_messages.git
   version: 9b4b088c62b528b73a9a7c93d3109b091dd42ec6
+  name: playbook_runtime_messages
 - src: git+https://github.com/devture/com.devture.ansible.role.playbook_state_preserver.git
   version: ff2fd42e1c1a9e28e3312bbd725395f9c2fc7f16
+  name: playbook_state_preserver
 - src: git+https://github.com/devture/com.devture.ansible.role.postgres.git
-  version: v16.0-8
+  version: v16.1-3
+  name: postgres
 - src: git+https://github.com/devture/com.devture.ansible.role.postgres_backup.git
-  version: a0cc7c1c696872ba8880d9c5e5a54098de825030
-- src: git+https://github.com/devture/com.devture.ansible.role.systemd_docker_base.git
-  version: v1.0.0-0
-- src: git+https://github.com/devture/com.devture.ansible.role.systemd_service_manager.git
-  version: v1.0.0-1
-- src: git+https://github.com/devture/com.devture.ansible.role.timesync.git
-  version: v1.0.0-0
-- src: git+https://github.com/devture/com.devture.ansible.role.traefik.git
-  version: v2.10.5-0
-- src: git+https://github.com/devture/com.devture.ansible.role.traefik_certs_dumper.git
-  version: v2.8.1-0
-- src: git+https://gitlab.com/etke.cc/roles/etherpad.git
-  version: v1.9.3-0
-- src: git+https://github.com/geerlingguy/ansible-role-docker
-  version: 7.0.1
-  name: geerlingguy.docker
-- src: git+https://gitlab.com/etke.cc/roles/grafana.git
-  version: v10.2.0-0
-- src: git+https://github.com/mother-of-all-self-hosting/ansible-role-jitsi.git
-  version: v8960-3
-  name: jitsi
-- src: git+https://gitlab.com/etke.cc/roles/ntfy.git
-  version: v2.7.0-2
+  version: b29a9c551dd09079f5ef26d494973a499088b9e8
+  name: postgres_backup
 - src: git+https://github.com/mother-of-all-self-hosting/ansible-role-prometheus.git
-  version: v2.47.2-0
+  version: v2.48.1-0
   name: prometheus
 - src: git+https://github.com/mother-of-all-self-hosting/ansible-role-prometheus-node-exporter.git
-  version: v1.6.1-0
+  version: v1.7.0-1
   name: prometheus_node_exporter
 - src: git+https://github.com/mother-of-all-self-hosting/ansible-role-prometheus-postgres-exporter.git
-  version: v0.14.0-0
+  version: v0.14.0-1
   name: prometheus_postgres_exporter
 - src: git+https://gitlab.com/etke.cc/roles/redis.git
-  version: v7.2.0-0
+  version: v7.2.3-2
+  name: redis
+- src: git+https://github.com/devture/com.devture.ansible.role.systemd_docker_base.git
+  version: v1.0.0-2
+  name: systemd_docker_base
+- src: git+https://github.com/devture/com.devture.ansible.role.systemd_service_manager.git
+  version: v1.0.0-3
+  name: systemd_service_manager
+- src: git+https://github.com/devture/com.devture.ansible.role.timesync.git
+  version: v1.0.0-0
+  name: timesync
+- src: git+https://github.com/devture/com.devture.ansible.role.traefik.git
+  version: v2.10.7-0
+  name: traefik
+- src: git+https://github.com/devture/com.devture.ansible.role.traefik_certs_dumper.git
+  version: v2.8.3-1
+  name: traefik_certs_dumper