synapse v1.62.0-v1.67.0 fails to create valid homeserver.yaml

[qknight]

Description

generate creates incomplete homeserver.yaml

v1.61.0

docker run -it --rm --mount type=volume,src=ts-matrix-compose_synapse,dst=/data -e SYNAPSE_SERVER_NAME=my.matrix.host -e SYNAPSE_REPORT_STATS=yes matrixdotorg/synapse:v1.61.0 generate
Setting ownership on /data to 991:991
Creating log config /data/my.matrix.host.log.config
Generating config file /data/homeserver.yaml
Generating signing key file /data/my.matrix.host.signing.key

This creates:

/data # cat homeserver.yaml | sed -E '/^$|^\s*#/d'
modules:
server_name: "my.matrix.host"
pid_file: /data/homeserver.pid
presence:
listeners:
  - port: 8008
    tls: false
    type: http
    x_forwarded: true
    resources:
      - names: [client, federation]
        compress: false
manhole_settings:
limit_remote_rooms:
templates:
retention:
caches:
  per_cache_factors:
database:
  name: sqlite3
  args:
    database: /data/homeserver.db
log_config: "/data/my.matrix.host.log.config"
media_store_path: "/data/media_store"
url_preview_accept_language:
oembed:
registration_shared_secret: "h;TP-&BZ^J;,K0fNsyK8yV^tV45L9=1IcXURN+Fo=xcj+7vrm9"
account_threepid_delegates:
metrics_flags:
report_stats: true
room_prejoin_state:
macaroon_secret_key: "VFyUQOK5&iEyaNdpk5@xai6@feMpGfi6Oswr#bY8Kb7b,YZrYs"
form_secret: "^ot3eBpaj9txWuBea6,GLRF7lcQ@Rwq2vIv:T:GrXqvHHSIcQy"
signing_key_path: "/data/my.matrix.host.signing.key"
old_signing_keys:
trusted_key_servers:
  - server_name: "matrix.org"
saml2_config:
  sp_config:
  user_mapping_provider:
    config:
oidc_providers:
cas_config:
sso:
password_config:
   policy:
ui_auth:
email:
push:
user_directory:
stats:
opentracing:
redis:
background_updates:

v1.62.0

docker run -it --rm --mount type=volume,src=ts-matrix-compose_synapse,dst=/data -e SYNAPSE_SERVER_NAME=my.matrix.host -e SYNAPSE_REPORT_STATS=yes matrixdotorg/synapse:v1.62.0 generate
Setting ownership on /data to 991:991
Creating log config /data/my.matrix.host.log.config
Generating config file /data/homeserver.yaml
Generating signing key file /data/my.matrix.host.signing.key

This creates:

/data # rm *
/data # cat homeserver.yaml
# Configuration file for Synapse.
#
# This is a YAML file: see [1] for a quick introduction. Note in particular
# that *indentation is important*: all the elements of a list or dictionary
# should have the same indentation.
#
# [1] https://docs.ansible.com/ansible/latest/reference_appendices/YAMLSyntax.html
#
# For more information on how to configure Synapse, including a complete accounting of
# each option, go to docs/usage/configuration/config_documentation.md or
# https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html
server_name: "my.matrix.host"
pid_file: /data/homeserver.pid
listeners:
  - port: 8008
    tls: false
    type: http
    x_forwarded: true
    resources:
      - names: [client, federation]
        compress: false
database:
  name: sqlite3
  args:
    database: /data/homeserver.db
log_config: "/data/my.matrix.host.log.config"
media_store_path: /data/media_store
registration_shared_secret: ".Ph+z5bR@u4;80GnKn~TB:J~4nLWAo6+n4X8JUlD#pLeIrTjmi"
report_stats: true
macaroon_secret_key: "Fa*,P2B2iU.AN:yUhnK.GXw2Aj~9pS2oC:HCSTdv#,ZVx:tiD^"
form_secret: "H+VSn:Z4jaqOaWnEfz^bm~*t24gHWzxDRXdwphzqj,sB=B3IIF"
signing_key_path: "/data/my.matrix.host.signing.key"
trusted_key_servers:
  - server_name: "matrix.org"

Note: This does not include the report_stats as it did before.

Steps to reproduce

docker run -it --rm --mount type=volume,src=ts-matrix-compose_synapse,dst=/data -e SYNAPSE_SERVER_NAME=my.matrix.host -e SYNAPSE_REPORT_STATS=yes matrixdotorg/synapse:v1.61.0 generate

vs.

docker run -it --rm --mount type=volume,src=ts-matrix-compose_synapse,dst=/data -e SYNAPSE_SERVER_NAME=my.matrix.host -e SYNAPSE_REPORT_STATS=yes matrixdotorg/synapse:v1.62.0 generate

In both cases compare the output:

docker run -it --rm --mount type=volume,src=ts-matrix-compose_synapse,dst=/data alpine
cd data
cat homeserver.yaml
# for the next run, clear the data folder!
rm *

Homeserver

self hosted

Synapse Version

v1.62.0

Installation Method

Docker (matrixdotorg/synapse)

Platform

Using docker builds as described in the docker/Dockerfile

Relevant log output

All included above.

Anything else that would be useful to know?

Can't use cherry-pick to identify the commit which has been used as I can't rebuild v1.61.0 up to ~v1.65.0, see
#11537 (comment)

[H-Shay]

This is due to a rather controversial change to remove the comments from the configuration generation code, as it was felt that the 2000+ line yaml file was not the best way to document the configuration options. You can read more about the reasoning for this change here: #8159.

[richvdh]

I'm intrigued. In what way is the generated homeserver.yaml not "valid"?