Synapse won't start if no package metadata is found for "matrix-synapse"

[MTRNord]

Description

The workers all fail wit this error while the main process works:

pr 27 15:06:08 nordgedanken.dev systemd[1]: Starting Synapse background_worker...
Apr 27 15:06:09 nordgedanken.dev matrix-synapse-background_worker[186827]: Traceback (most recent call last):
Apr 27 15:06:09 nordgedanken.dev matrix-synapse-background_worker[186827]:   File "/usr/lib/python3.9/runpy.py", line 188, in _run_module_as_main
Apr 27 15:06:09 nordgedanken.dev matrix-synapse-background_worker[186827]:     mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
Apr 27 15:06:09 nordgedanken.dev matrix-synapse-background_worker[186827]:   File "/usr/lib/python3.9/runpy.py", line 111, in _get_module_details
Apr 27 15:06:09 nordgedanken.dev matrix-synapse-background_worker[186827]:     __import__(pkg_name)
Apr 27 15:06:09 nordgedanken.dev matrix-synapse-background_worker[186827]:   File "/usr/lib/python3.9/site-packages/synapse/app/__init__.py", line 18, in <module>
Apr 27 15:06:09 nordgedanken.dev matrix-synapse-background_worker[186827]:     from synapse.util import check_dependencies
Apr 27 15:06:09 nordgedanken.dev matrix-synapse-background_worker[186827]:   File "/usr/lib/python3.9/site-packages/synapse/util/check_dependencies.py", line 58, in <module>
Apr 27 15:06:09 nordgedanken.dev matrix-synapse-background_worker[186827]:     set(metadata.metadata(DISTRIBUTION_NAME).get_all("Provides-Extra")) - DEV_EXTRAS
Apr 27 15:06:09 nordgedanken.dev matrix-synapse-background_worker[186827]: TypeError: 'NoneType' object is not iterable

Steps to reproduce

• Install synapse?

Version information

• Homeserver:

If not matrix.org:

• Version:v1.58.0rc2

• Install method:
  gentoo ebuild

• Platform:
  Gentoo "bare-metal"/on host

[DMRobertson]

The workers all fail wit this error while the main process works:

Is there a difference between how the main process and the workers are started?

[DMRobertson]

Sounds like it's related to #12223, except this time it applies to the matrix-synapse package rather than a dependency.

@MTRNord what does ls /usr/lib/python3.9/site-packages/matrix* yield?

[MTRNord]

This:

➜  matrix-synapse git:(main) ✗ ls /usr/lib/python3.9/site-packages/matrix* 
/usr/lib/python3.9/site-packages/matrix_common:
__init__.py  __pycache__  py.typed  regex.py  versionstring.py

/usr/lib/python3.9/site-packages/matrix_common-1.1.0-py3.9.egg-info:
dependency_links.txt  not-zip-safe  PKG-INFO  requires.txt  SOURCES.txt  top_level.txt

/usr/lib/python3.9/site-packages/matrix_synapse-1.58.0rc2-py3.9.egg-info:
dependency_links.txt  entry_points.txt  not-zip-safe  PKG-INFO  SOURCES.txt  top_level.txt

[MTRNord]

The workers all fail wit this error while the main process works:

Is there a difference between how the main process and the workers are started?

there shouldnt be. I am just using the systemd target based setup from the contrib folder. didnt check if that changed though since I did set it up.

(well it moved to https://github.com/matrix-org/synapse/tree/develop/docs/systemd-with-workers/system it seems)

[MTRNord]

Just to make sure the start command for workers is python -m synapse.app.generic_worker and the main process python -m synapse.app.homeserver. rest is the same on both.

[DMRobertson]

I have no idea how Synapse is installed in the gentoo ecosystem, but it looks like matrix_synapse-1.58.0rc2-py3.9.egg-info doesn't contain a METADATA file.

To confirm this: what happens if you run the following in the same environment that synapse would run in?

import importlib.metadata as m
d = m.distribution("matrix-synapse")
print(d)
print(d.metadata)

I'm really surprised that you don't see this on the main process, because AFAICS they should both import synapse.util.check_dependencies. This logic was present in Synapse 1.57 too, so I suspect this is a packaging artifact in the light of the poetry changes, rather than a Synapse bug per-se.

[deepbluev7]

@MTRNord, your ebuild is broken. Don't use the deprecated method of converting poetry to a setup.py, use this: https://gitlab.com/deepbluev7/neko/-/blob/master/net-im/matrix-synapse/matrix-synapse-1.58.0_rc2.ebuild#L7

[MTRNord]

@MTRNord, your ebuild is broken. Don't use the deprecated method of converting poetry to a setup.py, use this: https://gitlab.com/deepbluev7/neko/-/blob/master/net-im/matrix-synapse/matrix-synapse-1.58.0_rc2.ebuild#L7

Ah oh. Google suggested the line 6 of yours but not line 7. :o yeah that seems like the cause.

[DMRobertson]

I have no idea how Synapse is installed in the gentoo ecosystem, but it looks like matrix_synapse-1.58.0rc2-py3.9.egg-info doesn't contain a METADATA file.

Having said that, there is a PKG-INFO file which should also be considered as a metadata source, judging by the source code of importlib.metadata. I'd suggest looking at that file to see if its contents are sensible.

I'm reluctant to disable the checks that the dependencies have appropriate versions, because they're meant to help admins who've installed from source but missed necessary dependency upgrades. If the proposed ebuild fix doesn't work then maybe we can have some kind of opt-out mechanism, but I'd like to see if it's possible to fix the packaging first.

[DMRobertson]

(Could do something like the change proposed in bdb00ee)

[MTRNord]

I have no idea how Synapse is installed in the gentoo ecosystem, but it looks like matrix_synapse-1.58.0rc2-py3.9.egg-info doesn't contain a METADATA file.

Having said that, there is a PKG-INFO file which should also be considered as a metadata source, judging by the source code of importlib.metadata. I'd suggest looking at that file to see if its contents are sensible.

I'm reluctant to disable the checks that the dependencies have appropriate versions, because they're meant to help admins who've installed from source but missed necessary dependency upgrades. If the proposed ebuild fix doesn't work then maybe we can have some kind of opt-out mechanism, but I'd like to see if it's possible to fix the packaging first.

The issue indeed was the packaging. Nico helped me to get it working. :) Issue is fixed with that.

[DMRobertson]

Great stuff, thanks for confirming @MTRNord!