/sync hangs on synapse 0.33.3 with Twisted 16.6

[richvdh]

we've had a number of reports of people's clients hanging after upgrading to synapse 0.33.3.

It appears that the body of the response to /sync is not being correctly sent. It is limited to requests with large responses (more than, say, 64K, hence it affecting /sync), and only manifests itself when the client asks for a compressed body.

It also only happens on Twisted versions before 17.1.

[richvdh]

I think it may have been incidentally fixed by twisted/twisted#591 (certainly that seems to have rewritten a bunch of this code).

[richvdh]

For those who are being bitten by this bug: your best bet for now is probably to downgrade.

Note that you will need to manually carry out some database manipulation to downgrade.

The debs for 0.33.2 are still in the repo at https://matrix.org/packages/debian/pool/main/m/matrix-synapse/matrix-synapse_0.33.2-1_all.deb; download it and install with dpkg -i matrix-synapse_0.33.2-1_all.deb.

Before 0,33.2 will start, you will need to roll back the database schema upgrade that will have been done when you upgraded. This can be done by obtaining a sql shell on your database and:

update schema_version set version=50;
delete from applied_schema_deltas where version=51;
drop table monthly_active_users;

[richvdh]

FWIW the incompatibility with Twisted 16.6 was introduced by #3701.

[xsteadfastx]

i think i run into this with my own written bot with matrix-client in python. since 0.33.3 i get broken json:

> Aug 23 09:11:54 foobar pipenv[4205]:     Traceback (most recent call last):
> Aug 23 09:11:54 foobar pipenv[4205]:       File "mossbot.py", line 481, in connect
> Aug 23 09:11:54 foobar pipenv[4205]:         self.password
> Aug 23 09:11:54 foobar pipenv[4205]:       File "/opt/mossbot/.local/share/virtualenvs/mossbot-B1iKwa68/lib/python3.6/site-packages/matrix_client/client.py", line 183, in login_with_password
> Aug 23 09:11:54 foobar pipenv[4205]:         self._sync()
> Aug 23 09:11:54 foobar pipenv[4205]:       File "/opt/mossbot/.local/share/virtualenvs/mossbot-B1iKwa68/lib/python3.6/site-packages/matrix_client/client.py", line 426, in _sync
> Aug 23 09:11:54 foobar pipenv[4205]:         response = self.api.sync(self.sync_token, timeout_ms, filter=self.sync_filter)
> Aug 23 09:11:54 foobar pipenv[4205]:       File "/opt/mossbot/.local/share/virtualenvs/mossbot-B1iKwa68/lib/python3.6/site-packages/matrix_client/api.py", line 92, in sync
> Aug 23 09:11:54 foobar pipenv[4205]:         api_path=MATRIX_V2_API_PATH)
> Aug 23 09:11:54 foobar pipenv[4205]:       File "/opt/mossbot/.local/share/virtualenvs/mossbot-B1iKwa68/lib/python3.6/site-packages/matrix_client/api.py", line 526, in _send
> Aug 23 09:11:54 foobar pipenv[4205]:         verify=self.validate_cert
> Aug 23 09:11:54 foobar pipenv[4205]:       File "/opt/mossbot/.local/share/virtualenvs/mossbot-B1iKwa68/lib/python3.6/site-packages/requests/api.py", line 58, in request
> Aug 23 09:11:54 foobar pipenv[4205]:         return session.request(method=method, url=url, **kwargs)
> Aug 23 09:11:54 foobar pipenv[4205]:       File "/opt/mossbot/.local/share/virtualenvs/mossbot-B1iKwa68/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
> Aug 23 09:11:54 foobar pipenv[4205]:         resp = self.send(prep, **send_kwargs)
> Aug 23 09:11:54 foobar pipenv[4205]:       File "/opt/mossbot/.local/share/virtualenvs/mossbot-B1iKwa68/lib/python3.6/site-packages/requests/sessions.py", line 658, in send
> Aug 23 09:11:54 foobar pipenv[4205]:         r.content
> Aug 23 09:11:54 foobar pipenv[4205]:       File "/opt/mossbot/.local/share/virtualenvs/mossbot-B1iKwa68/lib/python3.6/site-packages/requests/models.py", line 823, in content
> Aug 23 09:11:54 foobar pipenv[4205]:         self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
> Aug 23 09:11:54 foobar pipenv[4205]:       File "/opt/mossbot/.local/share/virtualenvs/mossbot-B1iKwa68/lib/python3.6/site-packages/requests/models.py", line 748, in generate
> Aug 23 09:11:54 foobar pipenv[4205]:         raise ChunkedEncodingError(e)
> Aug 23 09:11:54 foobar pipenv[4205]:     requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

i would downgrade but this sql fiddling is a little bit too high for me.

[xsteadfastx]

another workaround for me:

set compress: false in homeserver.yaml

  - port: 8008
    tls: false
    bind_address: ''
    type: http

    x_forwarded: true

    resources:
      - names: [client, webclient]
        #compress: true
        compress: false
      - names: [federation]
        compress: false

[andrewshadura]

Actually, since @richvdh refreshed the repos, the proper way to install the old version is now to do: apt install matrix-synapse=0.33.2-1

[Half-Shot]

Seems like we would need to enforce compression: false for older twisted versions ?

[xsteadfastx]

@andrewshadura but this doesnt do the sql rollback or?

[richvdh]

@andrewshadura but this doesnt do the sql rollback or?

correct, it just replaces the downlad and dpkg -i

[richvdh]

ok so the conclusion about what we are going to do here is:

• we are going to ship backported packages of more recent Twisted for Debian Stretch, and Ubuntu Xenial and Artful (Buster and Bionic have recent enough Twisteds already)
• we are going to drop support for Debian Jessie. Sorry to anyone who is still using it, but it's out of support from Debian and you really ought to upgrade...

[richvdh]

In the meantime, @xsteadfastx 's workaround seems like a sensible alternative to downgrading.

[andrewshadura]

I think support for jessie needs to be dropped.

[4nd3r]

we are going to drop support for Debian Jessie. Sorry to anyone who is still using it,
but it's out of support from Debian and you really ought to upgrade...

wrong. debian jessie is supported until 2020: https://wiki.debian.org/LTS

[ptman]

@4nd3r Are all packages supported? Does it make sense for matrix to support Debian jessie? Would jessie be fine with using docker to run matrix?

[richvdh]

LTS really isn't the same as Debian. Either way, continuing to support it is way too much of an overhead for us given the benefits.

[4nd3r]

@ptman you can ask same with all the packages in stable, testing or unstable. it's community effort. lots of versions are often just carried over without major version upgrade.

using docker imho is bit too much hassle and useless overhead. most sane people run service per VM, so doing dist-upgrade for machine which only runs synapse and postgres (+deps) takes 15 minutes or less if you know what are you doing (postgres cluster upgrade being that "difficult part"). tho, there are lots of people who run mammoth machines (due to cheap VPS providers) which run multiple services. for them, dist-upgrade may be somewhat painful. ymmv.

LTS really isn't the same as Debian.

LTS track record for Wheezy was magnitude better than it was for Squeeze, so it's getting better. I support their effort.

but I don't mind if Jessie support is dropped. so, if it's too much effort to backport twisted, then just drop it. i'm not familiar with python ecosystem and who knows what will happen with those machines, which run other services which depend on (older version of ) twisted. so it would be even wise not to backport twisted to jessie.

[richvdh]

We've now added backports of Twisted 18.4 to the repo (many thanks to @andrewshadura for putting in the legwork there), and released Synapse 0.33.3-2 which depends on them.

[richvdh]

As for Jessie support: the problem here is that it comes with an opportunity cost.

I've already spent a day and a half this week rebuilding packages, uploading them to repos, and testing on various Debian/Ubuntu flavours, which is time that would otherwise have been spent improving the security and scalability of the Matrix protocol and Synapse. Building a recent Twisted for Jessie would have been a lot more effort. (As it is, we should all be very grateful to @andrewshadura that he was able to backport as far as Squeeze.)

We're not prepared to continue to support very old versions of Twisted. I could have spent this week trying to figure out how to keep things working on Twisted 16.x without regressing the bug which #3701 fixed, but the problem is that it's impossible to effectively test against these older versions, so we only discover incompatibilities once the final version is out and people start installing their Debian packages. It's also worth noting that the list of bugs fixed in Twisted since 16.6 is scary and the Twisted team themselves do not recommend using old versions. I have no desire to spend my time fielding reports of bugs which turn out to have been fixed in more recent Twisted versions.

Unfortunately, the only conclusion is that we're going to have to stop shipping Synapse packages for Jessie. Ultimately the success of the Matrix project hinges on getting wider uptake, and I believe that the lack of a secure, stable r0 release is a bigger blocker to that than lack of Jessie packages.

I understand that those using Jessie might find this frustrating, but I also hope you can understand this decision. For those in that situation: my recommendation is to upgrade to Stretch, but you might also look into setting up a virtualenv to run Synapse in, as per https://github.com/matrix-org/synapse/blob/master/README.rst#installing-from-source.

[richvdh]

Now that we have pinned to Twisted 17.1 or later, I'm considering this resolved.