Unencrypted uploads larger than 2GB fail but the file is kept in the media_storage_path dir

[ldot72]

Description

When uploading a file >= 2GB to an unencrypted room, the upload fails after the whole file has been transferred. The file is stored in media_storage_path, but is invisible to the client. Uploading unencrypted files >= 2GB is possible with Element, fails with nheko. Uploading files >= 2GB to encrypted rooms seems to fail before the file is actually transferred regardless of client, so won't lead to the issue described here.

Steps to reproduce

• Set up matrix-synapse with nginx. Set max_upload_size = 10000M in homeserver.yaml and client_max_body_size 10000M in nginx conf
• Upload a file that's larger than 2GB

2022-02-17 17:33:30,131 - synapse.rest.media.v1.media_repository - 180 - INFO - POST-13679 - Stored local media in file '/var/lib/matrix-synapse/media/local_content/Zh/eF/VxQasjIpEYksYkggHXqV'
2022-02-17 17:33:30,134 - synapse.http.server - 100 - ERROR - POST-13679 - Failed handle request via 'UploadResource': <XForwardedForRequest at 0x7f88f4670ca0 method='POST' uri='/_matrix/media/r0/upload?filename=testfile4.tar' clientproto='HTTP/1.0' site='8008'>
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/http/server.py", line 269, in _async_render_wrapper
    callback_return = await self._async_render(request)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/http/server.py", line 297, in _async_render
    callback_return = await raw_callback_return
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/rest/media/v1/upload_resource.py", line 91, in _async_render_POST
    content_uri = await self.media_repo.create_content(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/rest/media/v1/media_repository.py", line 182, in create_content
    await self.store.store_local_media(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/media_repository.py", line 319, in store_local_media
    await self.db_pool.simple_insert(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 931, in simple_insert
    await self.runInteraction(desc, self.simple_insert_txn, table, values)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 743, in runInteraction
    result = await self.runWithConnection(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 857, in runWithConnection
    return await make_deferred_yieldable(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 850, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 609, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 945, in simple_insert_txn
    txn.execute(sql, vals)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 310, in execute
    self._do_execute(self.txn.execute, sql, *args)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 343, in _do_execute
    return func(sql, *args)
psycopg2.errors.NumericValueOutOfRange: value "2447099114" is out of range for type integer
LINE 1: ...ion/octet-stream', 1645173210131, 'testfile4.tar', '244709911...

Expected: The upload succeeds.
What happens instead: The upload fails, but the file is kept on the disk.

Version information

If not matrix.org:

• Version: 1.52.0+bullseye1

• Install method:
  apt matrix-synapse-py3 via https://packages.matrix.org/debian/

• Platform:
  Debian 11 on a dedicated server

[dklimpel]

It seems to be a wrong type of database column.

synapse/synapse/storage/schema/main/full_schemas/54/full.sql.postgres

Lines 580 to 583 in f82d38e

	CREATE TABLE local_media_repository (
	media_id text,
	media_type text,
	media_length integer,

media_length is an int -> max. 2147483647
2147483647 bytes are 1.9999999990 gigabytes

[ldot72]

Changing media_length integer to media_length bigint fixes the issue as far as uploading an unencrypted file is concerned with the latest version of synapse.
The new limit would be 8192 petabytes.

[jtracey]

Sorry for the bump, but any chance this could be fixed? We would like this for our lab's internal server, and it seems like it would just need a very simple schema delta, which is the sort of change that's easy to make but a pain to maintain downstream.

[reivilibre]

change that's easy to make

Changing a type of a column is unfortunately not that simple (or at least not without taking a pesky lock on the table). I don't know the typical size of this table but there's a good chance it would ideally need a gradual migration. :(

[jtracey]

Fair enough, thanks for the update. We'll continue to work around it for now.

[reivilibre]

off the top of my head, a general idea of the steps involved for such a gradual migration:

1. add new bigint column and populate it for new files
2. (wait a schema version and Synapse version)
3. update SCHEMA_COMPAT_VERSION to prevent rolling back to a time before the new column was populated
4. start migrating old rows
5. (wait a schema version and Synapse version)
6. do the migration in the foreground for installations that have missed it
7. start reading from the new column — effectively, the feature now works
8. (wait a schema version and Synapse version)
9. update SCHEMA_COMPAT_VERSION to prevent rolling back to a time where the old column was being read
10. stop updating the old column
11. (wait a schema version and Synapse version)
12. drop the old column

I wrote this on the spot and haven't thoroughly checked it... I hope to document this kind of thing up soon since I had to come up with this for some other work we're doing