Apparent memory leak with use_presence option

[hyossing]

Hello,

Description

GC(Garbage Collection) time increases unbounded for specific config in homeserver.yaml.

use_presence: False

Our running synapse's homeserver.yaml is quite old, so we just keep the use_presence options.
Then,

In my opinion, the gc time increase more as requests are increases.

It is not easy to catch the problem, because It is a kind of aging issue.

• Restarting synapse solve the problem because leaked memory is gone.

So, it took a time to know about it.

If we set the settings to the latest, then the unbounded increase is not appear anymore.

# Presence tracking allows users to see the state (e.g online/offline)
# of other local and remote users.
#
presence:
  # Uncomment to disable presence tracking on this homeserver. This option
  # replaces the previous top-level 'use_presence' option.
  #
  # enabled: false

  # Presence routers are third-party modules that can specify additional logic
  # to where presence updates from users are routed.
  #
  presence_router:
    # The custom module's class. Uncomment to use a custom presence router module.
    #
    #module: "my_custom_router.PresenceRouter"

    # Configuration options of the custom module. Refer to your module's
    # documentation for available options.
    #
    #config:
    #  example_option: 'something'

Steps to reproduce

Archive.zip

1. unzip

2. yarn install

3. docker-compose up -d

4. create postgresql database

• CREATE DATABASE synapse ENCODING 'UTF8' LC_COLLATE='C' LC_CTYPE='C' template=template0

6. access grafana - http://localhost:3000
7. import synapse dashboard and check it does work. - synapse_rev1.json

8. node synapse-gc-test.js

• Run the script one or two days, then check the GC time increase.

Thanks you.
Hyosung

[erikjohnston]

Thanks, do you see an increase in memory usage too? This may be due to caches slowly filling up, so you may see this fixed if you enable time based expiry for caches (which is enabled by default in v1.53.0). Not all caches are affected by time based expiry, so please let us know if time based expiry fixes this issue.

[hyossing]

Hello,

I'll test the new config expiry_time for the GC time issue.

Let you know the result.

Thanks you.

[hyossing]

Hello,

I've tested it.

However, It is the same.

In my opinion, the use_presence options cause the problem.

Please, deprecate the option.

[erikjohnston]

Oh, hmm. I wonder if its because we've disabled time based expiry for the caches for get_users_in_room and get_users_who_share_room_with_user. In the caches section do you see those caches increasing in size?

[hyossing]

I didn't see that.

My test code does just send-to-device and sync to get it.

Only, device_inbox is used.

Thanks you.

[erikjohnston]

Oh, that is very interesting. Looks like my assumption that it was due to caches is wrong.

Can i ask what your test code does? Does it create lots of new users? Maybe the data leak is we're storing a presence update for each new user forever?

[hyossing]

Hello,

My test code is on the top of the comments in this issue. (Archive.zip)

• Apparent memory leak with use_presence option #12053 (comment)

I explained how to reproduce the issue.

Thanks you.

[erikjohnston]

This is likely fixed by #12213, which is in v1.58.0.

The cause was that /sync would call set_state which did not correctly no-op when presence was enabled. This caused us to add things to the wheel timer without ever taking things out of it.