Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[IDEA] buffer filter should set an expiration on redis lists #212

Open
frikilax opened this issue Nov 10, 2020 · 0 comments · May be fixed by #223
Open

[IDEA] buffer filter should set an expiration on redis lists #212

frikilax opened this issue Nov 10, 2020 · 0 comments · May be fixed by #223
Labels
enhancement New feature or request important

Comments

@frikilax
Copy link
Member

What is your feature about ?
the buffer filter currently has a "flaw" in data management:
if buffer sources are modified after some time, the old sources (aka redis lists) will probably not be empty, this create a "memory leak" of sorts, as those lists won't probably ever be deleted from Redis, stacking up and using unnecessary memory

If your request is about an existing filter/functionality
This Feature Request is about the buffer filter

Describe what you'd like
To prevent that from happening, a minimal change needs to be done:
when the buffer starts or triggers to get all cached data from Redis, it should set/reset an expiry on the key

Thanks to that, 2 problems are solved:

  • if sources names change inside the buffer filter, old sources will eventually be cleared out of Redis -> no "memory leak"
  • if Darwin happens to have some downtime, this technique can remove obsoleted data from Redis

Additional context
The expiry value should be coherent with data longevity AND the frequency at which the buffer needs to fetch cached data, a sensible option could be to take the maximum value between twice the fetch interval, and a fixed value corresponding to data "freshness" (time before considering data is obsolete).
In all cases, the expiry value should be strictly more than the configured fetch interval, to avoid losing data before fetching it

@frikilax frikilax added enhancement New feature or request important labels Nov 10, 2020
@frikilax frikilax linked a pull request May 17, 2021 that will close this issue
7 tasks
@frikilax frikilax linked a pull request Jun 8, 2021 that will close this issue
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request important
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant