Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve resilience of fact refresh to high load #6178

Open
wouterdb opened this issue Jun 26, 2023 · 4 comments
Open

Improve resilience of fact refresh to high load #6178

wouterdb opened this issue Jun 26, 2023 · 4 comments
Labels
discussion There is a discussion happening on this issue

Comments

@wouterdb
Copy link
Contributor

wouterdb commented Jun 26, 2023

Problem

When we need to refresh several thousands of facts, this creates load spikes we want to avoid

Proposed solution

  1. make fact_expire and fact_refresh per environment settings
  2. add a fact_remove parameter, that expresses how long after expiry the parameter should be kept (default to one week)
  3. add a fact_rate_limiter that expresses how many facts per hour any agent can get. (float) The idea is to never exceed this rate. (i.e. the effective wait time between requests is at least 1/fact_rate_limiter)
  4. never have more than one fact refresh in flight per agent (i.e. make it asynchronous through and through, the call from server to agent already is)

general improvements

  1. make sure facts that are not in the active version are not refreshed
  2. move the lineself._fact_resource_block_set[resource_id] = now closer to the if statement, to prevent races
    self._fact_resource_block_set[resource_id] = now
@wouterdb wouterdb added the discussion There is a discussion happening on this issue label Oct 23, 2023
@wouterdb
Copy link
Contributor Author

Do we need this asap or do we see it as part of larger agent rework

@bartv
Copy link
Contributor

bartv commented Oct 23, 2023

We can fix the issues at x by supporting facts that never expire

@wouterdb
Copy link
Contributor Author

wouterdb commented Oct 23, 2023

correct, thx, for reference: #6560

@bartv
Copy link
Contributor

bartv commented Oct 23, 2023

correct, thx, for reference: #6560

Cool. I would do this on short term and make the other stuff part of the agent rework

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
discussion There is a discussion happening on this issue
Projects
None yet
Development

No branches or pull requests

2 participants