Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Lazy load cardigann indexers + dynamic load and unload. #7851

Open
cadatoiva opened this issue Mar 25, 2020 · 3 comments
Open

[Feature] Lazy load cardigann indexers + dynamic load and unload. #7851

cadatoiva opened this issue Mar 25, 2020 · 3 comments

Comments

@cadatoiva
Copy link
Collaborator

Reading through old issues, I found some potentially lower hanging fruit that can really help our memory usage in #4720 The way we load our indexers is a huge memory sink, since Cardigann currently holds the entire YAML file in static memory and reparses the definition on every use of the tracker. So this enhancement has potentially two parts that can be implemented in one of several configurations to significantly lower running memory usage:

  1. We can "lazy load" the cardigann indexers. Possible implementation: we index all .yml files but not load in their definition unless they're a configured indexer. We read enough in to get the Add Indexer configuration page working, but then close the file and remove the definition from running memory. We could also delete the definition from memory when the indexer is deleted from the active indexer list. (dynamic load + unload)

  2. We can revamp how cardigann indexers are loaded in memory. Instead of holding so many strings to the point where the YAML file is almost completely loaded and held in memory, it might be good to roll them into enums or some smaller data type. Maybe even using reflection to basically compile a "new class" based on the definition when we do.

@ngosang
Copy link
Member

ngosang commented Mar 28, 2020

I support this feature request but with low priority.
I think we have to finish the httpclient refactor and fix several open issues before working on this.

@cadatoiva
Copy link
Collaborator Author

Yup, this is just the reminder that it's on the list of todo.

@ngosang
Copy link
Member

ngosang commented May 16, 2020

Related #6357 #7160

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Development

No branches or pull requests

2 participants