Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve registry performance #359

Closed
wants to merge 2 commits into from
Closed

Conversation

dtrai2
Copy link
Collaborator

@dtrai2 dtrai2 commented Mar 27, 2023

This pull request introduces a more dynamic importing of the logprep components.

Previously every logprep start imported literally all components, even those which weren't used due to the configuration. This lead to an unnecessary overhead for importing classes and their corresponding used classes/methods (e.g elasticsearch, opensearch, or hyperscan,...) even though they were unimportant for the configured run.

This is not a huge performance improvement, but it should still improve the startup a little bit.

- import logprep components only when needed
@dtrai2 dtrai2 added the enhancement New feature or request label Mar 27, 2023
@dtrai2 dtrai2 requested a review from ekneg54 March 27, 2023 13:36
@dtrai2 dtrai2 self-assigned this Mar 27, 2023
@codecov-commenter
Copy link

codecov-commenter commented Mar 27, 2023

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.07 🎉

Comparison is base (21e317f) 91.46% compared to head (71559a8) 91.54%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #359      +/-   ##
==========================================
+ Coverage   91.46%   91.54%   +0.07%     
==========================================
  Files         121      121              
  Lines        8522     8492      -30     
==========================================
- Hits         7795     7774      -21     
+ Misses        727      718       -9     
Impacted Files Coverage Δ
logprep/configuration.py 100.00% <100.00%> (ø)
logprep/connector/confluent_kafka/input.py 100.00% <100.00%> (ø)
logprep/connector/confluent_kafka/output.py 100.00% <100.00%> (ø)
logprep/registry.py 100.00% <100.00%> (ø)

... and 1 file with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

Copy link
Collaborator

@ekneg54 ekneg54 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a bad feeling with this. From developer perspective it is much more to write now. From a security perspective I'm not fine with importing from arbitrary strings and from performance perspective as you say it has not a big impact.

Is there another perspective I'm not aware of for this change?

@ekneg54 ekneg54 closed this Apr 17, 2023
@ekneg54 ekneg54 deleted the dev-improve-registry-performance branch June 6, 2023 09:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants