Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implementation of C++ metrics parallelization #734

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

dbukki
Copy link
Collaborator

@dbukki dbukki commented Apr 12, 2024

Fixes #733

Added parallelization support to the C++ metrics parser plugin:

  • Using getFilterPathsQuery, we can now perform the same check as what cc::util::isRootedUnderAnyOf used to do, but on the ODB-side. This is good because:
    • The number of records retrieved for manual processing by the client's handler is now drastically reduced.
    • The filePath members can now be removed from views that only needed them for this purpose. This further reduces the amount of data needed to be transferred per record as well.
  • The parallelCalcMetric method of the metrics parser now serves as the backbone of all parallel computations.
  • Existing metrics have been refactored to perform their outermost loop using the new parallelization mechanism.

Further considerations:

  • Buffering: Preload all records to already know the size on startup. (This is a memory overhead, but it might be worth it.)
  • Progress reporting: Building on the idea of buffering, report the current progress of each metric onto the log to increase interactivity. (Reporting at regular intervals in time and/or in steps.)
  • Caching & Flushing: Collect all computed metrics into a single shared thread-safe container and persist everything at once at the end. This could eliminate the "one transaction for each worker" phenomenon currently observed. (Depending on the size of the overhead of opening and closing transactions, this could potentially give us a speed boost.)

@dbukki dbukki self-assigned this Apr 12, 2024
@mcserep mcserep added Kind: Enhancement 🌟 Plugin: C++ Issues related to the parsing and presentation of C++ projects. Plugin: Metrics Issues related to the code metrics plugin. labels Apr 15, 2024
@mcserep mcserep requested review from intjftw and mcserep April 24, 2024 14:03
@mcserep mcserep added this to the Upcoming Release milestone Apr 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Kind: Enhancement 🌟 Plugin: C++ Issues related to the parsing and presentation of C++ projects. Plugin: Metrics Issues related to the code metrics plugin.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Parallel parsing in C++ metrics
2 participants