Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf(license): enable license classifier only with "--license-full" #3086

Merged
merged 2 commits into from
Oct 28, 2022

Conversation

knqyf263
Copy link
Collaborator

@knqyf263 knqyf263 commented Oct 27, 2022

Description

Initializing license classifiers and performing license classification is expensive and it is disabled by default. However, it is used only when scanning Debian because Debian licenses are not stored in a machine-readable format. In other words, scanning Debian activates the license classification and leads to high CPU/memory utilization.
This PR changes it to try to detect licenses without license classifiers by default and enable classifies only when --license-full is specified.

Checklist

  • I've read the guidelines for contributing to this repository.
  • I've followed the conventions in the PR title.
  • I've added tests that prove my fix is effective or that my feature works.
  • I've updated the documentation with the relevant information (if needed).
  • I've added usage information (if the PR introduces new options)
  • I've included a "before" and "after" example to the description (if the PR is a user interface change).

@knqyf263 knqyf263 self-assigned this Oct 27, 2022
@knqyf263 knqyf263 changed the title feat(license): enable license classifier only with "--license-full" perf(license): enable license classifier only with "--license-full" Oct 27, 2022
@knqyf263 knqyf263 requested a review from afdesk October 27, 2022 12:09
@afdesk
Copy link
Contributor

afdesk commented Oct 27, 2022

yes, this PR is really improve performance.

at first, I've run trivy v0.33.0.

$ trivy -v
Version: 0.33.0
Vulnerability DB:
  Version: 2
  UpdatedAt: 2022-10-27 12:08:13.914255809 +0000 UTC
  NextUpdate: 2022-10-27 18:08:13.914255509 +0000 UTC
  DownloadedAt: 2022-10-27 14:05:26.677439944 +0000 UTC

$ rm -r ~/.cache/trivy/fanal/ && trivy i perl:5.34.0

the maximum memory consumption was 511,2 MB:
изображение

then I've run the current branch:

$ ./tr -v
Version: dev
Vulnerability DB:
  Version: 2
  UpdatedAt: 2022-10-27 12:08:13.914255809 +0000 UTC
  NextUpdate: 2022-10-27 18:08:13.914255509 +0000 UTC
  DownloadedAt: 2022-10-27 14:05:26.677439944 +0000 UTC

$ rm -r ~/.cache/trivy/fanal/ && ./tr i perl:5.34.0

the maximum memory was 311MB:
изображение

@knqyf263 knqyf263 merged commit 7cc83cc into main Oct 28, 2022
@knqyf263 knqyf263 deleted the debian/license_full branch October 28, 2022 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants