Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: go-rpmdb update #1757

Merged
merged 7 commits into from
Apr 24, 2023
Merged

chore: go-rpmdb update #1757

merged 7 commits into from
Apr 24, 2023

Conversation

spiffcs
Copy link
Contributor

@spiffcs spiffcs commented Apr 21, 2023

Summary

  • Bump go-rpmdb library to latest version

Notes

  • this is not failing on the RPMDB tests because of the adapter import change, but I'm not sure if we still need to load the sqlite adapter into our main.go implementation for the rpmdb functionality to still work correctly
  • I tested this against almalinux and found the correct packages for the RPMDB cataloger, but wanted to double check with the team before making this change

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
@github-actions
Copy link

github-actions bot commented Apr 21, 2023

Benchmark Test Results

Benchmark results from the latest changes vs base branch
goos: linux%0Agoarch: amd64%0Apkg: github.com/anchore/syft/test/integration%0Acpu: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz%0A                                                          │ ./.tmp/benchmark-11a071c.txt │%0A                                                          │            sec/op            │%0AImagePackageCatalogers/alpmdb-cataloger-2                                   11.64m ±  1%25%0AImagePackageCatalogers/ruby-gemspec-cataloger-2                             908.3µ ±  1%25%0AImagePackageCatalogers/python-package-cataloger-2                           3.107m ±  3%25%0AImagePackageCatalogers/php-composer-installed-cataloger-2                   776.7µ ±  1%25%0AImagePackageCatalogers/javascript-package-cataloger-2                       405.6µ ±  1%25%0AImagePackageCatalogers/dpkgdb-cataloger-2                                   586.6µ ±  2%25%0AImagePackageCatalogers/rpm-db-cataloger-2                                   533.2µ ±  3%25%0AImagePackageCatalogers/java-cataloger-2                                     13.01m ±  1%25%0AImagePackageCatalogers/graalvm-native-image-cataloger-2                     92.96µ ±  1%25%0AImagePackageCatalogers/apkdb-cataloger-2                                    608.7µ ±  1%25%0AImagePackageCatalogers/go-module-binary-cataloger-2                         97.20µ ± 13%25%0AImagePackageCatalogers/dotnet-deps-cataloger-2                              1.230m ±  1%25%0AImagePackageCatalogers/portage-cataloger-2                                  386.3µ ± 13%25%0AImagePackageCatalogers/nix-store-cataloger-2                                275.0µ ±  1%25%0AImagePackageCatalogers/sbom-cataloger-2                                     117.0µ ±  1%25%0AImagePackageCatalogers/binary-cataloger-2                                   208.3µ ±  1%25%0AImagePackageCatalogers/linux-kernel-cataloger-2                             41.53m ±  5%25%0Ageomean                                                                     814.6µ%0A%0A                                                          │ ./.tmp/benchmark-11a071c.txt │%0A                                                          │             B/op             │%0AImagePackageCatalogers/alpmdb-cataloger-2                                   5.067Mi ± 0%25%0AImagePackageCatalogers/ruby-gemspec-cataloger-2                             140.2Ki ± 0%25%0AImagePackageCatalogers/python-package-cataloger-2                           984.2Ki ± 0%25%0AImagePackageCatalogers/php-composer-installed-cataloger-2                   180.1Ki ± 0%25%0AImagePackageCatalogers/javascript-package-cataloger-2                       98.65Ki ± 0%25%0AImagePackageCatalogers/dpkgdb-cataloger-2                                   169.9Ki ± 0%25%0AImagePackageCatalogers/rpm-db-cataloger-2                                   178.2Ki ± 0%25%0AImagePackageCatalogers/java-cataloger-2                                     2.784Mi ± 0%25%0AImagePackageCatalogers/graalvm-native-image-cataloger-2                     8.750Ki ± 0%25%0AImagePackageCatalogers/apkdb-cataloger-2                                    145.1Ki ± 0%25%0AImagePackageCatalogers/go-module-binary-cataloger-2                         10.06Ki ± 0%25%0AImagePackageCatalogers/dotnet-deps-cataloger-2                              409.8Ki ± 0%25%0AImagePackageCatalogers/portage-cataloger-2                                  85.91Ki ± 0%25%0AImagePackageCatalogers/nix-store-cataloger-2                                48.91Ki ± 0%25%0AImagePackageCatalogers/sbom-cataloger-2                                     14.20Ki ± 0%25%0AImagePackageCatalogers/binary-cataloger-2                                   31.96Ki ± 0%25%0AImagePackageCatalogers/linux-kernel-cataloger-2                             62.65Mi ± 0%25%0Ageomean                                                                     192.8Ki%0A%0A                                                          │ ./.tmp/benchmark-11a071c.txt │%0A                                                          │          allocs/op           │%0AImagePackageCatalogers/alpmdb-cataloger-2                                    86.83k ± 0%25%0AImagePackageCatalogers/ruby-gemspec-cataloger-2                              2.280k ± 0%25%0AImagePackageCatalogers/python-package-cataloger-2                            15.95k ± 0%25%0AImagePackageCatalogers/php-composer-installed-cataloger-2                    3.797k ± 0%25%0AImagePackageCatalogers/javascript-package-cataloger-2                        1.321k ± 0%25%0AImagePackageCatalogers/dpkgdb-cataloger-2                                    2.989k ± 0%25%0AImagePackageCatalogers/rpm-db-cataloger-2                                    3.878k ± 0%25%0AImagePackageCatalogers/java-cataloger-2                                      39.47k ± 0%25%0AImagePackageCatalogers/graalvm-native-image-cataloger-2                       228.0 ± 0%25%0AImagePackageCatalogers/apkdb-cataloger-2                                     3.667k ± 0%25%0AImagePackageCatalogers/go-module-binary-cataloger-2                           281.0 ± 0%25%0AImagePackageCatalogers/dotnet-deps-cataloger-2                               6.326k ± 0%25%0AImagePackageCatalogers/portage-cataloger-2                                   1.660k ± 0%25%0AImagePackageCatalogers/nix-store-cataloger-2                                  884.0 ± 0%25%0AImagePackageCatalogers/sbom-cataloger-2                                       394.0 ± 0%25%0AImagePackageCatalogers/binary-cataloger-2                                     896.0 ± 0%25%0AImagePackageCatalogers/linux-kernel-cataloger-2                              2.795k ± 0%25%0Ageomean                                                                      2.649k

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
@spiffcs
Copy link
Contributor Author

spiffcs commented Apr 21, 2023

Thanks @kzantow! - I think this one is important enough too that we get @wagoodman also to give it some 👀 if there is something obvious I missed or if there is another dependency he might have been thinking about going with

@spiffcs spiffcs requested a review from wagoodman April 21, 2023 20:17
Copy link
Contributor

@wagoodman wagoodman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I need to check one more thing but don't have a laptop handy

@westonsteimel
Copy link
Contributor

Probably worth checking that grype works as expected when updated with this version of syft also just due to the SQL driver registration logic. I remember there was something weird happening there and that's why we use our own sqlite fork in grype (to get around the registration in init logic)

@@ -3,6 +3,8 @@ package integration
import (
"testing"

_ "modernc.org/sqlite"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this registers the sqlite driver only in test code, which wont get compiled into the final syft binary (which means the test will pass but syft will be unable to open sqlite RPMDBs since there will not be a registered driver)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed code that keeps this import, but I've moved it to the main package, this way library users can choose their own sqlite driver (we shouldn't be registering drivers that have side effects on the stdlib within our lib)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed another commit that keeps the import in main and this integration test (since this package does not use main it wouldn't have a sqlite driver).

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman
Copy link
Contributor

This will be an apparent breaking change for library users since syft used to wire up the sqlite driver into the stdlib via the go-rpmdb lib, however, this is no longer the case with this update, so lib users will need to register their own driver. This is a silent failure since not importing the driver will not result in a panic or error returned. I've added a log line to explicitly check if the sqlite driver is registered and log a warning if it is not.

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@spiffcs spiffcs merged commit c038f13 into main Apr 24, 2023
9 checks passed
@spiffcs spiffcs deleted the rpmdb-update branch April 24, 2023 14:34
spiffcs added a commit that referenced this pull request Apr 26, 2023
* main:
  chore(deps): bump modernc.org/sqlite from 1.21.2 to 1.22.0 (#1758)
  chore: go-rpmdb update (#1757)
  chore(deps): bump github.com/CycloneDX/cyclonedx-go from 0.7.1-0.20221222100750-41a1ac565cce to 0.7.1 (#1706)
  fix: Improve pnpm support (#1752)
  feat: Add template func `hasField` (#1754)
  fix: only cache java packages and not source content (#1750)

Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
@willmurphyscode willmurphyscode added the changelog-ignore Don't include this issue in the release changelog label May 5, 2023
GijsCalis pushed a commit to GijsCalis/syft that referenced this pull request Feb 19, 2024
Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Co-authored-by: Alex Goodman <alex.goodman@anchore.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
changelog-ignore Don't include this issue in the release changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants