feat: 1944 - update purl generation to use a consistent groupID #2033
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Keith Zantow <kzantow@gmail.com>
Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
spiffcs
force-pushed
the
1944-consistent-purl
branch
from
f03e747
to
f8a6582
Compare
|
@westonsteimel I combined the work from 2032 and the single groupID answer code together in this branch - this does not alter any CPE generation or change how java packages are deduped - I'll be updating those in a separate PR I'm adding two commits here -
Syft Maindb crawler results : currently running:
^ I have to run these again but my ISP just went out and I don't want to pull this down over data 😬 Latest Comparison |
Benchmark Test ResultsBenchmark results from the latest changes vs base branch |
spiffcs
force-pushed
the
1944-consistent-purl
branch
from
2ce6400
to
fe85967
Compare
Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
spiffcs
force-pushed
the
1944-consistent-purl
branch
from
fe85967
to
9f3157c
Compare
Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
spiffcs
changed the title
wagoodman
reviewed
Aug 18, 2023
| // 2. The group ID from the POM project | ||
| // 3. The group ID from a select map of known group IDs | ||
| // 3. The group ID from the Java manifest | ||
| func GroupIDFromJavaMetadata(pkgName string, metadata pkg.JavaMetadata) (groupID string) { |
There was a problem hiding this comment.
all of these new functions probably don't belong in the cpe package since they have been replicated to not affect the CPE generation logic today. My vote would be to put this under the java cataloger package (as near duplicate functions) and unexport all of the functions (I'd duplicate the looksLikeGroupID function too... since there isn't a specific definition for this and the impl may change based on the use case).
There was a problem hiding this comment.
I think I parsed this comment correctly - check f1e05da
If there are some edits or things to move around from here we can do a quick sync on the package organization and exports/imports
wagoodman
reviewed
Aug 18, 2023
wagoodman
reviewed
Aug 18, 2023
wagoodman
reviewed
Aug 18, 2023
Comment on lines
384
to
385
| parentKey := fmt.Sprintf("%s:%s:%s", groupID, parentPkg.Name, parentPkg.Version) | ||
| pomProjectKey := fmt.Sprintf("%s:%s:%s", pomProperties.GroupID, pomProperties.ArtifactID, pomProperties.Version) |
There was a problem hiding this comment.
this seems to be trying to correlate the parent name and the current pom properties artifact ID and changing the virtual path when this occurs. Can you leave a code comment to why this is important? (and this might look test worthy)
There was a problem hiding this comment.
updated with commentary and test updates in 06bd7b9
There was a problem hiding this comment.
to clarify: why is the parent package name and pom properties artifact ID being compared? (instead of artifact id vs artifact id or name vs name?)
There was a problem hiding this comment.
Added comment explaining the significance and why we use the values we're using: 1bbb6c5
Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
kzantow
reviewed
Aug 21, 2023
Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
|
Latest run: 1e4f54a |
Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
|
Ratio when removing ArtifactID: 77a6fd4 Conclusion is we had less correct by about 10,000 with a larger sample size so I'm reverting the commit. |
kzantow
reviewed
Aug 22, 2023
spiffcs
force-pushed
the
1944-consistent-purl
branch
from
05f7f67
to
c3b3e79
Compare
This reverts commit 77a6fd4. Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com>
spiffcs
force-pushed
the
1944-consistent-purl
branch
from
c3b3e79
to
1bbb6c5
Compare
This was referenced Aug 28, 2023
Closed
This was referenced Sep 1, 2023
Merged
GijsCalis
pushed a commit
to GijsCalis/syft
that referenced
this pull request
Feb 19, 2024
…ore#2033) Separate the logic for CPE and PURL generation. PURL generation needs a single answer for groupID based on a priority of discovering the field. CPE generation still uses multiple potential groupID to populate the candidate cpe. Improve GroupID detection. Currently syft does not use any hierarchy for GroupID detection and treats all sources as equal. It treats fields from the manifest file with priority. This change adds a hierarchy to the fields and returns a single answer based on that hierarchy. --------- Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com> Signed-off-by: Keith Zantow <kzantow@gmail.com> Co-authored-by: Keith Zantow <kzantow@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR aims to improve 3 related aspects of syft's java cataloging. Pulls in #2032
Separate the logic for CPE and PURL generation. PURL generation needs a single answer for groupID based on a priority of discovering the field. CPE generation still uses multiple potential groupID to populate the candidate cpe.
Improve GroupID detection. Currently syft does not use any hierarchy for GroupID detection and treats all sources as equal. It treats fields from the manifest file with priority. We should try to obtain a GroupID answer from Pom Properties, then the Pom Project and finally, if a GroupID has not been found, the Manifest. Manifest should not return an answer if one was found in PomProject, and PomProject should not return a GroupID answer if PomProperties found one.
Note: we've removed the
child matches parent by virtual pathcase because the virtual path now has more uniqueness that depends on a packages metadata. The parent did not fulfill this requirement and thus a new package was created, virtual paths no longer match, and the two packages were not merged