Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature] Update deps.dev collector to take advantage of the new API #1768

Open
pxp928 opened this issue Mar 12, 2024 · 6 comments
Open

[feature] Update deps.dev collector to take advantage of the new API #1768

pxp928 opened this issue Mar 12, 2024 · 6 comments
Assignees
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed

Comments

@pxp928
Copy link
Collaborator

pxp928 commented Mar 12, 2024

Is your feature request related to a problem? Please describe.

With the new release of the API for deps.dev -> https://blog.deps.dev/api-v3/

  • Take advantage of the batch query support
  • Utilizing purl when querying for information (no need to decompose the purl as we are currently doing)

packageInput, err := helpers.PurlToPkg(purl)
if err != nil {
logger.Infof("failed to parse purl to pkg: %s", purl)
return nil
}
// if version is not specified, cannot obtain accurate information from deps.dev. Log as info and skip the purl.
if *packageInput.Version == "" {
logger.Infof("purl does not contain version, skipping deps.dev query: %s", purl)
return nil
}
component.CurrentPackage = packageInput
err = d.collectAdditionalMetadata(ctx, packageInput.Type, packageInput.Namespace, packageInput.Name, packageInput.Version, component)
if err != nil {
logger.Debugf("failed to get additional metadata for package: %s, err: %v", purl, err)
}
)

Describe the solution you'd like
upgrade the current deps.dev collector to use the latest API features

@pxp928 pxp928 added enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed labels Mar 12, 2024
@pxp928
Copy link
Collaborator Author

pxp928 commented Apr 29, 2024

@rakshitgondwal would you be interested in working on this?

@rakshitgondwal
Copy link
Contributor

Sure @pxp928, I can take this up after #1710

@pxp928
Copy link
Collaborator Author

pxp928 commented May 24, 2024

@rakshitgondwal have you started any work on this? If not, @nathannaveen would like to take it over.

@rakshitgondwal
Copy link
Contributor

Hi @pxp928, nope I didn't get the time to do so, and sure someone else can take this over.

@nathannaveen
Copy link
Contributor

Hey @pxp928, could I work on this?

@nathannaveen
Copy link
Contributor

nathannaveen commented Jun 21, 2024

Recently I have been working on this issue, and realize that since the deps.dev v3alpha version isn't stable it won't exactly fit our needs.


For example, when doing a purl batch lookup https://docs.deps.dev/api/v3alpha/#purllookupbatch, if any of the repos passed to the batch request aren't contained in the deps.dev database, it returns nil.

So the only way to check whether a purl is contained in the deps.dev database is to manually do a purlLookup for each individual purl. Which removes our need for the purlLookupBatch.

This issues will probably disappear when the deps.dev/api/v3alpha becomes stable. But for now, this won't work for what we want.


Additionally, we directly use the osv-scanner repo.

osv_scanner "github.com/google/osv-scanner/pkg/osv"

The issue with this, is that osv-scanner calls the deps.dev resolve API, which only works with deps.dev/api/v3 not v3alpha.

Note that I didn't realize we directly use the osv-scanner repo at the time of writing this issue #1947.


So I think the best way to go about this is to wait for the version to become stable and then implement it.

If you want to run some tests, this is the batch request that will return nil: main...nathannaveen:guac:nathan/depsdevNewAPI#diff-82819d87163ba23dd5d158849f788068d5eadd6e1818517c879ec08b0745131cR455

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

3 participants