Take a look at Google's Insights project (https://deps.dev/)

[sschuberth]

Google just announced the Insights project. In large parts it has similarities with ORT's analyzer and advisor components, and pages like this one have similarities with the web app report.

I'll reach out to the Insights team to learn about how they are doing things.

[sschuberth]

Their FAQ is a very interesting read:

"How is Insights implemented?

Most of the back end is written in Go and reimplements the dependency analysis algorithm (version resolution) of each packaging system. The code has been tested for agreement with the native tooling, but is faster because it is decoupled from package installation. It constructs the graph without actually building and installing the software."

[tsteenbe]

Sofar insights projects only npm, Maven, Go modules, and Cargo for which static analysis should work really well (apart from maybe Go). Would be interested to learn how they did Go support and how they are planning to do PyPI.

[sschuberth]

I'll share what can be learned from their response once I get it.

[sschuberth]

Insight's data is now also available via BigQuery. Maybe this data could be used by the Analyzer to speed up the lookup of package metadata.

[sschuberth]

I'll share what can be learned from their response once I get it.

BTW, I never heard back from them, so I just reached out again.

[sschuberth]

BTW, the deps.dev people posted a great blog article that highlights the importance of "context" during dependency resolution, but their API currently doesn't offer a way to provide this context.

[Aponace]

@sschuberth I have actually implemented an identical mechanism about a year ago in Python and Neo4J. From my experience, aside from it being cool and all, you need a very large infrastructure that's being updated with new version releases (which even deps.dev doesn't do - they update maybe twice a year) and a complex resolution logic to match the actual tree to the one generated by the package managers (which depends on the context of where the package is used and alongside which other packages). In other words, it's not a practical solution unless you have a lot of money and research invested in it on a dedicated central infrastructure. You can't get the exact picture of the dependency tree unless you use the native package managers resolution mechanism.

[sschuberth]

You can't get the exact picture of the dependency tree unless you use the native package managers resolution mechanism.

I absolutely agree @Aponace, and that's why ORT's analyzer has always aimed to use the native package manager's resolution mechanism.

However, it was brought to my attention that several ORT users are not necessarily interested in the exact picture, but something "close enough" (that can be determined much faster) is sufficient. From a license compliance aspect, that might even be true, as licenses usually do not change between "similar" package versions. But from a security compliance aspect, that might be a problem, as also patch level package versions may introduce severe security risks.

[sschuberth]

Their new API now also supports PURLs.

[sschuberth]

Also @jamietanna's Dependency Management Data (DMD) seems to be using deps.dev, which might be an interesting integration point.

[jamietanna]

Thanks Sebastian, yes, you should be able to use DMD alongside ORT (https://dmd.tanna.dev/cookbooks/getting-started-ort/) in the meantime to get this data! But would be great to have first-class support

[heliocastro]

@jamietanna Just to mention, the docs are outdated, we moved from ort-extended to ort a while ago.
We have this two images:
ghcr.io/oss-review-toolkit/ort-minimal - The (kind of) minimal image with python, rust, ruby and node
ghcr.io/oss-review-toolkit/ort - The one that have everything and is the direct replacement for ort-extended

[jamietanna]

@jamietanna Just to mention, the docs are outdated, we moved from ort-extended to ort a while ago. We have this two images: ghcr.io/oss-review-toolkit/ort-minimal - The (kind of) minimal image with python, rust, ruby and node ghcr.io/oss-review-toolkit/ort - The one that have everything and is the direct replacement for ort-extended

Thanks! I'm on holiday for the next week, but can look to update when I'm home. Or I'd welcome a Merge Request on GitLab if you were happy to update for me ☺