Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide visibility into dependency tree permutations #19

Open
achrinza opened this issue Mar 18, 2022 · 0 comments
Open

Provide visibility into dependency tree permutations #19

achrinza opened this issue Mar 18, 2022 · 0 comments

Comments

@achrinza
Copy link
Member

achrinza commented Mar 18, 2022

With the exception of the LoopBack CLI, the Node.js packages under the purview of the LoopBack project are commonly used as libraries (i.e. as a dependency of another Node.js project). Due to the nature of transitive dependencies, every dependency tree generated by LoopBack users - even if its the same package version - may differ significantly as new versions of direct and deep dependencies are released. This makes it difficult for the maintainers of LoopBack to monitor the impact of another Node.js package on LoopBack users, as it is unclear whether that dependency may have been installed as some kind of deep dependency of a LoopBack package.

While LoopBack users do have the tools necessary to check if their specific installation is compromised, not all users may have the means or time to check the impact of these issues. Hence, this issue is to track how the LoopBack project can reduce the effort needed to provide a brief and accurate insight into the impact of a Node.js package vulnerability to users of LoopBack packages.

Although it is not expected that the LoopBack project do this work for its users, it would be a plus-point if we can streamline this process and provide actionable insights for operators who want a more hands-off experience when managing LoopBack-based projects.

This issue is to track the brainstorming of ideas and solutions toward solving the aforementioned problem statement.

Research done thus far

The technical problem at hand is that there's no easy way to find out every possible Node.js package that may be installed by users of LoopBack packages.

The goal is to reduce the user's initial analysis to a simple glance at their package.json direct dependencies.

This problem isn't easily solved with readily-available APIs or solutions as:

  1. It requires heavy traversing of every permutation of the dependency tree, and
  2. significant calls to the NPM public registry to retrieve package metadata

One way to work around this heavy traversal requirement is to host a read-only replica of the NPM public registry, as suggested by https://docs.npmjs.com/policies/crawlers (emphasis mine):

if you want a full copy of our metadata it is always going to be faster to access the data via replication, which takes only an hour or two to provide full data and will thereafter automatically stay in sync.

Behind the scenes, the NPM public registry is powered by CouchDB, and its database ~95 GB in size according to https://replicate.npmjs.com/:

"sizes": {
  "file": 95630138970
}

This means that it may be feasible to host a copy of the NPM public registry, and perform queries against it to build the dependency tree permutations.

Replication API documentation: https://github.com/npm/registry/blob/9e368cf6aaca608da5b2c378c0d53f475298b916/docs/REPLICATE-API.md

Further areas of exploration:

  • Logic for getting dependency tree permutations for a specific package version (e.g. @loopback/core@1.0.0)
  • File format for storing this information
  • Automated and optimised periodic updating of dependency tree permutations
  • Corrolating and detecting existence of vulnerable dependency
@achrinza achrinza changed the title Provide visibility into SBOM permutations Provide visibility into dependency tree permutations Mar 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant