Implemented pipeline for automated vulnerability fetching and processing #380
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Objective
This PR aims to introduce a new feature that leverages the NVD to retrieve and analyze newly released CVEs. By fetching NVD data, filtering out undesired entries, and using a parallel approach for processing vulnerabilities, we enhance the tool's ability to find fixes for vulnerabilities automatically.
Key Changes
Filtering: We have implemented a filtering mechanism to ensure that Prospector processes only vulnerabilities linked to OSS products. This involves extracting the product name from the vulnerability description and matching it with a previously mapped list of projects. This list includes additional metadata like the corresponding GitHub repository for each product.
Parallel Processing: To improve processing, we've introduced parallel execution using a worker-based architecture. Tasks are distributed across multiple workers managed by a Redis queue.
Database Storage: We have implemented a database system to store the outcomes of CVE analysis. The db contains both raw and processed vulnerability data, allowing for easier management via SQL queries. Furthermore, the database simplifies future studies on pipeline coverage as well as Prospector capabilities.
REST API and Frontend: REST API endpoints have been developed to provide access to the database, fetch new CVEs from NVD by specifying a time range, filter and analyse them, and view the results. Additionally, a simple webpage has been created to facilitate interaction with the APIs.