Add V2_importer to collect advisories from EUVD #2046
Conversation
Signed-off-by: Sampurna Pyne <sampurnapyne1710@gmail.com>
Samk1710
force-pushed
the
add-euvd-importer
branch
from
267ed3b to
19d4d45
Compare
Signed-off-by: Sampurna Pyne <sampurnapyne1710@gmail.com>
| advisory = self.parse_advisory(raw_data) | ||
| if advisory: | ||
| yield advisory | ||
| except Exception as e: |
There was a problem hiding this comment.
Please avoid using general exceptions.
| first = advisories[0] | ||
| assert isinstance(first, AdvisoryData) | ||
| assert first.advisory_id == "EUVD-2025-197757" | ||
| assert "EUVD-2025-197757" in first.aliases | ||
| assert "CVE-2025-13284" in first.aliases | ||
| assert first.summary == "ThinPLUS vulnerability that allows remote code execution" | ||
| assert first.date_published is not None | ||
| assert len(first.severities) == 1 | ||
| assert first.severities[0].system.identifier == "cvssv3.1" | ||
| assert first.severities[0].value == "9.8" |
There was a problem hiding this comment.
I think it would be easier if you test using util_tests.check_results_against_json(result, expected_file) and with an expected file.
| if self._cached_data is not None: | ||
| logger.info(f"Using cached data: {len(self._cached_data)} items") | ||
| return self._cached_data |
There was a problem hiding this comment.
Why do we have _cached_data? It is because the API returns repeated data
There was a problem hiding this comment.
_cached_data prevents a second full API fetch.
The base importer calls fetch_data() once to count advisories and again to iterate through them.
Caching ensures both steps use the same dataset snapshot while avoiding duplicated network requests and API load.
|
|
||
| logger.info(f"Fetching data from EUVD API: {self.url}") | ||
|
|
||
| while True: |
There was a problem hiding this comment.
We should avoid loops without a condition. Maybe looping over the total 452584 advisories is a good idea.
| logger.error(f"API returned status {response.status_code} for page {page}") | ||
| retry_count += 1 | ||
| if retry_count < max_retries: | ||
| sleep_time = min(10 * (2 ** min(retry_count - 1, 5)), 60) |
There was a problem hiding this comment.
Why this sleep_time? We run the importers multiple times. If one request fails, we can have just one retry.
( please avoid complex retry )
There was a problem hiding this comment.
I added the retry logic as I faced some API failures during importer run (in my case it was network failures), but as you mentioned that we run importers multiple times, it shouldn't be an issue. I will apply one retry on failure as suggested.
2 participants
EUVD Importer
Overview
This pull request introduces a new importer for the EU Vulnerability Database (EUVD) provided by ENISA. The importer retrieves vulnerability advisories via the EUVD JSON API and integrates them into VulnerableCode.
Data Source
https://euvdservices.enisa.europa.eu/api/searchTest Run Results
Unit & pipeline tests
Importer Run (Full EUVD Dataset)
Importer log (451,638 advisories)
The full EUVD pipeline completed successfully in 6.4 hours locally, relevant log excerpt: