Failing to download CVEs

[Adley-Nastri]

When running cve-bin-tool, a SHAMismatch error occurs , leading to the whole rest of the process to completely halt.

 INFO     cve_bin_tool.CVEDB - Downloading CVE data...                                           cvedb.py:199
[17:33:10] ERROR    cve_bin_tool.CVEDB - SHAMismatch:                                              error_handler.py:136
                    https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2017.json.gz (have:
                    9A41A405D604C16E8EE7B04D7DB9812E49F5A93167335F0A954654FF844BE8CC, want:
                    13EA5C65187399E1F91836316952F496F76108810BD475B4C47A0BF91D8C993B)
Downloading CVEs... ------------------- --------------------  48% 0:00:03┌─────────────────────────────── Traceback (most recent call last) ────────────────────────────────┐
│ C:\Program Files\Python38\lib\site-packages\cve_bin_tool-2.1-py3.8.egg\cve_bin_tool\cvedb.py:149 │
│ in cache_update                                                                                  │
│                                                                                                  │
│   146 │   │   │   # exit(100)                                                                    │
│   147 │   │   │   os.unlink(filepath)                                                            │
│   148 │   │   │   with ErrorHandler(mode=self.error_mode, logger=self.LOGGER):                   │
│ > 149 │   │   │   │   raise SHAMismatch(f"{url} (have: {gotsha}, want: {sha})")                  │
│   150 │                                                                                          │
│   151 │   @staticmethod                                                                          │
│   152 │   async def get_curl_versions(session):                                                  │
└──────────────────────────────────────────────────────────────────────────────────────────────────┘
SHAMismatch: https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2017.json.gz (have:
9A41A405D604C16E8EE7B04D7DB9812E49F5A93167335F0A954654FF844BE8CC, want:
13EA5C65187399E1F91836316952F496F76108810BD475B4C47A0BF91D8C993B)
Downloading CVEs... ------------------- --------------------  48% 0:00:03

This screen persists and yields nothing.
The tool was working well last week .
I have tried the tool on both Windows and Ubuntu under WSL.

[imsahil007]

After doing a little bit of research I find out there is something wrong with NVD. After refreshing the meta information with a one-second interval I found this output:

lastModifiedDate:2021-03-29T01:00:22-04:00
size:1087082
zipSize:76433
gzSize:76293
sha256:4951BD34DC65248100D328F5637E1B019152DBED1A2374BE37D5DA559CEF9267

lastModifiedDate:2021-03-26T10:00:19-04:00
size:1144916
zipSize:87642
gzSize:87502
sha256:C2312FCF11C5FBDB0E3D832590ABCD243627CA5BD8ED89CA102B9238D429F4BA

Clearly, the first one is right and it is showing meta information for older data. This is the same reason we have been getting SHA mismatch warnings for quite some time.

[param211]

Yes. The meta and actual files on the data feed aren't in sync.
@Adley-Nastri while NVD fixes the issue, you can use the tool with the -u never flag to skip the database update.

[keesj-exset]

Is the tool also not using outdated calls?

https://nvd.nist.gov/General/News/New-NVD-CVE-CPE-API-and-SOAP-Retirement

[param211]

Whoa! An NVD API definitely changes things!

[param211]

NVD data feed is back to normal. SHAMismatch errors aren't showing up now.

[param211]

@terriko we should probably explore switching to the API.
From here

The legacy SOAP services will be retired three months after the CVE and CPE APIs are released. The current retirement date is scheduled for June 1st, 2020.

I'm not sure if this covers the json files on the data feeds webpage (SOAP == XML?). But in the API docs they have already started referring to these feeds as legacy feeds

Legacy Data Feeds
Historically the same information has been available programmatically via data feeds found at
https://nvd.nist.gov/vuln/data-feeds. Consumers of these legacy feeds are encouraged to migrate away
from the feeds and use the new services instead. Despite their popularity, the feeds required downloading
and processing large files even when a subset of data would suffice. This was time consuming and a
burden to networks. In contrast, the new services allow callers to specify query parameters to filter the
response. For instance, you may be interested only in vulnerabilities for a certain time period, for specific
products, and so on.

[param211]

I explored the API a bit, the way it's structured we could download all the CVEs at the initial run of the tool and for subsequent runs update only the updated/new CVEs.

Retrieving All CVE
Presently NVD contains more than 120,000 vulnerabilities relating to thousands of vendor products. If
your goal is to fetch all records, then multiple consecutive requests are required. For example, 120+
requests for 1,000 results per page. However, NIST firewall rules in place to prevent denial of service
attacks on NVD can thwart your application. To avoid this, it is recommended that your application
sleeps for several seconds between requests in order that your legitimate requests are not denied.
In addition, applications are discouraged from repeatedly requesting all the records every day. Rather,
download all of them initially, then use date range parameters to retrieve new and recently modified
records since your last request. See CVE by Date Range.

This might help with the rate-limiting issue too. #1081

[terriko]

Switching to the new setup sounds like a good plan. @param211 do you mind opening a separate github issue with your research on that? Does this look like a few days of work or a whole GSoC project worth of work to you?

[param211]

@terriko sure I'll open a new issue. At first glance, it looks like only some parts of cvedb.py will need to be rewritten. And the JSON structure is unchanged. Probably can get it done within this week.

[terriko]

I think the issue with NVD that caused this bug report is fixed, and we're tracking the plan to use NVD's new api in #1125 so I'm going to go ahead and close this particular issue.