adding publisher datetime and updating nvd feed download #571
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
nvd.go and xml.go: added publisher datetime from xml feed and updated the XML feed download to complete before moving onto downloading the next feed
Reason:
The NVD site frequently resets connections when downloading the feeds if they are not downloaded in a timely manner. When all the feeds are requested at once but then not downloaded immediately the NVD site will reset the connection. It is more reliable to download each after making the request instead of opening all the connections then downloading them one at a time.
Also added is the Publication Datetime from the XML. This field is useful for compliance as many vulnerability programs for PCI base their remediation priority off this date field.
Example feed error:
{"Event":"could not decode NVD data feed","Level":"error","Location":"nvd.go:93","Time":"2017-08-25 18:48:15.515078","data feed name":"2014","error":"read tcp 192.168.10.9:54476-\u003e129.6.13.177:443: read: connection reset by peer"}
NOTE: This is a ref to this PR. Just a bit more cleaned up. #496