-
Notifications
You must be signed in to change notification settings - Fork 199
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add CPEs to recog fingerprints #172
Conversation
This reverts commit c501b45.
…uct validation. mid-revamp of remapping. add support for hw => h CPEs
ls xml/*.xml | parallel --gnu "./update_cpes.py {} cpe.xml remap.json && xmllint --format --noblanks {} > {}.bak && mv {}.bak {} || echo {}" 2> errors.txt && git diff
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It probably wouldn't hurt to include a link to the CPE dictionary download page ( https://nvd.nist.gov/products/cpe ) or similar reference to make finding the source files easier.
update_cpes.py
Outdated
if vendor and product: | ||
if not cpe_type in cpe_vp_map: | ||
print(cpe_type) | ||
logging.error("Didn't find CPE type %s", cpe_type) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Adding the vendor/product to this output would provide useful context.
For example:
Existing:
a
ERROR:root:Didn't find CPE type a
Possible output
using logging.error("Didn't find CPE type '%s' for '%s %s'", cpe_type, vendor, product)
a
ERROR:root:Didn't find CPE type a for Oracle XML DB
update_cpes.py
Outdated
|
||
def main(): | ||
if len(sys.argv) != 4: | ||
raise ValueError("Expecting exactly 3 arguments; recog XML file, CPE 2.3 XML dictionary, JSON remapping, got {}".format(len(sys.argv) - 1)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Tossing at logging would result in cleaner output.
Traceback (most recent call last):
File "./update_cpes.py", line 200, in <module>
try: sys.exit(main())
File "./update_cpes.py", line 54, in main
raise ValueError("Expecting exactly 3 arguments; recog XML file, CPE 2.3 XML dictionary, JSON remapping, got {}".format(len(sys.argv) - 1))
ValueError: Expecting exactly 3 arguments; recog XML file, CPE 2.3 XML dictionary, JSON remapping, got 0
vs
CRITICAL:root:Expecting exactly 3 arguments; recog XML file, CPE 2.3 XML dictionary, JSON remapping, got 0
using
logging.critical("Expecting exactly 3 arguments; recog XML file, CPE 2.3 XML dictionary, JSON remapping, got %s", len(sys.argv) - 1)
exit(1)
update_cpes.py
Outdated
else: | ||
logging.error("Unexpected CPE %s", final_cpe_name) | ||
|
||
return vp_map |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
May want to add a sanity check here. If no records are imported from the XML then log an error and exit. Ran into this because it seems that I don't have the correct XML file.
This is a proposal for adding CPEs to recog along with code to keep these values maintained. The CPEs added are based off of
service
,os
, andhw
fingerprints in recog that have a vendor and product that map to a known valid combo in the CPE 2.3 dictionary.When present, a version will also be included in the CPE value:
In cases where the original fingerprint would use the interpolated value for the version, this CPE functionality follows the same convention, requiring consumers to interpolate the correct version when emitting the CPE fingerprints, for example:
The CPEs are 2.3 URIs, which is backwards compatible with 2.2.
This doesn't add CPEs to everything. There are many factors that contribute to this:
service
, but not anos
.I don't believe this should have any impact on current consumers because this is just another
param
element with a new name.There are two parts to this PR:
Feel free to review both, however it may be better to examine the CPEs added to the recog XML first.
For using the updating code, you'll need to fetch the CPE 2.3 XML dictionary:
Now run the update code against every XML file in recog, providing as arguments the path to the recog XML file, the path to the CPE 2.3 XML dictionary and the path to a JSON file that maps Rapid7 vendor and product names to their CPE equivalents. I expedite this with
parallel
and save the error output for further remapping or recog bug fixing: