Add CPEs to recog fingerprints

[jhart-r7]

This is a proposal for adding CPEs to recog along with code to keep these values maintained. The CPEs added are based off of service, os, and hw fingerprints in recog that have a vendor and product that map to a known valid combo in the CPE 2.3 dictionary.

When present, a version will also be included in the CPE value:

    <param pos="0" name="service.cpe23" value="cpe:/a:vmware:zimbra_desktop:1"/>

In cases where the original fingerprint would use the interpolated value for the version, this CPE functionality follows the same convention, requiring consumers to interpolate the correct version when emitting the CPE fingerprints, for example:

    <param pos="0" name="service.cpe23" value="cpe:/a:vmware:zimbra_desktop:{service.version}"/>

The CPEs are 2.3 URIs, which is backwards compatible with 2.2.

This doesn't add CPEs to everything. There are many factors that contribute to this:

• Not every vendor or product name used in Recog is valid in the CPE 2.3 dictionary. The maintenance code does support manual remapping of vendors and products.
• Similarly, not every vendor or product might be valid or applicable for the given fingerprint type. For example, Microsoft Exchange is a valid vendor and product for a service, but not an os.
• ...

I don't believe this should have any impact on current consumers because this is just another param element with a new name.

There are two parts to this PR:

• Updates to the recog XML adding CPEs where possible
• Python code to maintain the CPE values in recog XML, including a JSON file to support remapping between Rapid7-isms and CPE. Note that the maintenance code is written in python 3 for lack of a ruby XML library that did what I needed, plus I just wanted to do this in python.

Feel free to review both, however it may be better to examine the CPEs added to the recog XML first.

For using the updating code, you'll need to fetch the CPE 2.3 XML dictionary:

curl -s  https://nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz | pigz -d > official-cpe-dictionary_v2.3.xml

Now run the update code against every XML file in recog, providing as arguments the path to the recog XML file, the path to the CPE 2.3 XML dictionary and the path to a JSON file that maps Rapid7 vendor and product names to their CPE equivalents. I expedite this with parallel and save the error output for further remapping or recog bug fixing:

ls xml/*.xml | parallel --gnu "./update_cpes.py {} official-cpe-dictionary_v2.3.xml  remap.json && xmllint --format --noblanks {} > {}.bak &&  mv {}.bak {} || echo {}" 2> errors.txt

[tsellers-r7]

It probably wouldn't hurt to include a link to the CPE dictionary download page ( https://nvd.nist.gov/products/cpe ) or similar reference to make finding the source files easier.

[tsellers-r7]

Adding the vendor/product to this output would provide useful context.
For example:

Existing:

a
ERROR:root:Didn't find CPE type a

Possible output

using logging.error("Didn't find CPE type '%s' for '%s %s'", cpe_type, vendor, product)

a
ERROR:root:Didn't find CPE type a for Oracle XML DB

[tsellers-r7]

Tossing at logging would result in cleaner output.

Traceback (most recent call last):
  File "./update_cpes.py", line 200, in <module>
    try: sys.exit(main())
  File "./update_cpes.py", line 54, in main
    raise ValueError("Expecting exactly 3 arguments; recog XML file, CPE 2.3 XML dictionary, JSON remapping, got {}".format(len(sys.argv) - 1))
ValueError: Expecting exactly 3 arguments; recog XML file, CPE 2.3 XML dictionary, JSON remapping, got 0

vs

CRITICAL:root:Expecting exactly 3 arguments; recog XML file, CPE 2.3 XML dictionary, JSON remapping, got 0

using

        logging.critical("Expecting exactly 3 arguments; recog XML file, CPE 2.3 XML dictionary, JSON remapping, got %s", len(sys.argv) - 1)
        exit(1)

[tsellers-r7]

May want to add a sanity check here. If no records are imported from the XML then log an error and exit. Ran into this because it seems that I don't have the correct XML file.