Case Study: NIST 800 53 Mappings

If you prefer video, this same content is available on YouTube.

The primary goal of ATT&CK Sync is to reduce the effort required to stay in sync with new ATT&CK releases. In order to prove the concept, we use the Center’s NIST 800-53 Mappings project as a case study. This project is an ideal case study because it contains a large number of mappings that depend on ATT&CK (over 9,000 line items). Furthermore, we have previously updated these mappings from v8.2 to v9.0 and from v9.0 to v10.1, which gives us empirical evidence about the cost and level of effort to conduct these upgrades. For this case study, we are migrating the mappings from v10.1 to v12.1.

The concept of operation for this case study begins with the ATT&CK Sync changelog, which is created in a machine-readable format (JSON):

Download a sample JSON changelog.

We created a Python script to read the JSON changelog and use that data to mark up the control mappings spreadsheet, adding annotations for each mapping that is potentially affected by ATT&CK changes (such as modified language in the description, new detections, or new mitigations). The script is executed from a command line as follows. (See the Developers Wiki for details on how to set up the command line tool.)

$ poetry shell
$ python -m attack_sync.generate_mapping_excel \
    samples/attack-changelog-v10.1-v12.1.json \
    data/nist800-53-r4-mappings.xlsx \
    data/nist800-53-r4-mappings-output.xlsx

This command parses the specified JSON changelog, reads in the existing mappings Excel, and emits a new Excel containing the original mappings with the pertinent changes between ATT&CK v10.1 and v12.1 appended to each row. This script only reports changes in ATT&CK that are considered material to the 800-53 mappings project; for example it reports changes to descriptions, detections, and mitigations, but does not report changes in contributors, references, or other metadata.

Download sample Excel mappings.

In our case study, The application of ATT&CK Sync quickly revealed that more than 50% of the existing mappings required no review at all as the mapped ATT&CK techniques had no changes at all, and therefore did not require further review. The modified technique descriptions are displayed with red and green text to indicate word-by-word edits, which helps mapping analysts quickly review the changes to assess the impact on the validity of that mapping. Approximately another 20% were easily reviewed as the analysts were able to easily identify that changes were insignificant to the mappings (e.g., typo corrections, new but similar detections). This left about 30% where the associated ATT&CK techniques had changes requiring more in-depth analyst review and research, such as significant changes to technique descriptions and technique deprecations, for potential tailoring to account for the impact of the technique changes on the existing security control mappings. In addition, there were 20 new ATT&CK Enterprise techniques added from v10.1 to v12.1, which also required analyst review for security control selection and the generation of new mappings as applicable.

ATT&CK Sync has proven very effective at reducing the overall cost and effort of updating the NIST 800-53 mappings, allowing the analyst team to quickly trim down review from approximately 9,500 to under 3,500 mappings. This overall update resulted in the removal of 52 of the v10.1 mappings and the addition of 394 new mappings, resulting in a total of 9,794 mappings (4,865 for Rev4 and 4,929 for Rev5). The entire effort was completed in under 200 labor hours, as compared to an average of 800 labor hours for previous NIST 800-53 mapping updates, achieving about 75% reduction of effort.