Move markdown out of JSON files

[andresriancho]

User story

As a user I complain and say that writing markdown text inside JSON is hard:

• I have to take care of multi-line JSON lists
• My text editor doesn't highlight my markdown text, which leads to more syntax mistakes
• It's harder to edit online by just opening the file in github

Tasks

• Move vulnerability description and fix texts to external files
• Reference those files using this syntax: "fix": { "$ref": "#/files/fix/123" } and "description": { "$ref": "#/files/description/123" }
• Write unittests that make sure that all referenced files exist
• Write unittests that make sure that all files in the "fix" and "description" directories are referenced by at least one JSON

Reviewers

• @m0sth8 is this what we talked about yesterday? Anything else to add?

[m0sth8]

Yes! It's nice and easy to edit markdown files in github

[mdeous]

Has this issue progressed? I can help if you need.

[andresriancho]

Sorry I was unable to do any work on this. PRs are welcome 👍

[mdeous]

Ok, I started working on it, I have extracted all the descriptions and fixes to separate files using the script below (as some fixes are used in multiple vulns, the fix files can't be named after the vuln title, so for now I've decided to name them only after their id). I'll write the tests for this, and then I'll submit a PR.
All that will be left to do after that will be to find a meaningful title for every fix, but I believe this can be done later.

#!/usr/bin/env python
# coding: utf-8

import json
import os
from collections import OrderedDict

DB_DIR = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'db')
DESC_DIR = os.path.join(DB_DIR, 'description')
FIX_DIR = os.path.join(DB_DIR, 'fix')

for p in (DESC_DIR, FIX_DIR):
    if not os.path.exists(p):
        os.mkdir(p)

fixes = []

for f in os.listdir(DB_DIR):
    fpath = os.path.join(DB_DIR, f)
    if not os.path.isfile(fpath):
        continue

    # read JSON file
    with open(fpath) as ifile:
        data = json.load(ifile, object_pairs_hook=OrderedDict)
    title = f.replace('.json', '')
    vuln_id = data['id']

    # write description to markdown file
    description = '\n'.join(data['description'])
    descpath = os.path.join(DESC_DIR, title+'.md')
    with open(descpath, 'w') as ofile:
        print('writing description file: {}'.format(descpath))
        ofile.write(description)
    descref = '#/files/description/{}'.format(vuln_id)

    # store fix for later processing
    fix = '\n'.join(data['fix']['guidance'])
    if fix not in fixes:
        fixes.append(fix)
    fix_id = fixes.index(fix)
    fixref = '#/files/fix/{}'.format(fix_id)

    # rewrite JSON file
    print('rewriting JSON file: {}'.format(fpath))
    data['description'] = {'$ref': descref}
    data['fix']['guidance'] = {'$ref': fixref}
    with open(fpath, 'w') as ofile:
        json.dump(data, ofile, indent=2)

for fix_id, fix in enumerate(fixes):
    fixpath = os.path.join(FIX_DIR, str(fix_id)+'.md')
    with open(fixpath, 'w') as ofile:
        print('writing fix file: {}'.format(fixpath))
        ofile.write(fix)

[andresriancho]

Done! Thanks @mattoufoutu