loading rules takes a while

[williballenthin]

loading the ruleset of 300+ rules from their yaml files takes around 5s on my laptop. it's minorly annoying when I'm doing analysis, and also prevents me from running capa iteratively against a large corpus of samples.

we should profile the rule loading logic and see if we can make this much faster. running capa probably involves reading and parsing 300+ python files, but this happens within a few ms. so why is rule loading so much slower?

[williballenthin]

looks like yaml parsing is known to be slow:

https://stackoverflow.com/questions/27743711/can-i-speedup-yaml
https://stackoverflow.com/questions/47715566/cannot-load-cloader-with-pyyaml/62543781#62543781

definitely looks like we want to use the CLoader when possible. probably want to avoid using the ruamel parser except when doing linting/formatting. hopefully we can embed the shared object in the pyinstaller distribution.

[williballenthin]

7 seconds just to load the rules!

[williballenthin]

flamegraph generated using https://github.com/benfred/py-spy

profile.zip

% of runtime	phase
51%	loading rules
9%	indexing rules
10%	analyzing binary
26%	matching rules

[williballenthin]

sudo apt install libyaml-dev
pip --no-cache-dir install --verbose --force-reinstall -I pyyaml

[williballenthin]

PyYAML CLoader doesn't like our description syntax:

...and as i look at this, i'm not sure i quite understand how this should be parsed, either. hmm...

[williballenthin]

wow, if i strip the descriptions out, then PyYAML.CLoader loads the rules in around 0.3 seconds:

[williballenthin]

need to figure out #312 before we continue here.

should prefer to use PyYAML.CLoader when possible. hopefully we can embed this within pyinstaller binaries.