feature: adding multiprocessing support

[domanchi]

I never knew parallel processing could be so easy. ❤️ python

Summary

This increases the efficiency of scans by throwing some parallel processing to it.

Since scanning files are independent of each other, it seemed like a prime candidate for this. I considered parallelizing line scanning, but then realized that this may not be such a good idea, since lines are somewhat dependent on each other (e.g. context-based filters).

Testing Done

I ran this on our internal monolith, and these were the stats I got:

# Some stats on the repository itself
$ git ls-files | wc -l
35519
$ cloc $(git ls-files)
...
--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
...
--------------------------------------------------------------------------------
SUM:                          22948         660907         422368        3680522
--------------------------------------------------------------------------------
$ lscpu
...
CPU(s):              16

# Running with single process
$ time detect-secrets scan > /tmp/single.baseline
real    23m41.917s
user    23m20.518s
sys     0m3.527s

# Running with multiprocessor
$ time detect-secrets scan > /tmp/mp.baseline
real    6m4.887s
user    37m4.611s
sys     0m8.304s

# Comparing results
$ diff /tmp/single.baseline /tmp/mp.baseline
4961c4961
<   "generated_at": "2021-04-01T14:42:50Z"
---
>   "generated_at": "2021-04-01T15:05:30Z"

That's like a 74% optimization. Heck yeah!

[calvinli]

I took a quick peek and it does look like scan.scan_file is independent/doesn't rely on any shared state so this should work.