Added shared drive crawler

[Pourliver]

I added the automatic shared drive crawler I had in mind, as shared a little in #137.

This has been done after #140, so its been easier to implement since I knew the download flow and quirks.

I had to create a crawler_config folder since I didn't like the idea of mixing classes definition and plaintext files.

I tried to put more documentation in this class since the core logic may be hard to follow. I've tested it a lot and it works amazingly well, but please do test it.

[Res260]

Idea: log in a file each file, even if not downloaded

[Pourliver]

Idea: log in a file each file, even if not downloaded

To add to this idea, since we talked about it in person, the goal is to keep track of what the crawler is seeing, so we can further improve the default match / ignore regexes later.

[Pourliver]

The PR is ready for another review. I added a new logger that is not from the root logger, so we could log in another file without messing with STDOUT. This new file keeps track of every file we crawled.

[Res260]

Oops commented on the commet instead of the PR:

Besides the exception() log TODO, LGTM!

[Pourliver]

Sorry, but I'm not too sure I understand whats wrong @Res260 , could you please elaborate a bit?

[Res260]

I think I saw you added a log.exception() somewhere else in the code, so I thought it would be easy to implement instead of leaving a TODO in the code.

…

On Wed., Aug. 14, 2019, 13:16 Maxime Carbonneau, ***@***.***> wrote: Sorry, but I'm not too sure I understand whats wrong @Res260 <https://github.com/Res260> , could you please elaborate a bit? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#141?email_source=notifications&email_token=ADPMNL3QRDF5KDJYIF6FRYLQEQ4WPA5CNFSM4IITPNLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4JPPEQ#issuecomment-521336722>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADPMNLYIG64XYKMPOBRMKRTQEQ4WPANCNFSM4IITPNLA> .

[Pourliver]

You're right, I totally missed this one, my bad! Fixed in latest commit.

[Res260]

Lgtm

[Pourliver]

Fixed a small problem with the parser where the last line of each file got truncated, and simplified the match file since */*.doc* can also match */*.docker, which is unintended. It was only meant for */*.docx so I wrote both to the file. We need to avoid trailing *.

[Res260]

TODO: Add the pr changes to changelog.md

[Res260]

Tested, very dank. LGTM