Introduce JobQueue and and use it to list files

[Morriar]

The queue is backed by crossbeam::{channel, deque, utils} and provides parallelism, communication through channels and job stealing with backoff.

In this PR we migrate the file listing to the JobQueue, I'll follow up with indexing and resolution.

Benchmarks:

# listing files from core

Branch                       Avg Time (s)  Avg MRSS (MB)
main                         0.842         16.3
ignore-crate                 0.364         23.7
job-queue                    0.306         23.8

# listing files from shop/world/

Branch                       Avg Time (s)  Avg MRSS (MB)
main                         8.776         21.3
ignore-crate                 1.791         33.5
job-queue                    1.517         36.2

Closes #100
Closes #205

[vinistock]

I like this, but still wanted to bring up: does rayon provide us with something like this by default?

I know it has many modes for running code in parallel, so I wonder if bringing rayon back is worth it.

[Morriar]

I like this, but still wanted to bring up: does rayon provide us with something like this by default?

Yes to some extent.

I found that the main difference is the API we can expose. With the JobQueue the idea is to hide the complexity and just make people implement and push jobs. With Rayon the user of the queue has to handle lower level concept like spawning themselves.

In the end I went with the cleaner API to save ourselves some complexity in the usesites.