Switch files collection to use the ignore crate

[Morriar]

Proof of concept showing how we could reduce the time spent listing files using the ignore. I'm sure we can do even better but this already gets us to a better state.

Comparison on shop/world:

Timing breakdown    before              after

Initialization      0.003s (  0.0%)    0.002s (  0.0%)
Listing             5.587s ( 51.5%)    1.755s ( 23.1%)
Indexing            4.970s ( 45.8%)    5.568s ( 73.3%)
Querying            0.293s (  2.7%)    0.272s (  3.6%)
Database            0.000s (  0.0%)    0.000s (  0.0%)
Cleanup             0.000s (  0.0%)    0.000s (  0.0%)
Total:             10.853s             7.597s         

Comparison on corpus/huge:

Timing breakdown    before              after

Initialization      0.004s (  0.0%)    0.002s (  0.0%)
Listing             0.167s (  0.9%)    0.099s (  0.6%)
Indexing           11.354s ( 64.1%)   10.541s ( 62.3%)
Querying            6.184s ( 34.9%)    6.287s ( 37.1%)
Database            0.000s (  0.0%)    0.000s (  0.0%)
Cleanup             0.000s (  0.0%)    0.000s (  0.0%)
Total:             17.709s            16.928s

[st0012]

Is there other libraries that has less transient dependencies? 8 seems a lot
But if not, the time saved is IMO worth it as well

[Morriar]

Is there other libraries that has less transient dependencies?

We can reproduce something manually. See for example what Sorbet does: https://github.com/sorbet/sorbet/blob/master/common/common.cc#L289-L444

I also experimented with a combo of walkdir and globset but I think ignore comes with a nice API and quite a lot of features (including Windows support?).

Though this PR is merely suggesting that we can do better.

[vinistock]

We may want this one to be false. For example, Prism auto generates the node.rb file. When working on Prism, you'd still like to get features for it.

[Morriar]

The git_ignore was a good way to get node_modules and stuffs out of the way though.

iirc in the current indexer we have a config file .index.yml should we bring that back now so we can configure all of this?

[vinistock]

Yes, ideally configuration. I'm not sure if the answer is a file for the indexer or a configuration API, but something for sure.

[vinistock]

We may want to tune the generation of the synthetic corpus with deeper nesting of directories. I'm surprised that the ratio of time spent listing files is so much smaller in the corpus vs shop/world.