ignore: dynamically collect dvcignore

[efiop]

This allows us avoid collecting dvcignore for the whole repo if we only
care about particular paths. As a result, in a repo with 2 datasets
(2M + 0.5M files), creating a defunct stage takes ~4sec on 1.2.0, but
~1sec(most of it is actually dvc module initialization) with this PR.

This is also a pre-requisite for dynamic dvcignore and subrepo
collection (#4247) while walking
the tree.

Also, it is important to clarify that regular dvc status(without
arguments) has the same performance after this PR, because when we check
dataset for changes, we call things like tree.exists(), which call
dvcignore and make it collect dvcignore in the dataset itself, so we
still endup collecting dvcignore for the whole repo (including walking
into the datasets). This should be solved soon by telling dvcignore that
it shouldn't walk into the datasets searching for .dvcignores.

• ❗ I have followed the Contributing to DVC checklist.

• 📖 If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here.

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

[efiop]

@karajan1001 Would really appreciate if you could share your thoughts about this 🙂

[karajan1001]

Maybe here we also need

        ignore_pattern = self.ignores_trie_tree.get(dirname)
        if ignore_pattern:
            return

to prevent it from running multiply times. But if nowhere else except _get_trie_pattern calls it, it is not necessary.

Seems nowhere else calls it.

[efiop]

Good point! Indeed, for now _update is only called in _get_trie_pattern, so that is already handled for us.

[karajan1001]

@efiop
It's a wonderful work. It saves lots of time when we are scanning parts of the repo.

But I have a question that in

As a result, in a repo with 2 datasets
(2M + 0.5M files), creating a defunct stage takes ~4sec on 1.2.0, but
~1sec(most of it is actually dvc module initialization) with this PR.

There are lots of directories as well? If not so, it would a surprise to me that it took so much time after I had changed

dirs[:], files[:] = self(root, dirs, files)

to

dirs[:], _ = self(root, dirs, [])

preventing useless ignore check on all files.

[efiop]

@karajan1001 Your dvcignore implementation works flawlessly and is extremely efficient! 🙏 The thing that was taking time is that before, when dvcignore was initialized, we would walk into those datasets looking for dvcignores and no matter if there are lots of directories in it or not, os.walk still has to do listdir() for each directory, that has to list all of the files inside it, which takes a while with such a large number of files. That issue is actually a bigger problem for dvc, we will solve it by later telling dvcignore to not walk into datasets searching for dvcignores, but for now this dynamic collection mitigates that pretty well as an added bonus.

Thank you so much for your amazing work on dvcignores, what you've built is very neat, I'm truly enjoying working with the architecture you've created. 🙏

[karajan1001]

@efiop
I tried 1M files on my computer.

It takes considerable time.

This PR also improves performance on #4282.

[pared]

post-merge LGTM