remote: efficiently collect directories

Version: 0.62.1

Description: The current implementation for _collect_dir is an N+1 operation, where it walks the directory to list all the files and then for each one compute/request its checksum (get_file_checksum).

https://github.com/iterative/dvc/blob/4171aac0294fd316d51558d2593d10ff006221c2/dvc/remote/base.py#L195-L231

The state saves us from getting all the checksums again (the N operation).
However, there are remotes like S3 that have an operation to list the objects with their checksums and other stats (list_objects).

Let's discuss if it make sense to take advantage of this operation, and replace the N+1 (get_filechecksum(file) for file in walk(dir) if not state.get(file)) with the one that returns the list of files with some metadata already.

Related: #1654

[shcheklein]

@MrOutis again, could you please provide an example when it can benefit us? Am I correct that it will be a custom implementation for S3 to calculate a directory hash, for example when we need to use it as an external dependency and/or output?

If that is the case, does it mean, for example that it can be 1 vs N api calls via boto3? do we have some numbers to compare? It's hard to make any decisions w/p any context.

Yes, @shcheklein , this would be a custom implementation for S3.
I'll provide a more concrete example with some numbers when the support for directories on S3 is fully implemented (#2619)

I would leave this here:

    def _collect_dir(self, path_info):
        # See: https://github.com/iterative/dvc/issues/2648
        root = path_info.path

        return [
            {
                self.PARAM_CHECKSUM: entry["ETag"].strip('"'),
                self.PARAM_RELPATH: os.path.relpath(entry["Key"], start=root),
            }
            for entry in self._list_objects(path_info)
        ]

It is the implementation I was thinking for S3

[efiop]

Sounds good @MrOutis ! It would be interesting to see how much performance we are getting from overriding _collect_dir like that.

[efiop]

Stale