push: hangs after collecting information for a couple of minutes

version: 0.40.0+6408b5

❯ dvc push -r ssh
Preparing to upload data to 'ssh://localhost/tmp/data-storage'
Preparing to collect status from ssh://localhost/tmp/data-storage
[##############################] 100% Collecting information

The user shouldn't think that the process is misbehaving (thinking it is dead because it is not giving any signal of processing stuff -- without checking top)

[efiop]

So what is the cause of that? Did you find out? You are working on that status optimization, maybe it is related to that?

I still don't know, @efiop , just opened the issue to not forget about it

@efiop , A quick update, the problem is with this method:

# dvc/remote/local/__init__.py
    def _fill_statuses(self, checksum_info_dir, local_exists, remote_exists):
        for md5, info in checksum_info_dir.items():
            status = STATUS_MAP[(md5 in local_exists, md5 in remote_exists)]
            info["status"] = status

Usually the operation shouldn't take that much but here, when checksum_info_dir is like 100,002
it can take --on my slow computer-- ~120 seconds (2 minutes), even if it is not that much, it is not a good UI to not notify the user about what's going on.

We can optimize this by using a set because they have O(1) lookup instead of O(n). The difference is outstanding, taking ~0.04 seconds to compute:

    def _fill_statuses(self, checksum_info_dir, local_exists, remote_exists):
        # Using sets because they are way faster for lookups
        local = set(local_exists)
        remote = set(remote_exists)

        for md5, info in checksum_info_dir.items():
            status = STATUS_MAP[(md5 in local, md5 in remote)]
            info["status"] = status

[efiop]

@MrOutis Amazing catch! 🎉