output/dependency/remote: support directories for remotes other than local

[efiop]

Currently we only support directories for local remotes. Need to add support for:

• ssh
• hdfs Can't support because the checksum is based on crc's which has a significant chance of overlapping when you have many files in cache, which is unusable for dirs.
• s3
• gs No one asked for this Now someone did ask gs: support directories as external dependencies/outputs #2814

Most likely, this could be achieved in single pass by unifying things like RemoteLOCAL.load_dir_cache into RemoteBase.

[PeterFogh]

My team and I ready need the support of defining directories as output in our setup where we use SSH remote. It would simplify multiple of our DVC stages.

I hope this feature comes soon to SSH😁

[PeterFogh]

@efiop, Is this feature suited for a pull request? Because this is the one feature I need the most. Any recommendations on how to begin on the feature implementation for SSH remote, such as code files, class, function, etc?

[efiop]

@PeterFogh I have a patch for supporting it for all remotes lying around in a half-completed state. Let me see if I can speed it up.

[PeterFogh]

@efiop, that sounds great. I’m just in the mood for contributing to a open source project this weekend😁 maybe I can find an other (and maybe more simple) issue.
I’m looking forward to seeing your solution for this one.

[efiop]

@PeterFogh Sorry for stealing it 😉 Please take a look at help wanted and good first issue labels on our issues list if you still feel like contributing 🙂 Let me know if you need any suggestions.

[PeterFogh]

@efiop, no pressure, but do you think this feature will be released within the next week? I can use your estimate for the planning for my team :)

@PeterFogh , it is on the "Weekly tasks", so I'm sure @efiop is considering working on this, although I can't guarantee it would be released on the next week, at least it will be on a WIP status :)