api: get_url() returns path to `.dir` for directory

[skshetry]

Currently, get_url() returns path to .dir files. I'd propose to have some way to parse get_url() and return list or dicts (or, iterator) of files inside the directory (get_url(recurse=True)?).

dvc get . dir/ --show-url works the same way.

[Suor]

I am not a big fan of complicating .get_url(), this was always intended for files not dirs. Maybe we should not support .get_url() for dirs at all? What's the use case anyway?

[efiop]

Note: as we've discussed in #2994 , get_url might not even be able to access the cache url that it shows, so we won't always be able to parse .dir 🙁 User of get_url might download .dir himself and parse it as a workaround.

I would wait for requests from users before going forward with this.

[jorgeorpinel]

I'm adding some notes about .dir cache files as a preamble to document this in the api docs as well. See f4900e8..7d1b469 (part of #931).

UPDATE: OK, added to API as well (24f2d67a, part of treeverse/dvc.org/pull/908).