open_url for opening non-git files

[mike0sv]

There is get and get-url in cli, get-url can download files that are not necessary in dvc or git repo.
And there is only open in dvc.api. It would be nice to have open_url, which operates like get-url

[pared]

One thing to consider here is that api.get_url is not the same as dvc get-url which might make things confusing if we implement open_url under this name.

[efiop]

@mike0sv Do you mean something that could work with both git/dvc files and others like s3/gs/etc? Kinda like fsspec's open but with dvc support? 🙂

[mike0sv]

@pared I already tried to use api.get_url before I read it's docs :)
@efiop not sure. My case is that I know I have a file that is stored with dvc, but I only have it's url (not to actual file on dvc remote). And api.open is tied to repo, so I need to extract it from my url somehow. And get-url seems to already have the same logic (if the file is there, download it, if not, do the dvc magic and get it from remote)

[mike0sv]

The dvc support for fsspec will also be very nice, not only for this case

[skshetry]

@mike0sv, how do you get the url of the file?

[mike0sv]

In mlem it is possible to configure dvc as a storage backend for artifacts. When you do this, list of artifacts is written to metafile as a list or relative paths. When you later use that metafile to load model, you need to load those artifacts, and from metafile location + relative path you get the url

[skshetry]

What API do you use to get the url from DVC?

[mike0sv]

For now we don't use dvc for that. Here is the example of what we want to acheive
Inside metafile we have

artifacts: ["artifats/data.csv"]

and file structure like this

...
name
   - meta.yaml
   - artifacts
      - data.csv # <- tracked by dvc

Then, if we push all that to github https://github.com/org/repo and try to load https://github.com/org/repo/name, we will have artifact url https://github.com/org/repo/name/artifacts/data.csv. Or we can put it into s3 and get something like s3://bucket/name/artifacts/data.csv. Actual file may not be there, if we dvc added it and pushed it to remote

[skshetry]

Ahh, so if I understand you correctly, you want to just get a file object for a uri? Does it use dvc's config?

If not, then fsspec.open is probably what you are looking for.

[mike0sv]

Sorry for long delay. The problem is that I dont know beforehand if the file is in dvc cache or not. And if it is in git repo or not. I could of course check this in runtime, but since get-url exists that does that for me, I proposed similar alternative for api.open.
Btw, I seems that get_url doesnt work with urls like https://github.com/org/repo/tree/branch/path, but that might be aonther issue

[dberenbaum]

This seems related to a proposal from @efiop in #6494 (comment). If we make it so that dvc get can get either a repo file or any other URL, we could do something similar in the API.

[mike0sv]

If this will be transposed to dvc.api.open as well, that will do the job