api: high level dvc.api import/export meets data catalog

[Suor]

- what do you do for a living?
- oh, different things, import, export, family business
(any Hollywood smuggler and now a dvc dev)

This feature was first proposed by @dmpetrov and was discussed several times here and there. Recent conversations/debates with @dmpetrov and @shcheklein resulted into some more or less defined approach to it, which I will lay out here.

The need: we want things to be easier reused across repos, so we want some higher level api than just get()/read()/open(), presumably some call, which will return a ready to use dataframe, an instantiated model, etc.

The design

So here goes the design so far. The exporting side will need to list available things and write some instantiating glue code. Like:

# dvc-artifacts.yml
artifacts:
    - name: face_rec_model:
      description: Tensorflow face recognition model 
      call: export.make_face_rec
      deps: ["data/face_rec_model.dat", "data/extra.dat"]

# export.py
import tflearn

def make_face_rec(face_rec_model_data, extra_data):
    # ... construct a model
    return model

Then the importting side might simply do:

import dvc.api

model = dvc.api.summon("face_rec_model", "https://path/to/repo", rev=...)
model.predict(...)

Summon is a good (to my liking) and short name that refers to summoning a genie or a tesla :)

This will provide a flexible base to export/summon anything. Some questions arise about requrements, repeatetive glue code, Python/R interoperability and easing things for a publisher. I will go through this.

Requirements

Instantiating a model or a dataframe will require some libraries installed like pandas or PyTorch, or even their particular versions. In basic scenario we might ignore that - in most cases a user will know what is this about an will have it installed already. The thing is we can't provide it seamlessly since installing python libs inside dvc.api.summon() call will be surprising and might break some things. So deps management will be separate anyway. The possible way:

artifacts:
    - name: some_dataframe
      ...
      reqs: 
        - pandas>=1.2.3,<2
        - otherlib>=1.2

$ dvc summon-reqs <repo-url> <name> 
pandas>=1.2.3,<2
otherlib>=1.2
$ dvc summon-reqs <repo-url> <name> | xargs pip install

Glue code

There are many common scenarios like export csv-file as dataframe, which will produce repetitive glue code. This could be handled by providing common export functions in our dvcx (for dvc export) library:

artifacts:
    - name: some_dataframe
      call: dvcx.csv_to_df
      deps: ["path/to/some.csv"] 
      reqs: 
        - dvcx
        - pandas

In this particular scenario one might even use pandas.read_csv directly. There should be a way to parametrize those:

artifacts:
    - name: some_dataframe
      call: pandas.read_csv
      args:
        sep: ;
        true_values: ["yes", "oui", "ja"]
      deps: ["path/to/some.csv"] 
      reqs: 
        - dvcx
        - pandas

Note that deps are passed as positional arguments here.

Python/R interoperability

This is a big concern for @dmpetrov. This will mostly work for simpler cases like dataframes, there is probably no practical way to make PyTorch models usable in R, there might be for some simpler models like linear regression. Anyway to solve those I suggest providing dvcx library for R, sharing function and param names, so that same yaml file will work for R too. This is to my mind 2nd or even 3rd layer on top of basic summon functionality, so might be added later.

Making it easier for a publisher

Raised by @shcheklein. All this additional yaml/python files might stop someone from publishing things. I don't see this as an issue since:

• making it easier for the publisher will make it harder for a user,
• yaml files as above is probably the most easier way to state what is meant to be reused from outside, the alternative is plain text (more on this below)
• we already have a way to use things not explicetly published with .read()/.open()/.get()

So the "plain text alternative" - one might just write somewhere in a readme - I have this and this things and provide snippets like:

import dvc.api
import pandas

with dvc.api.open("cities.csv", repo="http://path/to/repo") as f:
    cities = pandas.read_csv(f)

This is easy to use, does not require any new concepts and any additional effort from our side.

What's next?

In my opinion the basic summon functionality might be implemented in the near future as having high value/effort rate and more or less clear design.

Still all of this was only discussed in my 1-1s with @dmpetrov and @shcheklein, so some input from the rest of the team would be highly desirable. So @efiop @MrOutis @pared @jorgeorpinel please join.

[shcheklein]

Good one, @Suor . Thanks!

Just to start the discussion, the first question I would clarify is why DVC is not enough and we need to specify (effectively) types and list of exports in a new file.

And related one - if DVC-file is enough, can we let regular existing dvc.api be aware about types? And do not introduce new APIs?

[Suor]

@shcheklein in a general case an exported object might depend on several outs from different .dvc files. Even if we limit ourselves to single .dvc I don't like complicating stage files. We also start something representing catalog with that yaml file.

Existing api calls return strings or file handles if they suddenly start to return something else that would be crazy.

[shcheklein]

(DISCLAIMER: all stuff below is not an attempt to say that @Suor's approach is not right and we should go with DVC-files alone, please, read this as an attempt to challenge some assumptions, get a bit more examples, and understand better the decisions we make)

in a general case

Would love to have real-life examples before we make this decision. I would not overgeneralize until it's not necessary. @dmpetrov you should have some from the model zoo experiment? and may be from some other experiments? Would be great to see more use cases here, more example of what we are trying to solve, etc.

We also start something representing catalog with that yaml file.

not sure I get that point? Do you mean something that improves discoverability? Gives a way to higher level dvc list interface or something? Again, not sure why DVC-file is not enough here (granted with some additional changes).

Existing api calls return strings or file handles if they suddenly start to return something else that would be crazy.

that's true. I didn't mean reusing the same API calls literally. Some modification (or may be even another call) might be required. Probably this is the secondary question to the first one - what is the best way to export artifacts.

[Suor]

I will let someone else do the input )

Looks like a good API design)
I'm still not buy into the idea that this is needed, might be the lack of understanding of the problem. However, I'm down for trying new features.

Some questions:

• How would this change the current deploy workflow? Does it make sense for someone to write a Flask REST API using dvc.summon?

• Who is the "target" audience? In what cases someone would need to summon something?

• With the current desing, is it possible to retrieve a specific version of such artifact?

• When summoning something, what would DVC do? (i.e. What would be the implementation for the current desing?)

Cool stuff, @Suor 👍

[Suor]

With the current desing, is it possible to retrieve a specific version of such artifact?

The rev arg is still there to refer to any git revision same as in other api calls.

When summoning something, what would DVC do? (i.e. What would be the implementation for the current desing?)

With the example yaml file:

# dvc-artifacts.yml
artifacts:
    - name: face_rec_model:
      description: Tensorflow face recognition model 
      call: export.make_face_rec
      deps: ["data/face_rec_model.dat", "data/extra.dat"]

dvc.api.summon("face_rec_model", repo="<repo-url>", rev="<rev>") will:

1. Clone the repo-url to a tmp dir, checkout rev.
2. Read dvc-artifacts.yml and find face_rec key there.
3. Download and checkout all the deps ("data/face_rec_model.dat" and "data/extra.dat") into that tmp dir.
4. Run something like:

sys.path.insert(0, tmp_dir)

# This is extracted from "call" and "deps"
import export
artifact = export.make_face_rec("data/face_rec_model.dat", "data/extra.dat")

sys.path.popleft()

return artifact

1. Clone the repo-url to a tmp dir, checkout rev.
2. Read dvc-artifacts.yml and find face_rec key there.

Wouldn't it be "Read dvc-artifacts.yaml" and then checkout to revision? Checking out before reading dvc-artifacts.yaml might give us a working directory where dvc-artifacts.yaml doesn't exist.

echo "v1" > foo
dvc add foo
git add foo.dvc
git commit -m "add foo v1"

echo "v2" > foo
dvc add foo
git add foo.dvc
git commit -m "add foo v2"

touch dvc-artifacts.yaml
git add dvc-artifacts.yaml
git commit -m "add artifacts"

If you want to access foo v1, checking out before reading dvc-artifacts.yaml will not work.

---

The overlaps between args and deps is a little bit confusing

[Suor]

Wouldn't it be "Read dvc-artifacts.yaml" and then checkout to revision?

The dvc-artifacts.yml resides in source repo, so you need to checkout that first. The rev is specified in an api call, not in dvc-artifacts.yml so single name there point to all available git revisions in a way.

Checking out before reading dvc-artifacts.yaml might give us a working directory where dvc-artifacts.yaml doesn't exist.

Yes, this is the issue. But if you don't have dvc-artifacts.yml or don't have specific artifact referenced there at specific revision then the instantiating code (export.py) won't be there also and you won't know how to instantiate the thing.

It is possible to separate dvc-artifacts.yml from dvc data repo somehow:

# dvc-artifacts.yml in https://github.com/company/dvc-catalog
artifacts:
    - name: face_rec_model
      description: Tensorflow face recognition model 
      repo: https://github.com/company/face-rec
      rev: 1.1 # Or should we allow specifying this in a .summon() call
      call: export.make_face_rec  # refers to export.py in `dvc-catalog` repo
      deps: "data/face_rec_model.dat" # refers to a file managed by `face-rec` repo
    
    - name: cats_dogs_model
      description: Tensorflow face recognition model 
      repo: https://github.com/company/cats-dogs  # other repo
      rev: v2
      call: ...
      deps: ...

This way you can reference artifacts from external sources and wrap them into summonable form. This could be the future extension. This, however, implies at least two source repos (catalog and data) and might be an overkill for many use cases.

There are lots of questions here also arise. Like distinguishing source and catalog revs, code refs in call and overall usability. So let's postpone this.

This extension possibility is however an another argument for separating artifacts decl files from stage files.

[Suor]

The overlaps between args and deps is a little bit confusing

It is. I didn't come up with something better for now.

[dmpetrov]

@Suor thank you for initiating the discussion! This is a really great direction for DVC.

In this issue, only data reusage/getting part is discussed which is the most important part from the implementation point of view. I think it makes sense to discuss all the data catalog scenarios here to make better requirements for this feature.

But first, what makes a set of files a data catalog:

1. Description: just text
2. Attributes: data summary, metrics, other attributes (license, author, input description, etc)
3. Data types

Data catalog scenarios

• [Eyeballing]. Take a look at data/model summary/attributes/metrics without downloading.
• [Discoverability]. Take a look what data (ML models, dataframes, etc) I have (in my repo, in my workspace=list of repos). List objects (all, by type, by attribute/metric).
• [Getting/Reusing]. Get data in one line of code (from bash, Python, R, etc) no matter what type it is.
• [Updating] ("Making it easier for a publisher"). I should be able to update a changed object in the remote repo (ideally in one line).
• [Interaction] While working in Notebook, get data, modify and then upload a new version back.

Example

A special use case: model zoo.

Ideally, we should be able to create a repository with many models like https://github.com/cadene/pretrained-models.pytorch with all the information available through DVC API.

So, instead of reading text README we can use programmatic API:

• List all the models (50+ models)
• Show metrics available
• List all metrics for all the models
• Get description for a model (by name)
• Download a model (as a file from command line or as a PyTorch object from code or Notebook)
• Upload an updated version of the model back to the repo (advanced case)

Analogs

https://intake.readthedocs.io/en/latest/

Interoperability and supported types

@Suor proposing implementing all through Python which limits interoperability.

I'd suggest creating a more universal approach with different types/plugins: there are dozens of data frame types - each can be used from any language. See https://intake.readthedocs.io/en/latest/plugin-directory.html.

Python can be one of the supported types/plugins and all the ML models (like PyTorch) will be used through this type. So, we won't block future interoperability for common types.

Initially, only two types are needed CSV and Python (which covers the rest types initially).

Open questions

1. Should we jump to the implementation of the high-level API/Summon without data catalog features? (It looks like they are related)
   • Can we show enough value without catalog features?
2. What is a minimum set fo features ew need to show the value?

[dmpetrov]

One of the technical challenges that I see today is importing\getting\summoning multi-file resources. For example, to import a PyTorch model you need to get:

1. a weights file
2. a model definition file
3. the model file might have some dependencies to other files (which are not even specified)

@Suor What would be the best way to overcome this issue? Do we need a specific API call? Should we clone the repo and keep it until the importing and instantiation happening. You should have more context on this. Could you please specify this and even open an issue?

[Suor]

@dmpetrov depending on several artifacts is demonstrated in the very first example, they need to be specified though. This is the only way for dvc to know what it should fetch and checkout. I don't know any reliable way to do that implicitly, there are a few ways to make it unreliably though, e.g. monkeypatching open() or inspecting code :).

Depending on any files managed by git is not an issue with current implementation, we do a clone anyway. I guess we will need to stick to that because people will tend to forget some utils.py they import or config.yaml they read along the way, and since these are not programmers but data scientists, who will be exporting stuff, we should design the system to have forgiving, DWIM style.

[Suor]

List all the models (50+ models)

This would look cool in a jupiter notebook.

Show metrics available

I guess we can implement this on top of dvc pipelines somehow. We can also show/visualize the whole pipeline: let people see the data and the code used, and all the data transformations.

This looks like an advanced thing for me.

Upload an updated version of the model back to the repo (advanced case)

I don't think this is possible without breaking a run or an import stage, which has that model as an out. Only possible for add stages, which means this model pipeline is already broken and we don't know how it was build, etc.

@Suor proposing implementing all through Python which limits interoperability.

We discussed this and implementing it whichever way limits interoperability or extensibility. I see how more declarative way will facilitate eyeballing and maybe some alternative implementations for very limited number of data types like csv.

Still not sure this is a good idea to have from the start. E.g. we have 2 ways of doing things instead of one.