`dvclive.log`: Support images

[daavoo]

Now that dvc has support for logging images, it makes sense to also add support on the dvclive side.

A few questions arise:

dvclive.log or dvclive.log_image

Should the existing dvclive.log method be extended to accept a new input type (Image-like) or should we add a new separate method ?

For reference:

• wandb has a single log method that handles every type.
• CometML has a separate log_image
• MLFlow uses a separate general method log_artifact
• Neptune has a single log method that can receive images (File.as_image).

Valid inputs

What kind of objects should be accepted to log as images? How would be handled by dvclive?

On the dvc side, the only condition is that the image is stored in a valid format .

On dvclive we should consider what we will accept as inputs and how to handle them.

The more common Image-like objects I can think of are:

• 2D/3D array/tensor (i.e. Numpy, PyTorch)
• Pillow Image
• Matplotlib Figure

In addition we need to consider if we want to also accept a string indicating a path to an existing image file (i.e. dvclive.log_image("foo.png")), for cases where the user takes care of saving the image by itself.

[daavoo]

Should the existing dvclive.log method be extended to accept a new input type (Image-like) or should we add a new separate method?

I'm more inclined to extend dvclive.log to keep the API as simple as possible, although I have no strong preference.

What kind of objects should be accepted to log as images? How would be handled by dvclive?

I think that we should use Pillow internally for saving the images and just accept anything that can be converted to a Pillow image (like the list above).

Not sure if we should consider an already saved image a valid input, as it might get tricky regarding how DVC would handle it in terms of caching (i.e. should we copy or move it in order to let DVC handle the output as configured in the dvc.yaml)

[daavoo]

cc @dberenbaum @pared

[pared]

I don't see a reason why would we need separate methods. It seems to me that in most cases we could determine the way of storing the object by its type.

We need to remember that we might not have the necessary tools for that (if we have vanilla dvclive it might be hard to determine whether a given object is pytorch's tensor).

I think that we should use Pillow internally for saving the images and just accept anything that can be converted to a Pillow image (like the list above).

I am fine with using Pillow, though we need to decide whether we add a dependency to dvclive or create a new setup option dvclive[images].

The biggest problem I see is integration with DVC. This will heavily depend on how we deal with treeverse/dvc#5980 - and actually, we might need to do it on dvc side first, and later adjust here.

[daavoo]

I don't see a reason why would we need separate methods. It seems to me that in most cases we could determine the way of storing the object by its type.

Agree

We need to remember that we might not have the necessary tools for that (if we have vanilla dvclive it might be hard to determine whether a given object is pytorch's tensor).

PyTorch (or others) won't be really needed for the check, I think. You can just implement the check parsing the strings in X.__class__ and once you reach the point where you confirm that it is a PyTorch tensor instance and want to convert, well, it means that you do have PyTorch installed and lazy imports will do the work

I am fine with using Pillow, though we need to decide whether we add a dependency to dvclive or create a new setup option dvclive[images].

Feels to me that Images should be a first-class citizen and hence Pillow (or whatever) a required dependency. No strong opinion, though.

The biggest problem I see is integration with DVC. This will heavily depend on how we deal with iterative/dvc#5980 - and actually, we might need to do it on dvc side first, and later adjust here.

@pared Do you mean that the existing image support might change? As of today, DVCLive would only need to take care of saving images in the correct folder, right?

[pared]

@pared Do you mean that the existing image support might change? As of today, DVCLive would only need to take care of saving images in the correct folder, right?

My concern is how should we treat logged images - when we are logging int/float values we expect that the user will run dvc plot on this file. Each step is saved as entry in tsv file. How do we do that in the case of images? save it under images/{image_metric_name}/{step}? Then how do we display it using DVC? We are unable to compare different images on the same commit.

[daavoo]

My concern is how should we treat logged images - when we are logging int/float values we expect that the user will run dvc plot on this file. Each step is saved as entry in tsv file. How do we do that in the case of images? save it under images/{image_metric_name}/{step}? Then how do we display it using DVC? We are unable to compare different images on the same commit.

I see. I need to try the image support on dvc side a little bit as I didn't try it yet.

[dberenbaum]

Related to #82.

[dberenbaum]

@daavoo What would you expect to log here and what output would be helpful? . Would you want to compare all steps across time? Would you want this logged like live metrics in the html that get updated at each step?