plot_2d_or_3d_image with colour

[rijobro]

I have my ground truth image with overlaid segmentation. The image is in grayscale and the segmentation is in colour, the two have been blended together and a 2d slice is correctly displayed with plt.imshow. I'd like to log this image to tensorboard, for which I use plot_2d_or_3d_image. Since the image is 3D, a gif is created, but the colour information is lost. I use max_channels=3, but this creates 3 independent images. Would it be possible to enhance the plot_2d_or_3d_image code to allow colours if num_channels=3?

partially related: #3242.

matplotlib

tensorboard

[Nic-Ma]

Hi @rijobro ,

Thanks for the idea, let me try to investigate it ASAP.

[Nic-Ma]

Hi @rijobro ,

I tried to enhance the plot function for RGB colorful 3D image, but unfortunately, it's very hard to set RGB GIF image with PIL.GifImage and Tensorboard, I got many errors in this function:
https://github.com/Project-MONAI/MONAI/blob/dev/monai/visualize/img2tensorboard.py#L35
I think for most common cases, we don't use RGB images in 3D, usually some multi-channel images.
Can we postpone this feature request for now?

Thanks.

[rijobro]

Hi @Nic-Ma thanks for looking into this for me. In searching for a solution I found tensorboardX which seems to do what I want:

from tensorboardX import SummaryWriter
writer = SummaryWriter()
image, label = d["image"], d["label"]
blended = blend_images(image, label)  # CHWD
blended = blended[None]  # add batch=1
writer.add_video("test", blended, epoch, fps=20, dataformats='NCHWT')

If we wanted, this could be incorporated into plot_2d_or_3d_image?

    if d.ndim == 2:
        d = rescale_array(d, 0, 1)  # type: ignore
        dataformats = "HW"
        writer.add_image(f"{tag}_{dataformats}", d, step, dataformats=dataformats)
        return

    if d.ndim == 3:
        if d.shape[0] == 3 and max_channels == 3:  # RGB
            dataformats = "CHW"
            writer.add_image(f"{tag}_{dataformats}", d, step, dataformats=dataformats)
            return
        dataformats = "HW"
        for j, d2 in enumerate(d[:max_channels]):
            d2 = rescale_array(d2, 0, 1)
            writer.add_image(f"{tag}_{dataformats}_{j}", d2, step, dataformats=dataformats)
        return

    if d.ndim >= 4:
        if hasattr(writer, "add_video"):
            writer.add_video(tag, data, step, fps=20, dataformats='NCHWT')
        else:
            spatial = d.shape[-3:]
            for j, d3 in enumerate(d.reshape([-1] + list(spatial))[:max_channels]):
                d3 = rescale_array(d3, 0, 255)
                add_animated_gif(writer, f"{tag}_HWD_{j}", d3[None], max_frames, 1.0, step)
        return

[Nic-Ma]

Hi @rijobro ,

Sounds interesting an idea, let me try to investigate again, will get back to you soon.

Thanks.

[Nic-Ma]

Hi @rijobro ,

I submitted PR to totally add TensorBoardX support and 3D RGB GIF support: #3250
Could you please help review it?

Thanks in advance.