Better visualization/interpretation API

[lorenzoh]

The block plotting API can be improved. Right now, it is

• focused on supervised tasks, and
• specific to Makie.jl plotting

Based on blocks we should be able to create an interface that is backend-agnostic (i.e. plot with Makie.jl, show as text, or as image) and usable for different tasks.

Backend-agnostic interface

The backend-agnostic functions are prefixed with show- and

• showsample(method, sample): Show an unencoded sample.
• showprediction(method, input, targetpred): Show a
• showxy(method, x, y): Show an encoded sample, decoded until interpretable
• showoutput(method, x, y, output): Show an encoded sample and a model output, decoded until interpretable

For every function, there is a version that shows multiples, i.e. a vector of unencoded data or a collated batch of encoded data:

• showsamples, showpredictions, showxys/showbatch, showoutputs

The corresponding blocks (e.g. what makes up a sampleblock) can be derived from the learning method. The high-level function are all lowered to the generic showblock(display, block, data). For example, showsample(method, sample) will lower to `showblock(display, method, sampleblock, sample).

Here, display is a backend for showing. Possible backends are:

• Makie.jl for plotting
• Textual

Block types implement a method that indicates that they support a backend. A backend is then automatically chosen given all block types and considering

• which backends the block types support: the backend must be supported by all blocks
• what display options are available: e.g. in a text-only environment, we can't show Makie figures;
• if there are multiple backends that fulfill these conditions, choose the one with highest fidelity, e.g. Makie > Text

Concrete examples

To give some context, let's take an image classification learning method as an example. Here we have sample = (image, label) and sampleblock = (Image{2}(), Label(classes)).

• showsample(method, sample) hence calls showblock(display, (Image{2}(), Label(classes)), (image, label))

  • display here will be the Makie backend, as it both Image and Label should be showable through Makie.
  • For Makie, by default, each block in a tuple of blocks will get its own axis, however some simplifications can be made, e.g. by overwriting showblock(::MakieBackend, ::Tuple{<:Image, <:Label}, _) we could implement a method where the label is put as a title instead of having its own axis. Same logic applies for a segmentation task where the mask could be drawn over an image.
  • In a text-only environment, the blocks could also be rendered as text, e.g. using ImageInTerminal.jl to render the image.

• showxy calls showblockinterpretable(encodings, (ImageTensor{2}(3), OneHotTensor{0}(classes)), (x, y))

  • It applies decodings to the blocks until they are interpretable:
    • ImageTensor is not interpretable by itself and is decoded back into an Image
    • OneHotTensor could be decoded back into a Label and rendered as text, or a showblock could be defined for OneHotTensor{0} that visualizes it as a barplot of probabilities/logits.
  • After the decodings, showblockinterpretable then defers to showblock(display, (Image{2}(), OneHotTensor{0}(classes)), (imagedecoded, y))

Backends

A Makie backend that plots data and creates static figures akin to the current plot* functions

A text backend that draws to a terminal, using fancy printing packages like PrettyTables.jl, ImageInTerminal.jl and UnicodePlots.jl

A interactive Makie backend that augments the figure with interactive sliders, e.g. to show slices of a heatmap or such. Probably out of scope for FastAI.jl itself but could be an extension package.