Add Dynamic Image widget

[zorbash]

Hi Dashbit friends.

I am experimenting with a dynamic image widget and wondering whether it'd make sense to submit a pull request to have this be part of Kino / Livebook.

Rationale

Initially I needed it show a video feed in one of my nerves notebooks with:

video = fn y ->
  # Picam  captures stills using the camera module on Raspberry Pi
  Kino.Image.new(Picam.next_frame, :jpeg)
  |> Kino.render

  Process.sleep(100)
  y.(y)
end

video.(video)

My intention was to take a photo every 100ms, then update an image output cell with that image, but rendering Kino.Image appends a new image to the notebook instead.

There are though numerous use-cases, especially educational, where a dynamic image can be used with an svg to visualise an algorithm in an animated fashion. I'm am already working on visualising towers of hanoi and game of life.

Demo

marquee.mp4

Implementation

Currently, my implementation consists of a LivebookWeb.Output.ImageLive view in livebook and a Kino.ImageDynamic GenServer and minor changes in a few other files.

Please let me know your thoughts and thank you for Livebook and Kino 🙌

[josevalim]

Thank you @zorbash for the issue.

I am wondering if instead, we should introduce two new APIs:

1. One that clears the current output
2. Another for running things periodically

This way you can have a loop that clears the screen and adds the image, without it being image specific. :)

[zorbash]

I made a quick attempt to implement (1), which resulted in the following Livebook diff:

diff --git a/lib/livebook/evaluator/io_proxy.ex b/lib/livebook/evaluator/io_proxy.ex
index 5e290db..c4debf7 100644
--- a/lib/livebook/evaluator/io_proxy.ex
+++ b/lib/livebook/evaluator/io_proxy.ex
@@ -199,6 +199,14 @@ defmodule Livebook.Evaluator.IOProxy do
     {:ok, state}
   end

+  defp io_request(:livebook_clear_output, state) do
+    state = flush_buffer(state)
+    send(state.target, {:clear_evaluation_output, state.ref})
+
+    {:ok, state}
+  end
+
+
   defp io_request(_, state) do
     {{:error, :request}, state}
   end
diff --git a/lib/livebook/session.ex b/lib/livebook/session.ex
index 3818d47..cc78b5b 100644
--- a/lib/livebook/session.ex
+++ b/lib/livebook/session.ex
@@ -634,6 +634,11 @@ defmodule Livebook.Session do
     {:noreply, handle_operation(state, operation)}
   end

+  def handle_info({:clear_evaluation_output, cell_id}, state) do
+    operation = {:clear_cell_evaluation_output, self(), cell_id}
+    {:noreply, handle_operation(state, operation)}
+  end
+
   def handle_info({:evaluation_response, cell_id, response, metadata}, state) do
     operation = {:add_cell_evaluation_response, self(), cell_id, response, metadata}
     {:noreply, handle_operation(state, operation)}
diff --git a/lib/livebook/session/data.ex b/lib/livebook/session/data.ex
index 07f2044..db0bb50 100644
--- a/lib/livebook/session/data.ex
+++ b/lib/livebook/session/data.ex
@@ -393,6 +393,23 @@ defmodule Livebook.Session.Data do
     end
   end

+  def apply_operation(data, {:clear_cell_evaluation_output, _client_pid, id}) do
+    with {:ok, cell, _} <- Notebook.fetch_cell_and_section(data.notebook, id) do
+      data
+      |> with_actions()
+      |> set!(
+        notebook:
+          Notebook.update_cell(data.notebook, cell.id, fn
+            %Cell.Elixir{} = cell -> %{cell | outputs: []}
+            cell -> cell
+          end)
+      )
+      |> wrap_ok()
+    else
+      _ -> :error
+    end
+  end
+

and a Kino.clear/0

  def clear do
    gl = Process.group_leader()
    ref = Process.monitor(gl)

    send(gl, {:io_request, self(), ref, :livebook_clear_output})

    receive do
      {:io_reply, ^ref, :ok} -> :ok
      {:io_reply, ^ref, _} -> :error
      {:DOWN, ^ref, :process, _object, _reason} -> :error
    end

    Process.demonitor(ref)

    :"do not show this result in output"
  end

kino_clear.mp4

It works, but with my implementation the animation is not as smooth. Any pointers?

[jonatanklosko]

@zorbash I'm currently looking into a slightly different solution, will let you know how it goes soon!

[jonatanklosko]

I've just crated #49. With the new API the video feed would look like this:

Kino.animate(100, nil, fn acc ->
  img = Kino.Image.new(Picam.next_frame(), :jpeg)
  {:cont, img, acc}
end)

[zorbash]

Thank you @jonatanklosko that was quick and it looks like an elegant solution for my use-cases.

My only note is whether there should also be a Kino.animate/2 where the nil initial value of the accumulator is implied, similar to how Enum.reduce/2 doesn't need the initial value.

[jonatanklosko]

My only note is whether there should also be a Kino.animate/2 where the nil initial value of the accumulator is implied, similar to how Enum.reduce/2 doesn't need the initial value.

Actually it was modeled similarly to Enum.reduce_while/3, which always requires the accumulator. Enum.reduce/2 is quite different, because without the default value it just has different semantics. We could have a version without an accumulator at all, but passing the accumulator is not much work, so I'd stick to a single function for now :)

[jonatanklosko]

@zorbash we need to release new Kino and Livebook versions together, but with Livebook we are waiting for Elixir 1.13, so it may take a little bit. Feel free to try kino main alongside livebook main though :) And thanks for starting the discussion, I'm very happy with the result 🐱