py-cell and py-notebook

[JeffersGlass]

@fpliger and I had a great extended talk about the py-cell / py-notebook idea that's been kicking around for awhile - here's a proposal based on what we discussed.

Note: This prposal assumes that all execution by py-cells is in Workers. There are some use cases (i.e. the PyScript docs themselves) where a main-thread version could make sense, but I think let's focus on the API for Workers first, and then figure out the adjusted/reduced version for main thread.

py-cell

A py-cell is visual element represented by a CodeMirror Editor. Users may type code directly into the editor, and run it* by clicking an on-screen run button, or by hitting shift+enter. Developers can pre-load code into a py-cell by including code inside the relevant HTML tag, or by using its src attribute.

The py-cell is visually and conceptually similar to the <py-repl> tag from PyScript Classic; much of that code could likely be reused.

*What it means to "run to the py-cell's code" is explored below.

HTML Tag

Question: What should the HTML tag be? Since the py-cell is ultimately a visual element, a custom element like <py-cell> would seem to make sense. However, if we want to have the functionality of developers being able to 'pre-enter' Python code into the cell in-line, then maybe <script type="py-cell"> is better to avoid parsing issues. Or maybe we should ditch the inline-code feature?

<!-- Option 1: Custom Element isn't reliable with in-line Python code -->
<py-cell>
    print("bumme<hr>!")
</py-cell>

<!-- Option 2: Custom Script Type -->
<script type="py-cell">
    print("No <hr> can stop me!")
</script>

py-notebook

A py-notebook is a collection of py-cells which share a common interpreter and its config. This allows developers to link several py-cells together to create multiple cohesive demos on a single page without them interfering with each other.

This page would contain two separate py-notebooks:

HTML Tag and Code Execution

The <py-notebook> custom element is a container for several py-cells (and other content, if desired). The first time a py-cell inside a given py-notebook is asked to execute code, an interpreter of the appopriate type is bootstrapped in a worker (presumably Pyodide, but we can imagine an <mpy-notebook> in the future etc), using the available config, if any (see below)

Once the interpreter is bootstrapped, any time a cell iin this notebook asked to execute code, it uses that same interpreter; i.e. it calls my_interpreter.runAsync(code_in_the_codemirror). Thus, multiple py-cells inside a py-notebook share the same interpreter:

<py-notebook config="config.toml"> 
    <script type="py-cell" id="one">
        msg = "These two cells run in the same REPL"
    </script>

    <script type="py-cell" id="two">
        print(msg)
    </script>
</py-notebook>

Config

The <py-notebook> tag has a config attribute that functions the same as our current script type="py" tag's config attribute: it can take a URL pointing to a JSON or TOML file, or an inline representation of the same.

Question: Should users be able to write a <py-config> tag inside a py-notebook tag and have the notebook use that tag?? It would obviously affect how script type="py" tags find their own configs on the page, since we'd want to ignore py-configs inside notebooks in that case... but if a <py-config> tag is our promoted way of configuring interpreters, it might be nice to write one right into the HTML

Other Thoughts:

1. What happens if a py-cell is not inside a <py-notebook>? One option is to just error; another is to treat it as if it had a "plain" notebook around it:

<script type="py-cell">
  print("foo")
</script>

<!-- ... could be equivalent to: -->

<py-notebook>
  <script type="py-cell">
      print("foo")
  </script>
</py-notebook>

2. Personally, I would find it useful in documentation sites and sites using dynamic frontend frameworks to have multiple py-cells that share a single interpreter that are not confined to being inside a single parent tag. Having an attribute like notebook on a py-cell could accomodate this:

<div>
    <py-notebook config='config.json' id="rich-notes">
        <script type="py-cell">
            from rich.pretty import pprint
        </script>
    </py-notebook>
</div>

<div>
  <!-- For formatting reasons, this is separated in the DOM, but still wants to use the interpreter from the earlier notebook: -->
  <script type="py-cell" notebook="rich-notes">
      pprint("This is pretty printed:", [1,2,3,4])
  </script>
</div>

3. There are some additional features (from Classic and other discussions) we could add here:autogenerate, scripts that run on interpreter bootstrap, save/import/export, etc.. However, I'd like to focus on this core proposal to start with, and get to those in future proposals.

[bugzpodder]

is there a concept of cell orders and re-running cells would trigger rerunning all the downstream cells

[JeffersGlass]

Interesting... there's not currently in this proposal, but we could certainly look at it.

[WebReflection]

A few personal thoughts around this proposal:

• I think the proposal in general looks good
• I think we should start simple ... by simple I mean having <py-notebook> working, ignoring all extra indirections with lonely cells
• it's indeed not clear to me:
  • what's the purpose of a lonely cell if it can't even have its own config?
  • what's the difference between a <py-cell> and a <py-notebook> if not just the fact the former would require tiny less typing around?
• we don't have a mechanism to directly evaluate code at distance in a single worker ... we might use the sync proxy to eval at distance but this is currently unexplored, although it should work
• I have the feeling this is a big plugin and I wonder if it should be a stand-alone project as in @pyscript/notebook as the <script type="py-cell"> is effectively a custom type by its own and <py-notebook> (or <mpy-notebook> and related <script type="mpy-cell">) could work without any py-script around ... sure thing we can use same lazy bootstrap like we do already with py-terminal but as this proposal is big I have the feeling the plugins folder might soon become a bit chaotic to deal with ... true that we can also create a subfolder for notebook and call it a day

thoughts?

[fpliger]

I think we should start simple ... by simple I mean having working, ignoring all extra indirections with lonely cells

+1 on starting simple.... ironically, I think that it's much simpler to just have an "editor" component call it py-cell or anything else (doesn't matter) than a "notebook" component, that should encompass multiple concepts around it (config, collection of cells, history, persistency, how cell and results interact, etc...

A lot of the above is up for discussion but, again, I think we are very aligned on let's start simple

what's the purpose of a lonely cell if it can't even have its own config?

Again, I think the term cell makes the discussion more complicated than it needs because it kind of implies a cell being part of something more. It's just an editor component that can execute the code in it. As such, I'm +1 on actually allowing it have its own config (so you can set up the environment where you'll execute the code and possibly other editor related things)

(in a "for the future" type of discussion, that also plays well with the concept of notebook in the future and it can inject/pass along [part of] the notebook configuration to the "cells" as well

what's the difference between a and a if not just the fact the former would require tiny less typing around?

Jupter notebook are good examples of this. The notebook is a collection of cells. Each cell has code, can require code execution and displays its results (well, it also has editor features like syntax highlighting, autocompletion, hints, etc..). A notebook is a higher level concept that encompasses a collection of cells but also kernels (where you can execute code coming from cells), history, cells precedence, etc...).

we don't have a mechanism to directly evaluate code at distance in a single worker ... we might use the sync proxy to eval at distance but this is currently unexplored, although it should work

What do you mean with "at distance"?

have the feeling this is a big plugin and I wonder if it should be a stand-alone project as in @pyscript/notebook as the <script type="py-cell"> is effectively a custom type by its own and (or and related <script type="mpy-cell">) could work without any py-script around ... sure thing we can use same lazy bootstrap like we do already with py-terminal but as this proposal is big I have the feeling the plugins folder might soon become a bit chaotic to deal with ... true that we can also create a subfolder for notebook and call it a day

When you say "stand-alone project" do you mean "its own repo"? In that case I'm not entirely opposed, since I agree it's a big (and I mean big) plugin that may even need it's own plugin API) but I'd suggest we try to see if just adding a subfolder in the plugins folder first and see how it plays out. Mostly for YAGNI sake (as it often can play both ways... under or over estimating the need :) )

[WebReflection]

I'm OK with everything you said but to answer this:

What do you mean with "at distance"?

I mean you write code or edit code and you click run and that code needs to end up to the pyodide interpreter that is behind a worker,

[fpliger]

@JeffersGlass thanks for the write up! Geeking out on a Friday evening is easy but putting that into a proposal takes effort! 🙏

Given that we discussed it already, one can guess I'm aligned with most of it... So I just wanted to bring a few additional thoughts:

A py-cell is visual element represented by a CodeMirror Editor.

As I mentioned when we talked, I'm not entirely bought on the py-cell name. I mean, it does fit the Notebook narrative but I think these "editor" blocks do serve more use cases than just a notebook-like scenario. As you mentioned, the pyscript-mkdocs plugin for instance.. or in general, just being able to use an editor block that is also able to execute code (and maybe more depending on how you "frame" it) is extremely powerful.

The py-cell is visually and conceptually similar to the tag from PyScript Classic; much of that code could likely be reused.

Think that's a great starting point that we can build from. In general I think we should offer hooks and ways to plug in the py-cell and py-notebook as well (I know no one is surprised that I'm proposing a pluggable API lol ) since I think there's A LOT of functionality that we (and users) may need to add on top of the Editor experience. For instance, being able to create a history of executions (saving code and results), a way to save work, a way to add code completion, etc... I think the design could be quite simple but quite powerful as well

HTML Tag
Question: What should the HTML tag be?

Again, not a big fan of py-cell 😆 . py-editor? py-repl (again)? @mchilvers will probably have better thoughts here....

Also, I'm +0 on the proposal of having a CustomElement and a <script type="....">. As usual, the annoying case is when you have inline code :/

py-notebook

I'm really excited about it and I think we can come up with some really really cool way of delivering custom user experience depending on how users can customize their "notebook" concept. This is something I'd love to brainstorm and design. More specifically, how do we create a flexible framework to allow freedom during the exploration phase and easy to create fancy/powerful presentation of the results without too much effort. I'm sure threre's a sweet spot somewhere.

Notebook config

Yes!

Question: Should users be able to write a tag inside a py-notebook tag and have the notebook use that tag?? I

I'm extremely debated about this one. On one hand I don't like the risk of inflating the page and the number of configs floating around BUT, at the same time, I don't have a good alternative of the flexibility of having everything needed in the page itself (which makes a good case for portability). I think this needs a bit more discovery...

What happens if a py-cell is not inside a ? One option is to just error; another is to treat it as if it had a "plain" notebook around it

-1 on just erroring. I think a py-editor (maybe py-cell makes less sense from a naming pov?) does make a lot of sense on its own. As you mentioned, documentation, interactive learning experience and just having a single page editor does have a ton of use case for a "single editor that is not a notebook"

[WebReflection]

in order to work on this, I need to be sure we're all aligned in what we want:

• a <script type="py-editor"> (or py-cell or ...)
• such script will automatically bootstrap an editor place representing the code inside (if any)
  • if there is a src attribute the editor shows that file/content instead as we do already with other scripts
• the script itself can have a config property that points at a node in the page that is a config already or to an external URL
• the editor has a Run button that executes the code in a worker
  • when such Run is pressed, an extra node would show the output of the operation
    • should Run always clean up on click?
    • should the output grow automatically or just scroll to its end after a new Run is clicked?