Pluto.Jl as an environment for literate programming

[Rahuketu86]

I am new to Pluto and Julia ecosystem as a whole. I come from the python world. I recently took the task to learn julia and build a package / library in it.

There are a few tools which has become a mainstay of my toolkit in python. They include nbdev and jupyterbook . Nbdev in particular is a literate programming environment for python which combines utilities to develop python packages , documentation, testing and deployment all in a nice useful comprehensive package based on jupyter notebooks. My first impulse for choosing a personal project was to start by implementing a version of nbdev specific to julia language. After exploring a bit more around the julia eco-system & watching a few videos on youtube, I came to know about Pluto.

I now feel Pluto.jl can become base of a much more robust julia package(/ library) development environment or a literate programming tool. However it still lacks a few things to get the same level of productivity I get with nbdev . I believe an extension to Pluto.jl base package or an utility package built around Pluto.jl and providing a workflow for package management can help achieve the same. I would like to file this issue to put forth some of those ideas into discussion

1. Pluto.jl at present is very much an alternative for jupyter. As such it concentrates on individual julia file without providing any workflow or utility for library development or literate programming

2. Package documentation : An utility which can do bulk export of multiple Pluto.jl notebooks into something like an executable book site(https://jupyterbook.org/start/build.html) or nbdev (jekyll -documentation) site(https://nbdev.fast.ai/) can be incredibly useful

3. Concept of export cells/ flags : If an utility can be built which can introduce concept of flags limiting export / visibility of certain cells . It can go a long way for a more fine-grained documentation and control.

4. Concept of testing +code coverage+ combination with build actions similar to nbdev.

[fonsp]

We believe that you can already use Pluto notebooks to write a package, with testing and documentation! We will soon rewrite PlutoUI.jl to a series of Pluto notebooks as a proof-of-concept, so stay tuned!

What do you think is missing for "Literate programming"? The sample notebooks are a good example of this in Pluto

About 3: some people use a semicolon (;) at the end of their cell to disable output display, and then also hide the output. This creates a 'hidden' cell

[Rahuketu86]

I think in general, it is the overall cohesiveness of experience I get from something like nbdev + jupyter. One good thing about Pluto.Jl is the fact that it already does a few core things very well eliminating need to implement these features separately:-

• Plain Julia format files and reactive kernel ( It pretty much solves a lot of problems mentioned in Joel Grus talk:https://www.youtube.com/watch?v=7jiPeIFXb6U)

  • Advantage files can be version controlled easily
  • Can be imported easily in other modules without a separate export process eliminating need to implement bunch of code and regex done in something like nbdev

• Few things which I would like a literate programming environment to enable

  • Full package development from Pluto.jl ( Even though Pluto.jl is largely focused on exploration, I think with minor tweaks and separate utility package we can achieve this easily.). For me a package development is not just about being able to develop but also enable support for testing, documentation and clean up . This is in context with multiple notebook files

  • I would love to have following core features ( Lot of these ideas are inspired by nbdev exploratory programming paradigm : A good talk for the same is here : https://www.youtube.com/watch?v=9Q6sLbz37gk&t=5s)

    • A consistent table of content and navigation feature in Pluto.jl.[ Something like TOC and Collapsible heading like jupyter could go a long way here). It is a pain to write long notebooks if you don't have a quick and easy navigation.
    • Enable automatic documentation generation from Pluto.jl notebooks.
    • Since we are using notebooks we are writing our code with explanation in markdown + diagrams and plots. If we can develop an utility to export notebooks in a nice documentation site it would be extremely helpful. It should not be just docstring's but more like static export of Pluto.Jl notebook. [ Example : Jupyterbook, nbdev]
    • Limit the cells exported in documentation and Julia files [ notebooks for package development in exploratory mode]
    • Imagine 3 folders : nbs(Raw Pluto.Jl) file, src (cleaned export), documentation in our project structure
      Reason to do the this:-
      • I want to combine code development with exploration I do to figure out the code. Notebook is a good paradigm for it.
      • A lot of times , I do a lot of testing in exploratory mode for which I will write a bunch of functions for visualization , checking intermediate outputs. Testing out different data formats. Reactive style does limit the need of few of this exploration but doesn't completely eliminate it. After I have done this code, I still want to keep a copy of it for my future reference , even though they don't belong in my final production code. Filtering cells based on flags similar to nbdev can be a quick fix. However , I think this type of functionality can be added to core Pluto.jl experience similar to show and hide button. A general tagging mechanism can be used in a lot of different ways by different tools
        • Utility which can control behavior for exporting with raw notebooks saved in nbs
        • Cleaned notebook only have production code ( tagged for export in source cell)
        • Documented html export with a backend like jupyter-book , jekyll/ hugo or something custom for Pluto.jl ( which can work with binder)
      • In this paradigm, I would also write all my test in same notebook as I am building the function. Separate files for test and development doesn't really work well in notebook context.

  • A project template with ( as part of PlutoUtils)

    • Makefile / github action (workflow) template auto generated for new package :- With raw julia notebook this would be much simpler
    • Commands to test the code ( automatically through gh actions?)
    • Command to generate package binary ( may be combining Binarybuilder.jl)
    • Command to export to jupyter notebook.

[fonsp]

Thanks, those are cool ideas! I will experiment with these things myself for the PlutoUI package first, and we will probably have more to talk about afterwards!

We have a notion page where we write design docs and we'd love your contribution: https://www.notion.so/Pluto-jl-a9982e79b7bb4c658e6216c15a9d4cab Can you send me an email? fons@mit.edu

[mmikhasenko]

Is there an update on Pluto => Literate?

I've been using Pluto.jl extensively and almost exclusively since it's fun and have most functionality I need.
Now, slowly discovering the Documenter.jl package that is optimized to work with markdowns, I see benefits of the literate documents as documentation pages.

Btw, where does discussion on the Pluto development happen these days?