🎁 Built-in Pkg management

[fonsp]

Watch a 15min video explaining this PR, or read the documentation

Table of contents

1. Motivation
2. Try it out
3. More details
4. Screenshots
5. TODOs
6. Description of changes

1. Motivation

The two main goals are:

1. Package versions inside the notebook file 👉👉 by default!
2. using Example should install Example

This feature is designed with beginner programmers in mind, but I think that it will cover most use cases.

Example

A notebook with only this code:

using JSON

json([1,2,3])

will have the used version number of JSON.jl stored in the notebook file. Loading a Pluto notebook will instantiate from this data. So:

1. Package dependencies are reproducible
2. You do not need to use the Pkg REPL to install JSON

---

2. Try it out!

(Use Julia 1.5, 1.6 or nightly (1.8))

julia> ]
pkg> add Pluto#main
julia> import Pluto; Pluto.run()

Now, open a new notebook and import some packages!

---

3. More details

I want to offer a limited subset of Pkg functionality directly inside Pluto, mostly automated behind the scenes, to achieve these two goals. People who understand Pkg are not the intended audience, but there will be a way to opt out and use Pkg commands like before.

• It will be completely automatic: if you do nothing, a reproducible, self-contained package environment will be set up
• Project.toml and Manifest.toml inside the notebook file as plaintext. The file format will be forwards-backwards-compatible
• By using the magic command Pkg.activate anywhere in your code, Pluto's package management will be completely disabled, and you get the old behaviour. This way, notebooks that already use Pkg calls to manually set up a package environment will continue to work. This also means that we don't need to implement every feature to make advanced users happy.
• The previous PR added a GUI to choose versions. After talking with @staticfloat and @StefanKarpinski, they made the brilliant suggestion not to have a version-picker GUI, but to just use the latest version, with unobtrusive UI. We will still use the inline GUI to show status (when a package can be updated, for example), and maybe also for ] dev.

Notebook load times

This change will also make notebook loading faster, compared to "Pkg.add notebooks" like in https://computationalthinking.mit.edu/ because:

• Notebooks don't need to resolve a manifest from just the list of dependencies. The manifest is fixed and does not need to resolve when you load a notebook.
• Pkg handling is done on the server process, so only one registry update will be triggered if you launch multiple notebooks.
• (Advanced) we can do some tricks to make sure that all homeworks use the same versions of dependencies by modifying their embedded manifests. This means that the second homework loads faster, and we can precompile packages on binder.
• (Advanced) in projects like PlutoBindServer.jl and static notebook exports, we currently can't safely launch all notebooks in parallel because of Adding packages is thread-unsafe? JuliaLang/Pkg.jl#2219 (comment). This PR implements a "Pkg lock" (within the server process), which means that multiple notebooks with plutopkg can launch in parallel, improving export times.

---

WIP screenshots

(i did not have these two packages installed before!)

Schermopname.2021-01-15.om.23.59.31.mov

Schermopname.2021-01-18.om.12.52.47.mov

I LOVE removing documentation, this is from the new PlutoUI sample notebook:

(advanced) Edit notebook environments using Pkg REPL

I am adding a function Pluto.PkgUtils.activate_notebook("path_to_notebook.jl") that activates a notebook Pkg environment in the REPL. It uses FileWatching to sync changes from either side (i.e. Pkg REPL changes are written to the notebook, editing the notebook updates the Pkg REPL env).

Schermopname.2021-03-17.om.15.19.47.mov

For more (outdated) screenshots, see #661

---

5. TODO

TODO - UI

• Write a help page -- https://github.com/fonsp/Pluto.jl/wiki/%F0%9F%8E%81-Package-management
• Link to the help page from the popup
• Update sample notebooks
• Inline GUI mechanism (Pkg no GUI? #661)
• Show a message next to Pkg.activate about disabled PlutoPkg - Show a message next to Pkg.activate about disabled PlutoPkg #1240
• Show available updates in GUI -- Check if updates are available without updating JuliaLang/Pkg.jl#2582
• Update button
• Update terminal output
• Don't show GUI for local modules
• Show installed version with GUI
• Show installed version with GUI in static export
• Show to-be-installed version with GUI
• Autocomplete for all registered packages
• 'Busy' icon next to installing package
• Tell user that registry update is happening
• Detect when notebook restart is needed
• Communicate notebook restart urgency to user
• Experiment with relaying IO to user -- see Built-in Pkg: relay io to frontend terminal #1210
• Show that another notebook is blocking Pkg

TODO - backend

• Ability to restart a notebook
• Add packages when you run using Asdf
• Ability to update packages
• Remove packages when the last using Asdf is gone
• Ignore local modules
• Disable when you use Pkg.activate
• Actually use these versions
• Store Project.toml + Manifest.toml in file
• Pkg.instantiate the loaded notebook environment
• Pkg.resolve the loaded notebook environment and remove the Manifest.toml if that makes the env resolvable
• Remove compat entries for packages that became stdlibs
• Use the backward-forward-compatible format that I came up with after I implemented the backward-compatible one
• Write automatic [compat] entries to Project
• Conversion for old notebooks
• Show notebook while Pkg is being loaded
• Set LOAD_PATH to ["@"]
• Reset LOAD_PATH when PlutoPkg is disabled
• What to do when Pkg throws an exception? (Solution: Write a backup, clear the embedded Project+Manifest and require notebook restart.)
• Backup & continue on corrupted embedded Manifest/Proejct
• What to do with Tables.jl -- stack? requires.jl? copyofrequires.jl? Failed to precompile CSV #895
• What to do when workspace_use_distributed == false? Seems fine to ignore, lets revisit later
• What to do with Markdown and InteractiveUtils? Seems fine to ignore
• What to do when you Pkg.add without Pkg.activate? (solution: disable plutopkg)
• Think about first "New notebook" launch time -- Improve first "New notebook" launch time #1241
• Test on Pkg Julia 1.7
• Benchmark new Dynamic.jl fields, maybe cache

TODO - tests

• Create a custom registry for testing -- https://github.com/JuliaPluto/PlutoPkgTestRegistry
• Tests for auto-installing packages
• Tests for installing packages later
• Tests for loading a notebook with manifest
• Tests for file format -- bijective
• Tests for file format -- backwards compat with Pkg calls
• Tests for file format -- backwards compat without Pkg calls -- should write backup file?
• Tests for file format -- forwards compat

EXTRAS

• Don't recommend restart for stdlibs
• Don't recommend restart when nothing got removed from the manifest
• Update sample notebooks
• Plots.jl can't have a Manifest for every supported Julia version
• Try on windows
• Util scripts
• Hide Pkg io during tests
• A way to update the registry? Nah
• Write Julia version in Project.toml? No. Will be in Manifest.toml starting in Julia 1.7
• What are custom registries? Do I already support that? Maybe someone from Pkg/juliacomputing can help?
• We currently assume that package names are unique. Is this a problem? How can I test that?
• We use unexported Pkg API, what to do with that?
• Get people to try it out

FUTURE

• Show available updates in GUI and warn if it cannot be installed because it is version-bound? -- Compatibility debugging JuliaLang/Pkg.jl#2583
• "Version xx will be installed in this notebook" should respect constraints -- Check if updates are available without updating JuliaLang/Pkg.jl#2582 (comment)
• Factor out Pkg.Types.Context
• What to do when workspace_use_distributed == false?
• Show progress bar?
• Detect broken registry?
• Package from local path
• Package from git repo
• Add no-longer-supported warning for Julia 1.0 - 1.4
• Test "Pluto inside Pluto" on windows -- 5eb81bf (#844)
• Any future Pkg features to take into account?

---

6. Description of changes

Most of the added lines are because I updated our sample notebooks to use the new format, which includes a lengthy Manifest for the Plots.jl sample notebook. (907 additions are for sample notebooks)

Backend

• src/packages/Packages.jl is the most important. It contains the "Pluto package manager".
• test/packages/Basic.jl to get a more high-level overview of the new behaviour.
• src/packages/PkgCompat.jl contains calls to (internal) Pkg API.
• src/notebook/Notebook.jl contains new code to read and write the Pkg data in the notebook file.

Frontend

• frontend/components/PkgStatusMark.js is the little checkmark/icon that appears inline next to a package name. It is not a Preact component, but a real DOM element, because that works better inside a codemirror widget. This file also contains the helper text code.
• frontend/components/PkgPopup.js is the popup.
• frontend/components/CellInput.js contains nasty code to detect using and import statements and insert and update status marks inline. Note that this is just a visual frontend, it has no functional role in the package manager.

[fredrikekre]

using Example when there are multiple packages Example in the registry hangs (from the perspective of the notebook) since the prompt will show up in the terminal waiting for user input.

[DilumAluthge]

using Example when there are multiple packages Example in the registry hangs (from the perspective of the notebook) since the prompt will show up in the terminal waiting for user input.

Is there a way for the user to "drop into" a terminal and provide the user input manually?

[fonsp]

@DilumAluthge @fredrikekre Can you point me to the Pkg source code that asks for input? I can't seem to find it

[fredrikekre]

https://github.com/JuliaLang/Pkg.jl/blob/6c98f54bd16b315e732b2d21e7cb6e2cf1f89b70/src/Types.jl#L939-L947

[fonsp]

@DilumAluthge it looks like the prompt and input happen on the terminal where you launched Pluto, so you should be able to make the choice there, but I don't see an easy way to:

• detect a prompt and show a message in the browser to go to the terminal
• relay the prompt itself to (a terminal in) the browser

I think the best solution for now is to try to set temporarily isinteractive() to false on the server, causing the resolution to error. This would be one of the reasons to use the Pkg.activate opt-out to manually set up an environment.

[fonsp]

I am merging this to main so that more people can try it out, and to rebase other PRs. But please continue experimenting, and continue leaving feedback!

We will probably tag the 0.15 release with the built-in package manager on July 1st, and https://github.com/fonsp/Pluto.jl/tree/release-0.14 can be used to backport fixes to 0.14 in the meantime.

[dralletje]

WHAAAATTTTTTTTTTTTTT

[AshtonSBradley]

In trying this out (woop woop!), I found myself wanting to also use Pkg version syntax like using Makie@0.14.1 to generate a specific manifest.

Possible?

[fonsp]

use Pkg version syntax like using Makie@0.14.1 to generate a specific manifest.

@AshtonSBradley We can break this in two parts:

• the ability to specify a version. I talked about this a bit in the PR description, but we might add it back later at some point.
• inputting a version using that syntax: the notebook file must be a legal .jl file, so not exactly that, but we can do something keyboard-friendly

[Uroc327]

Is there an issue to track the "package from local path" feature?

[fonsp]

@Uroc327 no, feel free to open one!