Advanced pkg manager

[disberd]

This PR implements a modification to the integrated Package Manager that allows one to specify custom commands for adding packages (in a form of strings as you would type them in the julia PkgREPL), allowing people to have more control on the packages they add to a notebook, like:

• Adding un-registered packages
• Adding specific versions of packages
• Adding packages as developed ones, so that one can exploit Revise to track code changes from within the notebook

I have wanted these features since a long time (together with multiple people based on discourse/zulip and github discussions) but was always scared away by the amount of understanding of Pluto internals both on the front-end and back-end I needed to implement this.
As I had somehow more time in these past weeks I slowly tried to understand and implement a change to allow the points above while maintaining as much as possible the convenience of the Pluto PackageManager that we all love.

Disclaimer regarding #2118

I initially thought about starting from dealing with issue #2118 as a way to start dwelling into the PkgManager internals but at the time it looked like https://github.com/GunnarFarneback/RegistryInstances.jl was not registered yet.
Given that, it was a bit pointless to start an implementation as the API might have changed and Pluto could only depend on registered packages. When I decided to go for this implementation, I checked this open issue where it seemed that registration didn't happen yet. I only discovered 2 days ago that the RegistryInstances had been registered since 2 weeks, but I was already quite advanced on this PR so I decided to skip #2118 for a later step.

Some implementation details

I initially though about having a pop-up that would allow one to select an eventual version/repo/path with different fields, but I ended up resorting to a single pop-up that accepts a string that is then parsed to extract package details like the PkgREPL does.
I believe this solution would be more user friendly to users as people are more used to add packages from the PkgREPL.

This new PR internally uses the parsing commands from Pkg.REPLMode so that most of the work (with some additional Compat caveats due to changes in Pkg between 1.6 and 1.8) is done from Pkg itself.

In order to ease this, I created a new struct (PkgData) inside PkgCompat.jl that keeps useful information about a given installed package, like the pkg_str that is needed to parse the Pkg command, the finally installed version in the manifest as well as the function that is used to add the package to the environment (can be only Pkg.add or Pkg.develop) and the Pkg.PackageSpec that is actually used to specify the package details.

On the front-end, a new button has been added to the PkgPopup to allow specifying the custom string to be used using an HTML prompt (that defaults to add $package_name). The string is then sent to julia for validation/error_checking and the result is sent back to the notebook to assess whether the given string is acceptable or not. The validation step only looks at the string without trying to install the package immediately, so one could still provide a random github repository that does not contain a valid Julia package and the validation would still succeed at this step.
If the validation fails due to some ill-conditioned string, the prompt is re-issued forcing the user to change the string until a valid one is provided.

The pkg_str of packages that do not use the default one are stored in the notebook metadata. At the beginning I was trying to implement a way to reverse engineer the pkg_str from the manifest entry and from a custom compat, but for the moment the notebook_metadata solution seems more clean and will also give visual cue about custom packages from reading the source file.

Some usage examples

Custom package commands

To add packages like you used to before, nothing changes. Instead, if you want to provide custom Pkg commands, you can do so using the newly added button:

359a8ca9-017e-436c-aff3-03a60e876d0d.mp4

You noticed that illegal commands re-prompts the user to fix the problem and try to provide some hints on it.
You also see that as soon as a valid custom string was provided, the PkgStatus marker became orange (randomly selected color), and the finally installed version is 0.4.4, satisfying the constraint provided with the 0.4 version specifier in the command.
The marker remains orange for all packages with a custom command string in order to recognize those more easily.

Developing a package

One can also specify a command to develop a specific path:

f982338e-0bac-4677-9440-586dfb3c71dc.mp4

The color for dev'd path is purple to further differentiate with just added packages.

Conclusions

This PR still needs some modifications, but it is usable from my small testing in the current state so I would really love feedback and tests/comments from other people.

TODO

• add Tests
• find corner-case things to fix
• add custom compat (using hyphen-compat for packages that have a range of versions in the pkg_str version specifier
• add DocStrings/comments where needed
• Polish the front-end stuff (Here help/feedback from @fonsp, @pankgeorg, @Pangoraw would be helpful)
• Understand whether precompile statements should be added and where (@rikhuijzer)

[github-actions]

Try this Pull Request!

Open Julia and type:

julia> import Pkg
julia> Pkg.activate(temp=true)
julia> Pkg.add(url="https://github.com/disberd/Pluto.jl", rev="advanced_pkg_manager")
julia> using Pluto

[greimel]

Something that would be nice to have is relative paths. Right now you cannot really use them, because Pluto stores the tomls in some temporary folder.

e.g.

dev ../MyPkg

doesn't work.

[disberd]

Thanks for the feedback.
I'd have to check the feasibility of this now that I am back after some longish holiday :D

[pawbz]

Hi there, it would be awesome to have this feature, I understand that we can start using unregistered packages!!
Please let us know of the status...
Great work ~~

[disberd]

Thanks,
Yeah this has been parked for a while. Next time I manage to speak to @fonsp I'll try to see if it can be pushed forward or it's better to wait for some rewriting of the internals of the pkg manager