implement per notebook env in backend

[Roger-luo]

This PR implements per notebook env in the backend. I think no matter what you want to do with the frontend (#142 ), this always needs to be handled in the backend.

What this PR does is basically attach a new field project to every Notebook object. It's an empty string by default, thus the default behavior is the same as current. But now one can set the process environment via Pluto's API: open and run, e.g

run(;project="project path for all notebooks opened in this Pluto server by default")
open(session, notebook_path; project="this notebook's project path")

Or you can set the default global environment used in a Pluto server process via JULIA_PLUTO_PROJECT or PLUTO_PROJECT environment variable, where
the priority is set to the following:

1. the explicit project kwargs in all Pluto APIs
2. JULIA_PLUTO_PROJECT
3. PLUTO_PROJECT

I will implement the custom local environment feature in pluto CLI after this gets merged. I implemented the custom local environment feature in Pluto CLI: fonsp/PlutoUtils.jl#9 I think the Pluto CLI should be able to share a bunch of environment handling in Comonicon. I think this should resolve most of the problem in #142

[Roger-luo]

@fonsp any comments on this?

[fonsp]

Interesting! Why not?

Should we check for that?

[Roger-luo]

Sorry, I might be too brief in the note, I wrote this to explain why I force all workspace process to start with a --project flag so it will never inherit server process environment implicitly.

I think this behaviour is more natural, since in principal we don't assume different notebook shares the same environment at the backend (unless it is declared explicitly in the frontend).

[Roger-luo]

The other solution is to inherit the environment if project === nothing is true. But I find this is not something I'd intuitively expect when I work with notebook

[fonsp]

Does it currently inherit the environment? That's not good

[Roger-luo]

you mean master branch or this PR?

1. it inherits from the environment in master
2. it does not inherit in this PR

I was just wondering if I should preserve master branch behaviour (since it might break things if not), but then I realize it's not intuitive to inherit the environment.

[fonsp]

No that was unintended! It should not inherit - thanks for spotting the mistake

[fonsp]

I guess this is more a comment about Julia but can we use a longer name?

Is project_environment_path appropriate?

[Roger-luo]

yeah, project_environment_path sounds better

[fonsp]

Pluto already has a default ENV system that we can use, get_pl_env. This means that we only use the ENV name "PLUTO_PROJECT" (although see my later comment about the name).

Since we are making the naming decision right now, we should just pick a single ENV name instead of two different ones.

[fonsp]

If there's a convention that I should have used "JULIA_PLUTO_*" for all ENV names, then I'm okay with changing the names of all current pluto ENV variables. It's undocumented and probably not used by anyone

[Roger-luo]

I think the reason is we should provide environment variable starts with both PLUTO and JULIA_PLUTO just in case. Since this name PLUTO could be used by other things too. What do you think?

Pluto already has a default ENV system that we can use, get_pl_env. This means that we only use the ENV name "PLUTO_PROJECT" (although see my later comment about the name).

I could update this too, since I'm also a bit concerned about the way it is implemented: current implementation will break the ability to create a distributable binary from Pluto using PackageCompiler, and I think at some point, we will need to do it at least for the CLI when things get more complicated.

[fonsp]

About naming, I know that Julia just calls it "project", but I would rather call it pkg_environment_path. A longer name, but much clearer. What do you think?

I think that at least the word "path" needs to be in the name.

[fonsp]

And one more comment: can we use nothing instead of "" to denote that no environment is specified? Or does it need to be ""?

[Roger-luo]

I used "" mainly because I don't need to use multiple types for this field in this way, but this is probably not necessary, especially AbstractString was used for other fields. Yeah, I can change it nothing.

[Roger-luo]

@fonsp I renamed the field to environment_path, since after thinking about it more carefully, I find that the notebook environment is not strictly a project environment, it can also be a shared environment. It's fully depends on what the frontend passes to it. So I think environment_path would be a better name for this.

I also changed the default to nothing too.

for the env variable comment, I think I'll just open a separate PR to polish this part directly later, given it's not quite related to this one.

[Roger-luo]

@fonsp I implemented a new set of tools to handle env variables (but not configs tho) in #367 I'll do a squash later on that PR, after this gets merged.

[Roger-luo]

Is this good to merge now? @fonsp

[fonsp]

I'm just working on something else and then I'll merge!

[Roger-luo]

thanks! just let me know if there is anything else need me to revise.

[fonsp]

I made some changes - what do you think?

[Roger-luo]

yeah looks good. I should have used a more compatible way to write it. I'll move on to the other PR after this gets merged. @fonsp

[fonsp]

Thanks again!

Sorry if I edited the PR too much - I deleted some configuration options with the idea that we want to keep flexibility for #366

[Roger-luo]

Yeah, I'm good with your edits @fonsp I think this PR only aims to implement the feature first, so there is something you can keep track of.

I'm polishing the environment variable handling and trying to allow control more compiler flags to the workspace process in #367 and make the environment variables of Pluto more consistent. I'll merge your edits to that PR later, after this gets merged.

[fonsp]

I haven't actually tried this PR. I guess the checks are:

• Does it launch your notebook inside the global environment, even if the notebook server was launched from a different env? (Not sure how to @test this...)
• Is the ENV variable used?
• Is notebook.environment_path used?

[Roger-luo]

I don't know how to test this either... I guess to test this either we need mock workspace function or somehow execute the test in a workspace. I tried this PR tho, my notebook at least shows the correct project environment.

[fonsp]

Which of the three did you test?

[Roger-luo]

@fonsp I can check things above manually, but I'm not sure how to write them as tests.

if you start Pluto.tun via JULIA_PROJECT=@. julia in Pluto directory, before the notebook will use Pluto's Project.toml but now this is correct

if you start with PLUTO_DEFAULT_ENVIRONMENT_PATH=test julia --project, it is correctly using the specified environment

if we open a notebook directly under some project

using Pluto
s = Pluto.ServerSession()
nb = Pluto.SessionActions.open(s, "sample/Basic.jl"; project="sample")
Pluto.run(;session=s)

The notebook will now start at the specified project directory ("sample/Project.toml" in this case)

[fonsp]

Perfect, thanks for checking!

[marius311]

I have JULIA_PROJECT=@. and prior to this PR, if I opened a notebook, Pluto would automatically activate the environment for the folder that notebook is in (which was exactly what was desired). After this PR though, its always using the global environment, no matter where I start Pluto from or what folder the notebook is in. I tried JULIA_PLUTO_PROJECT=@. and that doesn't work. Is this intended?

[Roger-luo]

The behaviour that JULIA_PROJECT=@. does not effect the notebook process is intended. But I think @fonsp changed the name of the env variable in this PR (to PLUTO_DEFAULT_ENVIRONMENT_PATH, tho I personally prefers JULIA_PLUTO_PROJECT too), which I think should be removed by #366 later, then I'll support this environment variable in the CLI, and one should specify the environment as a kwargs of Pluto.run in REPL, if you are not using CLI. @marius311

[marius311]

Ah, thanks. PLUTO_DEFAULT_ENVIRONMENT_PATH=@. recovers the old behavior, which is what I wanted in my case.

Two minor comments, feel free to consider or not:

• Assuming it was also intended to make the Pluto default always be the global environment rather than @. relative to the notebook (that's what I'm seeing at least), I tend to think the old behavior made more sense, since if you're opening a notebook in a folder with a Project.toml, there's a high probability you wanted to use that environment (and it matches the IJulia default).

• Since most Julia users would know about JULIA_PROJECT but maybe initially not PLUTO_DEFAULT_ENVIRONMENT_PATH, wouldn't it be better that if PLUTO_DEFAULT_ENVIRONMENT_PATH wasn't specified, then it would default to JULIA_PROJECT, and then if that wasn't specified it defaulted to whatever you guys chose for Pluto?

[Roger-luo]

Assuming it was also intended to make the Pluto default always be the global environment rather than @. relative to the notebook (that's what I'm seeing at least), I tend to think the old behavior made more sense, since if you're opening a notebook in a folder with a Project.toml, there's a high probability you wanted to use that environment (and it matches the IJulia default).

@marius311 note when you run Pluto.run, you are not running a notebook, but a server process, which probably doesn't exist in either global shared environment, or @.. inheriting JULIA_PROJECT can as a result starting a environment that does not contain the notebook project e.g in the CLI case, as an application or the system admin would like to spawn the server process in a local project so it won't effect other things on the server. Thus we intentionally disable this.

Conceptually, the notebook process is not the same with server process, thus it should not inherit environment from server process or share the same environment variable. So it should be distinguished from JULIA_PROJECT which configs the server process only, so we know the user actually want to set the project in server process not notebook process.

I think for Pluto.run, the user should only has one way to configure everything in REPL, thus all environment variables will be removed based on #366 and if the user start Pluto in CLI, then the environment variable will be handled there.

Assuming it was also intended to make the Pluto default always be the global environment rather than @. relative

The default will be @. based on quite a few people's comments (actually I sometimes feel the julia compiler should do this too, but...). It's not @. currently mainly because we need to finish the new config system to make this happen, also because currently the notebook environment does not recognize @. so it's not expanded correctly. I'm trying to push this to happen ASAP, but need some time.

Since most Julia users would know about JULIA_PROJECT but maybe initially not PLUTO_DEFAULT_ENVIRONMENT_PATH, wouldn't it be better that if PLUTO_DEFAULT_ENVIRONMENT_PATH wasn't specified, then it would default to JULIA_PROJECT, and then if that wasn't specified it defaulted to whatever you guys chose for Pluto?

if you are using REPL, eventually there will only be one way to config Pluto via the kwargs of Pluto.run (as preferred by @fonsp too). I think for CLI, I will prefer to name the the environment variable as PLUTO_PROJECT and JULIA_PLUTO_PROJECT, and you will also have an option --project to specify the project path, but by default it starts from @., which fallbacks to global if no local environment exists.