Use direnv environment

[diktomat]

Check for existing issues

• Completed

Describe the feature

Zed should support direnv, to enable using a different environment (espeacially $PATH) for different projects. In combination with #4978 this would enable e.g. using different versions of language servers and tooling, or tooling just installed for the project instead of system-wide using Nix.

If applicable, add mockups / screenshots to help present your vision of the feature

Inspiration from other editors:

• wbolster/emacs-direnv
• misuzu/direnv-subl
• direnv/direnv.vim
• direnv/direnv-vscode

[szlend]

A few issues that almost every editor suffers from when implementing direnv support:

• Direnv initializes too late, causing problems with certain extensions that rely on direnv to set the PATH (e.g. for utilities like language servers and linters). Ideally there was a way for the direnv extension to hook into some really early phase of initialization where it can set environmental variables before other extensions initialize.
• When you already have a project open and you open another project in a new window, the environment in the new window gets inherited from the first window. Normally this isn't a problem because it's just global system environment variables. But with direnv it's annoying because the second project will inherit environmental variables from the first project, potentially loading incorrect versions of linters, language servers, etc. Ideally the editor would save the state of environmental variables before applying direnv. And any new window that you open would inherit from that saved state.

Just something worth considering when designing the extension API and implementing this feature.

[misuzu]

• Direnv initializes too late, causing problems with certain extensions that rely on direnv to set the PATH (e.g. for utilities like language servers and linters). Ideally there was a way for the direnv extension to hook into some really early phase of initialization where it can set environmental variables before other extensions initialize.

Yep, the extension API must have support for this usecase to properly implement direnv integration. This is the reason why https://github.com/misuzu/direnv-subl is flawed - ST just doesn't have the necessary API. The direnv integration that just works would be The Feature that can ease the pain of setting up any project for many people.

[szlend]

Another useful feature that I think a direnv extension should have is to be able to re-load the environment so you don't have to restart your editor. The way this works in vscode for example is that it changes the process environment and then restarts the extension host (and all language servers as a result).

Also initializing direnv can take some time. For example if you use direnv to load a nix shell, it might actually start downloading packages or even compiling. So it might make sense to show some sort of indicator that direnv is still evaluating something. And a way to cancel it.

[jeremylightsmith]

This is definitely keeping us from using zed on our project. We use direnv + nix, and we can't use tasks w/o better support.

With VSCode, it's quite easy to use direnv, you just have to launch the editor from a directory that direnv has already run with code .. Would it be possible to have the same behavior with zed?

[jeremylightsmith]

it seems the termina doesn't maintain its env between runs, because if it did, we could work around this by sourcing a script like this:

#!/bin/bash

if [ -z "${DIRENV_INITIALIZED}" ]; then
  echo "Loading direnv..."
  eval "$(direnv export bash)"
  export DIRENV_INITIALIZED=1
else
  echo "direnv is already loaded."
fi

[mersinvald]

I found a workaround that is semi-usable: running zed with zed --foreground ./ after loading the environment in shell works, the editor plugins can access my nix shell environment, but it's far from ideal: this way you can only have one project/environment open at any time.

If I try to open a second project with a different env, with --foreground I get zed is already running error, and without the flag the environment is shared between all windows.

[mrnugget]

When you open a project, we spawn a login shell in that project's root dir, get the env from that process, and store it to be used with some language servers.

It sounds like you have direnv/nix setup so that would work, is the problem now that it doesn't work with tasks?

[jeremylightsmith]

Thanks @mersinvald , but your workaround doesn't work for me, specifically with tasks, which is my problem. I need to be able to run the test under my cursor from my IDE. And when I open up a task to do that, it doesn't have any of the env that direnv setup :(

[szlend]

With VSCode, it's quite easy to use direnv, you just have to launch the editor from a directory that direnv has already run with code .. Would it be possible to have the same behavior with zed?

This definitely does not work well in vscode, except for the initial window. The direnv vscode extension fixes this and makes it somewhat usable, but it still causes cross-project env pollution from the initial vscode window.

I think how Zed does it is a step in the right direction, though it's difficult to diagnose issues when something goes wrong. It might be a good idea to document the env loading behavior and some steps to help diagnose issues (e.g. how do I inspect the window's env, what login shell Zed used to initialize the env, etc.)

[mrnugget]

I think how Zed does it is a step in the right direction, though it's difficult to diagnose issues when something goes wrong. It might be a good idea to document the env loading behavior and some steps to help diagnose issues (e.g. how do I inspect the window's env, what login shell Zed used to initialize the env, etc.)

Right now that's all printed in the logs (zed: open logs). So when you open Zed, you should see that we do the same in the home directory.

But yes, documentation is a good idea.

Not sure how Zed figures out which shell to use to say if this is a problem or not though.

It uses SHELL and falls back to passwd lookup if SHELL isn't set.

It doesn't spawn nix shell. It uses your $SHELL.

This is what we do on start, in $HOME:

zed/crates/zed/src/main.rs

Lines 309 to 312 in 05b6581

	{
	load_shell_from_passwd().await.log_err();
	}
	load_login_shell_environment().await.log_err();

And this is what we do for the language servers, per project:

zed/crates/project/src/project.rs

Lines 11636 to 11665 in 05b6581

	// What we're doing here is to spawn a shell and then `cd` into
	// the project directory to get the env in there as if the user
	// `cd`'d into it. We do that because tools like direnv, asdf, ...
	// hook into `cd` and only set up the env after that.
	//
	// In certain shells we need to execute additional_command in order to
	// trigger the behavior of direnv, etc.
	//
	//
	// The `exit 0` is the result of hours of debugging, trying to find out
	// why running this command here, without `exit 0`, would mess
	// up signal process for our process so that `ctrl-c` doesn't work
	// anymore.
	//
	// We still don't know why `$SHELL -l -i -c '/usr/bin/env -0'` would
	// do that, but it does, and `exit 0` helps.
	let additional_command = PathBuf::from(&shell)
	.file_name()
	.and_then(|f| f.to_str())
	.and_then(|shell| match shell {
	"fish" => Some("emit fish_prompt;"),
	_ => None,
	});
	let command = format!(
	"cd '{}';{} printf '%s' {marker}; /usr/bin/env; exit 0;",
	dir.display(),
	additional_command.unwrap_or("")
	);

[musjj]

It looks like that rust-analyzer can now pick up the binary from the environment: #12418. So I added this to my config:

{
  "lsp": { "rust-analyzer": { "binary": { "path_lookup": true } } }
}

But it's still tries to download and run its own binary anyways (and failing because I'm on NixOS).

[mrnugget]

But it's still tries to download and run its own binary anyways (and failing because I'm on NixOS).

Yes, because that commit you linked to hasn't been in any release yet.

[mersinvald]

@mrnugget it doesn't work for me without the foreground hack. If I just open the project, the env from .envrc wouldn't be loaded for anything except the integrated shell.
With foreground, both plugins and tasks work fine, as zed is loaded within the direnv-initialized PTY.
I have cargo and rustup setup only in the flake shell, so it's easy enough to test.

When you open a project, we spawn a login shell in that project's root dir,

Is there a way to modify what runs in this shell before editor initialization?

[mrnugget]

@mrnugget it doesn't work for me without the foreground hack

what doesn't work? Only certain language servers make use of this environment right now.

[mersinvald]

@mrnugget it doesn't work for me without the foreground hack

what doesn't work? Only certain language servers make use of this environment right now.

I would like the editor to respect the directory env regardless of what specific plugin needs it. In my case, the issue is with rust-analyzer.

Log without the foreground hack (neither rustup or cargo are installed in the system, only in a project flake)

Support/Zed/languages/rust-analyzer/rust-analyzer-2024-06-11", working directory: "/Users/XXXXXX", args: []
2024-06-11T12:58:46+04:00 [ERROR] Language server rust-analyzer-2024-06-11 (id 1) status update: Failed to load workspaces.
2024-06-11T12:58:46+04:00 [ERROR] Language server rust-analyzer-2024-06-11 (id 1) status update: Failed to load workspaces.
2024-06-11T12:58:47+04:00 [INFO] Language server with id 0 sent unhandled notification LogMessage:
{
  "level": 0,
  "message": "[DEBUG] [agent] [2024-06-11T08:58:47.175Z] Telemetry initialized",
  "metadataStr": "[DEBUG] [agent] [2024-06-11T08:58:47.175Z]",
  "extra": [
    "Telemetry initialized"
  ]
}
2024-06-11T12:58:47+04:00 [INFO] reload git repository ".git"
2024-06-11T12:58:47+04:00 [INFO] reload git repository ".git"
2024-06-11T12:58:47+04:00 [INFO] add connection to peer
2024-06-11T12:58:47+04:00 [INFO] add_connection;
2024-06-11T12:58:47+04:00 [INFO] waiting for server hello
2024-06-11T12:58:47+04:00 [INFO] got server hello
2024-06-11T12:58:47+04:00 [INFO] set status to connected (connection id: ConnectionId { owner_id: 0, id: 0 }, peer id: PeerId { owner_id: 443, id: 2668270 })
2024-06-11T12:58:47+04:00 [INFO] set status on client 153260: Connected { peer_id: PeerId { owner_id: 443, id: 2668270 }, connection_id: ConnectionId { owner_id: 0, id: 0 } }
2024-06-11T12:58:50+04:00 [ERROR] crates/languages/src/rust.rs:443: Os { code: 2, kind: NotFound, message: "No such file or directory" }
2024-06-11T12:58:51+04:00 [WARN] request completed with error: request or operation took longer than the configured timeout time
2024-06-11T12:58:51+04:00 [ERROR] crates/client/src/telemetry.rs:492: request or operation took longer than the configured timeout time

Caused by:
    [28] Timeout was reached
2024-06-11T12:58:51+04:00 [WARN] request completed with error: request or operation took longer than the configured timeout time
2024-06-11T12:58:54+04:00 [ERROR] crates/languages/src/rust.rs:443: Os { code: 2, kind: NotFound, message: "No such file or directory" }
2024-06-11T12:58:55+04:00 [ERROR] crates/languages/src/rust.rs:443: Os { code: 2, kind: NotFound, message: "No such file or directory" }
2024-06-11T12:58:57+04:00 [ERROR] crates/languages/src/rust.rs:443: Os { code: 2, kind: NotFound, message: "No such file or directory" }

[mrnugget]

In my case, the issue is with rust-analyzer.

That should then be possible with the upcoming Preview release and when setting {"rust-analyzer": {"binary": {"path_lookup": true}}}.

[jeremylightsmith]

Alright, I want to show you where it's not working for me. You can see that when I go into my project directory, direnv runs and we are finding elixir from nix. But after launching zed, even with foreground, within a task, it finds elixir from asdf :(. Which means all my tests fail and I can't do things like mix test

For VSCode, it works because the first time I run a task, it opens and fails, then I can manually type direnv allow, and then future runs of the task use that same environment that has been direnv'd.

[mrnugget]

within a task

Yeah, that's not supported yet.

[zimbatm]

This is a bit related to #5347 in the sense that you want to:

1. "remote" into a per-project shell session.
2. manage the lifecycle of that session (e.g., stop/start/reload).
3. make sure every program invocation in the code has a layer of indirection to execute in that session.
4. lean on the PATH of said session to find tools instead of complicated/static lookup/installation logic.

So far, all the IDEs I have seen do this wrong as they try to resolve and execute binaries from the IDE's context. Sometimes, they allow updating the IDE's ENV, and then it breaks down when working on two different projects at the same time.

Creating this architecture is also helpful for other things, like moving to a future where project code is sandboxed correctly and doesn't have arbitrary access to all of $HOME.

[musjj]

It looks like that direnv is now officially supported: https://github.com/zed-industries/zed/blob/main/docs/src/configuring-zed.md#direnv-integration
I was trying out the latest release and was pleasantly surprised that direnv just works (without having to manually trigger it in the shell).

[misuzu]

It looks like that direnv is now officially supported: https://github.com/zed-industries/zed/blob/main/docs/src/configuring-zed.md#direnv-integration I was trying out the latest release and was pleasantly surprised that direnv just works (without having to manually trigger it in the shell).

Does it work with multiple direnv environments loaded at the same time?

[jeremylightsmith]

Direnv does NOT just work. It mostly doesn't seem to work at all, and if you read the fine print on the docs, it says:

direnv integration currently only means that the environment variables set by a direnv configuration can be used to detect some language servers in $PATH instead of installing them.

I need it to find other things, like elixir in a path modified by direnv. It doesn't have the path as direnv sets it :(

[bjeanes]

I need it to find other things, like elixir in a path modified by direnv. It doesn't have the path as direnv sets it :(

This is still true after 0360cda ?

From the changelog on v0.147.0-pre:

Improved the environment-variable detection when running tasks so that tasks can now access environment variables as if the task had been spawned in a terminal that cded into a project directory. That means environment variables set by direnv/asdf/mise and other tools are now picked up (#12125).

Check out the video!

[jeremylightsmith]

I'm still on Zed 0.146.5, so looks like I need to wait till 147.2 hits? That's super exciting!

[jeremylightsmith]

Hmm, I installed zed preview, which is 147.2, and when I run this task:

  {
    "label": "show path",
    "command": "echo $PATH",
    "reveal": "always"
  }

it still shows the path without direnv's changes to it. I'm using nix, if it matters.

:(

[mrnugget]

How do you use direnv?

Here's what I use with Zed Preview 0.147.2:

1. Create the project:

mkdir ~/tmp/testing-direnv
cd ~/tmp/testing-direnv
echo export FOO=foo > .envrc
direnv allow .

2. Open Zed Preview
3. task: Spawn
4. Type in echo $FOO, hit opt-return
5. Task is spawned, prints foo

See:

So, how exactly is your direnv setup working, how do you start Zed and how to you open that project?

[jeremylightsmith]

Thanks for taking the time to try to help me out. I think I have things working now, but I did find a couple bugs along the way.

Here's a play project that someone can hopefully repro them with:

testing-zed.zip

It has this .envrc file:

export FOO=foo
export PATH=/first:$PATH:/last

And this .zed/tasks.json file:

[
  {
    "label": "show path",
    "command": "echo \"FOO=$FOO and PATH=$PATH\"",
    "reveal": "always"
  }
]

When I open zed and run the show path task, the behavior I see is:

1. I was using zed . to open zed. When I do this, I do see $FOO as being defined, but I don't see /first or /last on the path. Weird. When I open the application from spotlight, then I do see /first and /last on the path

2. /last is last on the path, but /first is not first. not sure why, but there's a bunch of stuff that comes first, including asdf in my actual project, which breaks which elixir I'm trying to use. (I worked around it by just removing asdf for now). On the command line, /first is first.

I'm on an M2 Max mac running Sonoma 14.6.

[mrnugget]

Okay, let me first answer (2), since (1) is tricky and I need to dive into it.

2. /last is last on the path, but /first is not first. not sure why, but there's a bunch of stuff that comes first, including asdf in my actual project, which breaks which elixir I'm trying to use. (I worked around it by just removing asdf for now). On the command line, /first is first.

That's because when spawning tasks, we spawn an interactive shell, which then sources your ~/.zshrc or ~/.bashrc or what you have.

Here, all the things that come before /first are things that I add in my .zshrc:

That's reproducible when running the same command that Zed runs when executing the task:

I'm not sure how to fix it yet. Workaround could be to change your shell config to detect when any of the ZED_* env vars are set and then not add to the PATH:

[mrnugget]

1. I was using zed . to open zed. When I do this, I do see $FOO as being defined, but I don't see /first or /last on the path. Weird. When I open the application from spotlight, then I do see /first and /last on the path

Okay, this is a bit of a nasty one. I think I know what's going on now:

• When you run zed . you are inside the directory with direnv enabled. direnv leaves breadcrumbs in the environment that it's enabled: DIRENV_DIFF, DIRENV_WATCHES, DIRENV_DIR are all set.
• zed is not Zed.app, it's the CLI. So the CLI then spawns Zed.app, via macOS URL handling. See here.
• Zed.app is then started and inherits environment variables from the process that spawned it: the CLI.
  • (You can confirm this by doing: export ENV_VARS_SMELL_FUNKY=true && /Applications/Zed.app/Contents/MacOS/cli ~/Downloads/testing-zed, opening a terminal inside the just-spawned Zed.app, and doing echo $ENV_VARS_SMELL_FUNKY)
• But at the same time, Zed.app thinks its stdout/stdin isn't connected to a TTY, as if it was spawned from the Dock. And that's correct! Because it isn't connected to the TTY that spawned it. So this piece of logic here kicks in and Zed.app spawns a login-shell in your home directory to gather the environment as if it were spawned from the terminal (a common thing that macOS applications do that need a proper environment)
  • (You can confirm this by using --foreground with the CLI: /Applications/Zed.app/Contents/MacOS/cli ~/Downloads/testing-zed --foreground. This will leave Zed.app connected to the CLI and thus inheriting its stdin/stdout (and environment!) — the logic to fetch the env from the home-dir doesn't kick in and your full direnv-populated env from the shell is used.)
• This then overwrites $PATH! So now, we have $PATH from the home-dir but at the same time DIRENV_* environment variables set.
• So then, when you spawn the task, we try to load the environment from your project (to get the direnv environment!), but direnv says "nu-uh, I already ran, there's DIRENV_* stuff in the environment" and doesn't do anything.
  • I confirmed this by adding a bit of code locally to delete all DIRENV_* variables before trying to get the project environment.
• Result: you end up with the home-dir $PATH but the other environment variables from your CLI (like $FOO)

So, what do we do about it?

I think there are a bunch of options and I need to think through the trade-offs of them, but here's what we/you could do, off the top of my head:

1. We: when Zed.app is spawned from the CLI, we should detect that and not re-populate the environment from the home-dir.

• Subtask: we should explicitly forward the environment from the CLI to the process, so that we can store that on environment on the project and reuse it, even if multiple Zed windows are open. That's the best solution.

2. We could do a hack: remove DIRENV_* variables before spawning the login-shell in the project dir. Hacky, nasty. Temptingly easy though. Still: super nasty.
3. You: instead of doing zed . you could do cd .. && zed my-dir, so that direnv is unloaded, the env vars removed and direnv is properly loaded.
4. You: you can run zed --foreground . too, which should fix the issue. But I don't consider that a fix, more like a workaround.

We really should do (1) and the subtask, but we're very busy now with an upcoming launch, so I'm probably not going to get to that in the next couple of weeks.

[domenkozar]

Using 0.147.2, with direnv activated and rust-analyzer available in $PATH, Zed still downloads the binary on it's own and fails to run it. No error is reported in the editor.

I wish there way a way to disable auto-downloading all together and using only $PATH.

[mrnugget]

@domenkozar did you turn the PATH-lookup on for rust-analyzer? It's not the default for rust-analyzer since r-a behaves different than other language servers somehow.

{
  "lsp": {
      "rust-analyzer": {
          "binary": {"path_lookup": true}
      }
  }
}

(Edit: documented as part of our docs party today: 1683186)

[domenkozar]

Same result:

2024-08-08T13:17:54.685708953+01:00 [INFO] starting language server "rust-analyzer", path: "/home/domen/dev/lanza.events", id: 5
2024-08-08T13:17:54.727934758+01:00 [INFO] starting language server. binary path: "/home/domen/.local/share/zed/languages/rust-analyzer/rust-analyzer-2024-08-05", working directory: "/home/domen/dev/lanza.events", args: []
2024-08-08T13:17:54.728599637+01:00 [ERROR] failed to start language server "rust-analyzer": failed to spawn command. path: "/home/domen/.local/share/zed/languages/rust-analyzer/rust-analyzer-2024-08-05", working directory: "/home/domen/dev/lanza.events", args: []
2024-08-08T13:17:54.728706493+01:00 [ERROR] server stderr: Some("")
2024-08-08T13:17:54.7288086+01:00 [ERROR] Hit 4 reinstallation attempts for "rust-analyzer"

Note that I'd expect path_lookup to be a global setting that LSPs respect.

[mrnugget]

It works for me:

$ cd tucan
direnv: loading ~/code/projects/tucan/.envrc
direnv: using flake
[...]
direnv: nix-direnv: renewed cache
direnv: export +AR +AR_FOR_TARGET +AS +AS_FOR_TARGET +CC +CC_FOR_TARGET +CONFIG_SHELL +CXX +CXX_FOR_TARGET +HOST_PATH +IN_NIX_SHELL +LD +LD_DYLD_PATH +LD_FOR_TARGET +MACOSX_DEPLOYMENT_TARGET +NIX_BINTOOLS +NIX_BINTOOLS_FOR_TARGET +NIX_BINTOOLS_WRAPPER_TARGET_HOST_aarch64_apple_darwin +NIX_BINTOOLS_WRAPPER_TARGET_TARGET_aarch64_apple_darwin +NIX_BUILD_CORES +NIX_CC +NIX_CC_FOR_TARGET +NIX_CC_WRAPPER_TARGET_HOST_aarch64_apple_darwin +NIX_CC_WRAPPER_TARGET_TARGET_aarch64_apple_darwin +NIX_CFLAGS_COMPILE +NIX_CFLAGS_COMPILE_FOR_TARGET +NIX_DONT_SET_RPATH +NIX_DONT_SET_RPATH_FOR_BUILD +NIX_ENFORCE_NO_NATIVE +NIX_HARDENING_ENABLE +NIX_IGNORE_LD_THROUGH_GCC +NIX_LDFLAGS +NIX_LDFLAGS_FOR_TARGET +NIX_NO_SELF_RPATH +NIX_STORE +NM +NM_FOR_TARGET +PATH_LOCALE +RANLIB +RANLIB_FOR_TARGET +RUST_SRC_PATH +SIZE +SIZE_FOR_TARGET +SOURCE_DATE_EPOCH +STRINGS +STRINGS_FOR_TARGET +STRIP +STRIP_FOR_TARGET +XDG_DATA_DIRS +__darwinAllowLocalNetworking +__impureHostDeps +__propagatedImpureHostDeps +__propagatedSandboxProfile +__sandboxProfile +__structuredAttrs +buildInputs +buildPhase +builder +cmakeFlags +configureFlags +depsBuildBuild +depsBuildBuildPropagated +depsBuildTarget +depsBuildTargetPropagated +depsHostHost +depsHostHostPropagated +depsTargetTarget +depsTargetTargetPropagated +doCheck +doInstallCheck +dontAddDisableDepTrack +mesonFlags +name +nativeBuildInputs +out +outputs +patches +phases +preferLocalBuild +propagatedBuildInputs +propagatedNativeBuildInputs +shell +shellHook +stdenv +strictDeps +system ~PATH

$ which rust-analyzer
/nix/store/x4irwblcsfq8df4z5lvq8vzwxfxfprhm-rust-default-1.77.2/bin/rust-analyzer

Then I open Zed.app, open the project, and in the logs it says:

2024-08-08T13:14:53.836573Z [INFO] found user-installed language server for Rust. path: "/nix/store/x4irwblcsfq8df4z5lvq8vzwxfxfprhm-rust-default-1.77.2/bin/rust-analyzer", arguments: []
2024-08-08T13:14:53.836649Z [INFO] starting language server. binary path: "/nix/store/x4irwblcsfq8df4z5lvq8vzwxfxfprhm-rust-default-1.77.2/bin/rust-analyzer", working directory: "/Users/thorstenball/code/projects/tucan", args: []

Can you show your settings.json? Or do you have a project with which to reproduce this?

[domenkozar]

{
  "assistant": {
    "version": "2",
    "default_open_ai_model": null,
    "default_model": {
      "provider": "anthropic",
      "model": "claude-3-5-sonnet-20240620"
    }
  },
  "lsp": {
      "rust-analyzer": {
          "binary": {"path_lookup": true}
      }
  },
  "vim_mode": true,
  "base_keymap": "VSCode",
  "theme": "One Dark",
  "ui_font_size": 16,
  "buffer_font_size": 16
}

[mrnugget]

Nothing wrong there. With your settings it also works for me:

2024-08-08T16:07:52.718096Z [INFO] found user-installed language server for Rust. path: "/nix/store/x4irwblcsfq8df4z5lvq8vzwxfxfprhm-rust-default-1.77.2/bin/rust-analyzer", arguments: []
2024-08-08T16:07:52.718193Z [INFO] starting language server. binary path: "/nix/store/x4irwblcsfq8df4z5lvq8vzwxfxfprhm-rust-default-1.77.2/bin/rust-analyzer", working directory: "/Users/thorstenball/code/projects/tucan", args: []