Add support for nix-shell in JetBrains products

[jgeerds]

This might be a lot of work but it would be awesome if you could use nix-shell inside JetBrains products (e.g. activating shell.nix of your current project).

I have a lot of friends who use virtualenv instead of nix just because PyCharm (the python IDE by JetBrains) have builtin support for it.

What do you think about it?

[edwtjo]

Yes of course, using profiles or shells for development is one of the main goals of this plugin.

[afranche]

Hi,

Any update on this project?

[Mic92]

Maybe nix-shell --run 'env' > .env could be used with https://plugins.jetbrains.com/plugin/7861-envfile for the time being.

[evanjs]

Just note that EnvFile is not currently supported across all JetBrains applications, unfortunately.

Most notably for me is CLion (for Rust envs, etc.)

Related: ashald/EnvFile#71

[jonringer]

I generally start my pycharm while inside a nix-shell with my virtualenv. It works really well because pycharm is already able to reason about virtualenvs

[ergose]

@jonringer - for the work-around win. :) Big thanks. I've been wracking my brain all afternoon trying to get it working. I'm now running NixOS on several machines and wanted to start using nix-shell instead of venv. Another vote for native support.

Some additional details to anyone still trying to figure out:

1. Create you a default.nix/shell.nix in your Python project directory (if default.nix for packaging, you'll also probably want a setup.py in there also).
2. Open terminal, nav to project dir, and type nix-shell to load up environment as normal
3. Open PyCharm from within the nix-shell (ex: nohup pycharm-community >/dev/null 2>&1 & - so can close terminal after pycharm opens.)
4. Add a Python Interpreter -> System Interpreter -> /run/current-system/sw/bin/python3.8 (assuming 3.8 on NixOS 20.09)
5. Create you a Run Configuration pointing to the interpreter you just setup.
6. Press the Run button and cross fingers. For me, main script ran as expected, debug seems to work, and PyCharm sees the packages pulled in from the shell.nix. That said though, I haven't tried to add more packages yet. We may have to update the shell.nix with any new packages and re-load PyCharm to add packages, that or maybe install same version in repo via PyCharm while programming and just remember to add it to shell.nix for next time.

[fehnomenal]

I created a plugin to load direnvs .envrc files at https://github.com/fehnomenal/intellij-direnv and use it with flakes and lorri in old projects.

It works, but is nothing but a big hack (because the JVM does not want you to modify the environment...). To natively support changing the environment on a per project basis, the IDE could clone the environment from the OS (like the JVM does it) and pass the clone to all processes it launches while allowing modifications. This would allow implementing direnv support without hacks.

[l0b0]

@ergose Thank you for the instructions! That wasn't quite enough to get IDEA to pick up the Python dependencies, I also had to add pip to the mkShell buildInputs, otherwise IDEA wouldn`t find the packages in the project SDK settings (and trying to install the packaging software within IDEA didn't work).

Unfortunately the /run/current-system/sw/bin/python… path does not exist when using Nix on Ubuntu.

Neither the envfile nor intellij-direnv seem to work for wiring up the SDK when using Nix outside of NixOS. The workaround is to change the path to the SDK every time the Nix shell is updated.

[SDAChess]

Hello,

I came across this plugin that tries to add this feature.
https://plugins.jetbrains.com/plugin/16864-enter-nix-shell
You can find the source here: https://github.com/fctorial/enter_nix_shell

Maybe there are some interesting ideas that could be exploited.
Also, the new flakes and nix develop environments could be a good option to add too.

[JojOatXGME]

Thanks for your suggestions. I am reluctant to integrate the solution of Enter Nix Shell or intellij-direnv into this plugin. 😔 They both work by changing the environment of the IDEA process. This means that you cannot have different environments per project. While changing the IDEA environment with these plugins might be easier than restarting the whole IDE, it might also work less reliable. For example, see this page from Enter Nix Shell (archive.org). Besides that, you could also end up with a mix of multiple environments, which could have strange effects in some edge cases. While you might very well use one of these plugins, I feel like the implementations are too unstable for an official solution. At least that is my personal opinion.

[JojOatXGME]

I used this weekend to explore the IntelliJ Platform API for alternatives. Unfortunately, I haven't found a good solution.

Run targets

I found Run targets (extension point com.intellij.executionTargetType). They allow users to execute run configurations within Docker containers or within WSL. It might also be usable to move the execution into a nix-shell. However, run targets are an ultimate feature and therefore not available for free. Also note that they do not affect the console panel or SDK detection. So it would only be a partial solution.

Run configuration extensions

EnvFile and ProjectEnv use com.intellij.​run​Configuration​Extension among other extension points. However, this extension point is specifically for IntelliJ (i.e. Java). Other IDEs have their own extension points. So we would have to implement it for every IDE separately. It is also not sure if all these extension points combined cover every IDE and type of run configuration.

Program runner

I also found the extension point com.intellij.programRunner. Program runners are the components which implement the execution of specific run configurations. I tried adding a custom program runner which claims it can execute all run configurations, which then modifies the environment and forwards it to the actual runner. Unfortunately, it looks like I cannot change environment variables at this level. I also couldn't make the runner work reliable in my tests. In many cases, IntelliJ continued to call the actual runner directly. Besides, this solution is also a bit hacky. I could imagine that it breaks some configurations.

Source code

plugin.xml

<extensions defaultExtensionNs="com.intellij">
  <programRunner implementation="org.nixos.idea.shell.NixProgramRunner" order="first"/>
</extensions>

NixProgramRunner.java

public final class NixProgramRunner implements ProgramRunner<RunnerSettings> {
    @Override
    public @NotNull @NonNls String getRunnerId() {
        return "nix-shell-wrapper";
    }

    @Override
    public boolean canRun(@NotNull String executorId, @NotNull RunProfile profile) {
        return true;
    }

    @Override
    public void execute(@NotNull ExecutionEnvironment environment) throws ExecutionException {
        ProgramRunner<?> runner = ProgramRunner.PROGRAM_RUNNER_EP.findFirstSafe(r -> matches(environment, r));
        // Somehow modify the environment?
        runner.execute(environment);
    }

    private boolean matches(@NotNull ExecutionEnvironment environment, @NotNull ProgramRunner<?> runner) {
        String executorId = environment.getExecutor().getId();
        return runner != this && runner.canRun(executorId, environment.getRunProfile());
    }
}

Execution target provider

The extension point com.intellij.executionTargetProvider also got my attention. However, on their own, execution targets only affect the UI. They seem to be used by Android Studio to choose the device where the app gets executed. But the implementation which looks at the configuration actually sits within the specific program runner (see last section) for Android apps. Adding an execution target causes them to be visible in the UI, but it does not impact the behavior at all.

Conclusion

As written in the first paragraph of this comment, it looks like there is no straightforward solution. From the options I have covered, the approach of EnvFile and ProjectEnv seems to be the most promising, but it might take a lot of effort. So I am still open for suggestions.

[JojOatXGME]

I created a post on the community board of JetBrains. Let's see if someone responds.
https://intellij-support.jetbrains.com/hc/en-us/community/posts/4418065675538-Project-specific-environment-variables
Besides that, if someone knows the process to contribute new extension points to the IntelliJ Platform, I would be interested.

[fehnomenal]

Yeah, you did the same research as me. The main problem is that the JVMs have no way of modifying the environment (at least not with a public API).

Over a year ago I created a very similar suggestion to yours at https://youtrack.jetbrains.com/issue/IDEA-260725 but nothing happened, so I would not have high hopes... At least someone got assigned I guess 🤷

For reference: https://youtrack.jetbrains.com/issue/IDEA-288756

[l0b0]

FYI Enter Nix Shell is not just abandoned, none of the links (source, issues, documentation) work anymore. Does anyone have a copy of the source code, so we could revive it?

[JojOatXGME]

FYI Enter Nix Shell is not just abandoned, none of the links (source, issues, documentation) work anymore. Does anyone have a copy of the source code, so we could revive it?

Interesting. It was still available in February. Unfortunately, I don't have a local copy of the repository. You can still download the binaries from JetBrains Marketplace (https://plugins.jetbrains.com/plugin/16864-enter-nix-shell) and use a decompiler. According to the page on the Marketplace, the plugin was available under MIT license.

I guess I should invalidate the links in my messages since they seem to point to some NSFW-site now.

[infiniteregrets]

EnvFile and ProjectEnv use com.intellij.​run​Configuration​Extension among other extension points. However, this extension point is specifically for IntelliJ (i.e. Java). Other IDEs have their own extension points. So we would have to implement it for every IDE separately. It is also not sure if all these extension points combined cover every IDE and type of run configuration.

Hmm it seems like runConfigurationExtension is not available in IDEA 2022.2? Can't really find a way to extend new env in the new API

[JojOatXGME]

Hmm it seems like runConfigurationExtension is not available in IDEA 2022.2? Can't really find a way to extend new env in the new API

Note that you have to add com.intellij.java as dependency because com.intellij.​run​Configuration​Extension is specific to IntelliJ and does not exist in other IDEs of JetBrains.

[qlonik]

FYI Enter Nix Shell is not just abandoned, none of the links (source, issues, documentation) work anymore. Does anyone have a copy of the source code, so we could revive it?

I've had a bit of an internet historian moment, and found the website of the developer of this plugin saved on web archive. It might have some interesting information and it also has some videos showing the plugin.

https://web.archive.org/web/20220131211828/https://fctorial.com/posts/nix_intellij.html

[l0b0]

@jonringer - for the work-around win. :) Big thanks. I've been wracking my brain all afternoon trying to get it working. I'm now running NixOS on several machines and wanted to start using nix-shell instead of venv. Another vote for native support.

Thank you for this! It does seem to break when changing the packages in the Nix shell, though, in a weird way:

Notice how, in the above, aws_cdk is marked as unknown even though aws-cdk-lib (which provides aws_cdk) is shown in the list of SDK packages. This inconsistency persists across IDEA restarts and even a full "Repair IDE" cycle. It's like IDEA no longer knows about the associated SDK. Even creating a new SDK pointing to /run/current-system/sw/bin/python doesn't seem to work.

Reported

[starquake]

I use Better Direnv and it seems to work:

• https://github.com/nix-community/nix-direnv
• https://plugins.jetbrains.com/plugin/19275-better-direnv

[l0b0]

@starquake One problem with that approach is that IDEA gets very confused about which packages are installed. This also happens when using nix-direnv + Better Direnv. We need a solution which works with more than one project.

[GenericNerdyUsername]

Would it be possible to do something like

nix-shell --run ./ide_launcher.sh & disown
exit

to put the whole ide in the new environment?

Problems I see with this:

• args to the launch script aren't preserved which could break things (eg remote dev)
  • possible fix; hook com.intellij.idea.Main or read args the normal java way (a quick skim through the launch script shows that no args go to the script itself)
• If you want to switch between environments quickly, you'll have to wait for the IDE to restart every time
• There might not be a way to easily make the whole IDE exit

[nhooey]

Here's a summary of the most significant things mentioned:

• GitHub: NixOS/nix-idea #1 Add support for nix-shell in JetBrains products
  • Comment: "Just note that EnvFile is not currently supported across all JetBrains applications..."

    • Most notably for me is CLion (for Rust envs, etc.)

      • GitHub: ashald/EnvFile #71 Support for Rust in IntelliJ & CLion
        • Comment: "I built a cargo subcommand that loads dotenv files..."
          • Rust Crate: cargo-dotenv
  • Comment: "I came across this plugin that tries to add this feature."
    • JetBrains Plugin: enter-nix-shell
  • Comment: "I created a post on the community board of JetBrains..."
    • IntelliJ Support Community: Project-specific environment variables

      • I created an issue:

        • IDEA-288756: Common extension point for environment variables

          • Right now, there is no straightforward solution to implement such plugin. EnvFile and ProjectEnv try to implement this by extending a bunch of extension points across various IDEs, including: [list of extension points]
            Despite using various extension points, they still miss support for some IDEs or features (e.g. EnvFile#71). Note that this approach is rather error-prone because …

            • it is easy to forget adding such extension point when introducing new features for the IDE, and
            • it is easy to forget implementing such extension point when implementing a plugin.

            • JetBrains Plugin: EnvFile
              • GitHub: Ashald/EnvFile
            • JetBrains Plugin: ProjectEnv
              • GitHub: BredoGen/ProjectEnv

          • An alternative solution is used by Enter Nix Shell and intellij-direnv. These plugins modify the environment of the IDEA process. However, this approach has different downsides, like being very unstable and not supporting different environments for two projects which are open at the same time.

            • JetBrains Plugin: enter-nix-shell
            • JetBrains Plugin: intellij-direnv
              • GitHub: fehnomenal/intellij-direnv
  • Comment: "The main problem is that the JVMs have no way of modifying the environment (at least not with a public API)"

    • Over a year ago I created a very similar suggestion to yours but nothing happened:

      • IDEA-260725: Modifiable environment per project

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/jetbrains-ides-support-for-nix-shell-w-nix-idea-plugin-per-project-environment/32321/1

[infiniteregrets]

hey we patch the env of all the intellij IDEs, maybe this approach could help?

https://github.com/metalbear-co/mirrord-intellij

[l0b0]

Here's a workaround, after trying many things. tl;dr: Symlink the python executable somewhere static, then point the SDK to that path.

Tested only with IDEA and poetry2nix repos.

Caveats:

• IDEA seems to require that you include pip in the Python derivation, or else it doesn't find the installed packages.
• If you're using direnv you have to symlink from both .envrc and shell.nix.
• IDEA does not seem to pick up on environment changes immediately, and requires a restart for new packages to show up.

Example repos: 1, 2.

[nhooey]

@infiniteregrets: hey we patch the env of all the intellij IDEs, maybe this approach could help?

https://github.com/metalbear-co/mirrord-intellij

@infiniteregrets: Could you elaborate on your approach, or even highlight where in the mirrord JetBrains extension code the environment patching is implemented?