How to wait for virtual environment to be activated before issuing commands in a terminal?

[abingham]

I've got an extension that executes commands in a terminal. It relies on the Python extension to activate the correct virtual environment in the terminal so that the command (which is installed into the venv) is available. However, I've found that when I just run:

const term = vscode.windows.createTerminal('My Terminal');
term.sendText('my-command');

my-command is issued before the environment is activated.

Is there a way to wait for the environment to be activated, perhaps a post-activation hook or something? I see that there's some system of "terminal providers" which I don't really understand, so is there some part of that API I can use?

[brettcannon]

We (and by that I mean @karrtikr 😁) are about to start looking into how we do terminal stuff to see if there's a better way to handle it. But I'm personally not aware of any hook from VS Code that can help in this situation based on how things are implemented.

But do you need to have PATH manipulated? Could you use -m with the selected Python interpreter to run the tool?

[abingham]

Just a quick follow-up: this approach seems to be working for my immediate needs, but it has problems. My command now runs with the correct Python environment, but the activation command is getting intercepted by the process I'm running. The command I'm running is scons, which is executing ffmpeg, and some part of ffmpeg is seeing the 'activate' command on stdin; notice the 'Parse error' in the output below:

frame=    1 fps=0.0 q=0.0 10000000000000000000000000000000size=       0kB time=00:00:00.00 bitrate=N/A speed=   0x    
Enter command: <target>|all <time>|-1 <command>[ <argument>]

Parse error, at least 3 arguments were expected, only 1 given in string 'e /Users/austin/.virtualenvs/core-python-getting-started-dev/bin/activate'
frame= 2545 fps=375 q=-1.0 L10000000000000000000000000003000size=    3709kB time=00:01:41.79 bitrate= 298.5kbits/s speed=  15x    
video:2033kB audio:1592kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 2.331437%

At the end of the scons execution, ffmpeg has produced the right output, but the virtual environment has not been activated in the terminal.

Anyhow, I don't think there's anything to be done about it right now. This seems to be working for me, but I thought I'd leave these details as they might inform how you design terminal interaction in the future. Thanks for you help on this and your excellent work on the Python extension!

[brettcannon]

You can turn off the activation in the terminal to avoid the race condition.

[abingham]

Can I turn it off on a per-terminal basis, e.g. in the createTerminal call? I still want it activated in other terminals which aren't dedicated to my extension.

[brettcannon]

The setting is extension-wide.

[DonJayamanne]

@abingham You can turn it off on a per-terminal basis in the case of createTerminal call by creating a hidden terminal.
I use such a technique in my Python Env Manager extension here,
let terminal = window.createTerminal({ hideFromUser: true, name, cwd });

All hidden terminals are ignored by the Python extension.
Hence you could create a hidden terminal (python will ignore this) & then you can show it at a later point in time.

[karrtikr]

Is there a way to wait for the environment to be activated

You actually don't need to wait for the terminal to finish activating before sending your commands, you can send your command after we send ours, and VSCode ensures they're executed in sequence.

There's currently a hack you can rely on based on an internal detail in our extension, this is where we send the activation commands:

vscode-python/src/client/common/terminal/activator/base.ts

Lines 30 to 46 in db8e1e2

	if (activationCommands) {
	for (const command of activationCommands) {
	terminal.show(options?.preserveFocus);
	terminal.sendText(command);
	await this.waitForCommandToProcess(terminalShellType);
	activated = true;
	}
	}
	deferred.resolve(activated);
	return activated;
	}
	protected async waitForCommandToProcess(_shell: TerminalShellType) {
	// Give the command some time to complete.
	// Its been observed that sending commands too early will strip some text off in VS Code Terminal.
	await sleep(500);
	}
	}

We wait for 500ms before sending every activation command, and max number of activation commands we currently send is two, which amounts to 1 second. A sleep of 1.5 seconds should practically always work, and that won't cause any delay as environment activation can itself take a second or two.

[karrtikr]

If this solution does not work for you, feel free to open up a feature request about us providing an API which captures when we finish sending activation commands. But as @brettcannon mentioned my investigated related to #11039 can also potentially handle this scenario 🙂

[msarahan]

I have a related question that might belong in a new feature request, but I figured I'd start here in case it doesn't need a new issue.

I have a python library that basically manages metadata around runs of machine learning tools. We are writing a VS Code extension for this library to provide a sort of UI for viewing the run information and starting new runs. The runs need the correct python environment activated. The easiest way to do this right now seems to be just to rely on the python plugin's existing environment configuration. This does seem to work fine when a python file is open (with the caveat that Austin described at the beginning about command ordering), but if a YAML file is open, then the terminal that we open lacks activation.

My overarching goal: get a terminal that has the project's configured environment activated. It might be good enough to just have the path to the python executable, but from my old conda experience, true activation is better for catching any random environment variables that may actually be critically important. This is why I think #18978 (comment) is maybe not a complete solution here. Also, since our tool does a fair bit of subprocess stuff, I'm concerned that PATH modification (and thus activation) may be a necessary evil here.

From looking at the vscode-python codebase, it seems like it would be ideal if I could access the vscode-python plugin's TerminalServiceFactory, essentially to be able to do this from my plugin: https://github.com/microsoft/vscode-python/blob/main/src/client/providers/terminalProvider.ts#L55-L57

Let me know if this belongs in a new issue, or if you need more info.

[brettcannon]

Do you need a true terminal, or do you just need a way to run a Python file using the selected Python environment? And if this is an internal extension then you could also create a task that does the activation and then runs the command using &&.

[msarahan]

We're not completely certain yet. Our library has been used primarily as a CLI tool in the past. Our VSCode extension provides more of a GUI for it. One way of running things is to do your latter option of just running a python file with the selected interpreter, and then ideally streaming results to some kind of output window. Another way would be to use a terminal, so that the terminal remained available for subsequent commands to be entered. I don't want to get too hung up on this. The major thing is that no matter whether we're using a terminal, the environment for execution needs to be fully populated, and to me, that means activation.

#11039 looks like a better way to do this. If there's a way I can help move that forward, I'll be happy to contribute. In the meantime, I'm just using vscode-python's api to call environment.getActiveEnvironmentPath(), and then using that absolute path to an interpreter to run the module, as you suggested above. It works for simple enough use cases. I haven't proven this, but I think it will fail if:

• a conda environment is created that contains python as well as another executable (say, R)
• We select that environment in VS code
• We run a python script that calls the other executable as a subprocess without an absolute path

In this situation, because we are not activating the environment, the other executable either won't be found, or may come from an unexpected source elsewhere on the user's system.

Another situation is where a library requires that certain environment variables be set in order to operate correctly. GDAL is one example in the conda ecosystem: https://github.com/conda-forge/gdal-feedstock/blob/main/recipe/scripts/activate.sh. Regardless of whether we are using a subprocess to run anything, the /abs/path/python -m approach will not work correctly here. Setting the environment variables is the only way to make things work, and activation is the most reasonable way to do that, unfortunately.

What I want to avoid is duplicating all the hard work that you all have done on how to do the activation. Specifically, I want to use your work from https://github.com/microsoft/vscode-python/tree/3698950c97982f31bb9dbfc19c4cd8308acda284/src/client/common/terminal/environmentActivationProviders. The task idea is likely fine, but is there a way for me to be ignorant of what kind of environment the user has, and for the vscode-python extension to provide me with activation commands for whatever environment/interpreter it has selected?

I especially empathize with you all on how hard it is to activate environments in the general case. Providing all of the hard-won activation functionality that you have implemented in the vscode-python plugin would make life much easier for anyone who is developing extensions that use python. Again, if there's some way I can help, please point it out to me.

[brettcannon]

If there's a way I can help move that forward, I'll be happy to contribute.

Somehow manage to end this pandemic so the entire team doesn't fall ill in a single week, even when geographically dispersed? 😉 In all seriousness, please leave a comment on that issue that you're happy to help as we are just starting to look into the state of our terminal support (this whole iteration is a bit shot due to the various illnesses everyone came down with).

Regardless of whether we are using a subprocess to run anything, the /abs/path/python -m approach will not work correctly here. Setting the environment variables is the only way to make things work, and activation is the most reasonable way to do that, unfortunately.

We have moved over to conda run for this. It also helps smooth things out within the extension as not being able to use the path to the interpreter alone is a conda-only requirement. We are looking at somehow providing the information necessary for you to know how to run code appropriately either with an absolute path or conda run via our proposed API (/cc @karrtikr ). But for explicit terminal activation, that's an open question and something to comment on at #11039 so it isn't lost.

is there a way for me to be ignorant of what kind of environment the user has, and for the vscode-python extension to provide me with activation commands for whatever environment/interpreter it has selected?

Currently, no. I have the idea of generalizing all of this info at brettcannon/python-launcher#168 .

[msarahan]

conda run is probably fine for us, too, at least in the use case where we don't keep the terminal open for later usage. Dumb implementation detail, but conda run is activating the environment by writing a temporary shell script that calls activate, then runs whatever your command is. Since this is called in a subprocess, it doesn't affect your current shell. I've never tried it, but I wonder what conda run bash would do. Maybe a hacky way to get an activated terminal.

I hope that your team gets good rest and recovers quickly. Most of all, I hope they don't get the long-covid junk or any of the other weird, long-lasting effects.