Should Thonny isolate 3rd party packages installed via its dialogs?

[aivarannamaa]

Thonny 2.1 used an automatically generated virtual environment under HOME as its default backend (this option is still available under Tools => Options => Interpreter). This often caused confusion and it was complex to maintain implementation-wise. So I demoted this in Thonny 3.0.

There was also a private directory in 2.1 for installing Thonny's plug-ins and their dependencies.

Since 3.0 "Tools => Manage packages" and "Tools => Manage plug-ins" install packages with pip install --user, ie. they end up in a standard location under HOME and are available for other Python interpreters (of same version) in the system.

Occasionally I miss the old behavior but I don't want to revert current approach yet -- need to collect more experience and opinions.

Please add your ideas below!

[Ivoz]

This often caused confusion and it was complex to maintain implementation-wise. So I demoted this in Thonny 3.0.

What kind of confusion? And for what audiences having what trouble? Beginners or people that already had an idea what they're doing? (Trying to better understand the viewpoints against this idea)

There was also a private directory in 2.1 for installing Thonny's plug-ins and their dependencies.

That sounds cool, as long as it's just adding a specific folder to python's import path, is there a lot more pain this tended to cause?

Since 3.0 "Tools => Manage packages" and "Tools => Manage plug-ins" install packages with pip install --user, ie. they end up in a standard location under HOME and are available for other Python interpreters (of same version) in the system.

This is ok, but the major factor is this distinction both isn't mentioned, and also isn't represented in the install packages dialog. I learned about it reading this issue.

If I open the "Manage packages" dialog, I am presented with an eclectic and long list of packages I presume Thonny uses to do its thing.

Naively, I'd hope to just see a clean slate. I can install things, then if I click uninstall enough times back to an empty list, I know I'm back to the clean slate. What were the problems with this, assuming that's somewhat how it used to work?

If I use the Thonny interface to install a new package, then I will quickly lose track of which I have installed, and what their dependencies are that got installed with it. This was especially present when I wanted to test out one package someone's code was using, and then returning to a "clean state" after was a nightmare trying to find all the surprising dependencies it installed.

I didn't cotton on to the fact they'd all likely be in a different directory (--user install) to the Thonny packages.

So mayyyybe simply making distinction between Thonny & user packages far more clear, if there is a clean way to do this, could be a good enough compromise solution.

One reason this is a pain point could also be because virtual environments are a "2nd class citizen" in terms of accessibility, I think, compared to the visual pip install dialog. I find this to be opposite of movement in other python tools, like pipenv, which are trying to bundle it up as part-and-parcel of using 3rd party libraries in an application one is developing. i.e, using a virtualenv is a matter of course for starting an application.

If I want to set up a virtualenv / venv to start a new project, then I have to know about and navigate into an options menu to do so, before exiting that and going to the pip dialog.

Another inspiration to take is from conda. That sets you up with a base environment, but they are very forward about pushing creating conda environments as a first and foremost step for developing projects.

I am not sure if there might be a nice wizard dialog design Thonny could adopt for guiding a user through setting up a project where they want to install 3rd party libraries.

I am also aware that extra steps and "what are virtual environments!?" can be unwanted stumbling blocks and knowledge holdups for beginners just wanting to get into the thick of things. But I am hoping there is a way to improve the current execution which can work for all parties.

[aivarannamaa]

The initial resistance to my original vision came from Raspberry guys, who did not want users to have to install packages which are already present in the system (eg. pygame). So we made RPi's Thonny to come with different default back-end (same as front-end). Similar requests came from several teachers who wanted to pre-install 3rd party packages in their Windows classroom. In short -- some stakeholders prefer the solution where users don't have to think about installing 3rd party packages at all.

There is a technical difficulty with private virtual env. To my knowledge a virtual env is linked to the main interpreter using absolute paths. For this reason the venv needs to be created after installing Thonny. This was technically cumbersome and reduced the smoothness of UX a bit (on first run Windows users had to look at a progress dialog for many seconds).

The original approach mostly caused confusion for people who already did know something about pip, but not enough to follow the details. It really was a hack (playing tricks with environment variables to make pip install packages to desired location).

[aivarannamaa]

I like your ideas about pushing package management to the project level.

At the moment Thonny lacks the concept of "project". If it will be introduced, it needs to be optional, ie. user needs to be able to create and run single scripts without caring about projects.

Projects have been discussed in #712 and in a forum thread (https://mail.python.org/pipermail/edu-sig/2017-November/011764.html). Maybe the concept of project should be really based on a pipenv?

[aivarannamaa]

What about adding a more advanced back-end, which tries to detect the need for separate environment from the presence of requirements.txt or Pipfile in the same (or an ancestor) directory of the script to be run?

If no such file is found then the script is run inside the main environment (the one running the front-end process). If it is found, then appropriate virtual environment is looked up (or arranged) transparently to the user. If previous run was performed in another environment, then the user will be notified about the change (either via dialog, or shell notification or just special formatting in the window titlebar)

Tools => Manage packages would present the packages in current environment. If a package is added or removed then beside updating the environment requirements.txt or Pipfile gets updated as well.

[aivarannamaa]

It would be nice if the change of the environment could be made more explicitly.

In next version Thonny's Files view allows "going into" or "focusing into" a folder:

I have plans about binding current working directory of the back-end process to the directory selected in the files view.

The event of changing directories in the Files view could also invoke changing the backend environment.

[Ivoz]

At the moment Thonny lacks the concept of "project". If it will be introduced, it needs to be optional, ie. user needs to be able to create and run single scripts without caring about projects.

Projects have been discussed in #712 and in a forum thread (https://mail.python.org/pipermail/edu-sig/2017-November/011764.html). Maybe the concept of project should be really based on a pipenv?

For sure. The get-up-and-go ability, straight into typing python in the editor and running it is one basic strength of an editor aimed more-so at beginners.

While I kind of like pipenv, there is one huge downfall. If you thought waiting for virtualenv to setup an environment is slow... wait till you try pipenv.

A step beforehand is just looking for a default virtualenv directory in the current folder (I have most often seen .env or .venv as a name) and using that as the python virtual environment.

The initial resistance to my original vision came from Raspberry guys, who did not want users to have to install packages which are already present in the system (eg. pygame).

While I can see their point of view, for the rest of the ecosystem this is antithetical. For separate projects we want to be using separate instances of dependencies, so when multiple projects happen, things don't get cluttered up. The very reason virtualenv gets created in the first place. However I'm unsure if I want to die on a hill arguing completely principled on this point yet, when there are other approaches to look at, as you're discussing (and I'm about to).

Definitely one place where python has broken it's "do a thing one way, and do it well" is in project structuring. You could look at hooking into cookie-cutter functionality if you wanted. That could give you a LOT of "project" functionality if people want to set it up, but it might also be a lot of un-worthwhile complexity. Pypa also has the somewhat conservative sampleproject but even that is throwing a LOT of stuff to learn at beginners.

I think for teaching purposes, most of the use-case is just "I would like x, y & z packages already installed so I can just start writing a code file against them". Maybe "also I would like one or two files already alongside and importable / editable".

If teachers want offline-functionality (i.e students don't have to spend 1-10 minutes pip installing things from the net after starting thonny), python & pip are also perfectly capable of doing this themselves. pip can install offline wheel files from a directory comparatively quickly.

I would suggest that maybe providing a way (or showing people how) to archive a set of wheels into a zip, and then having an option for thonny to take that zip and install all its wheels into the current virtual environment / project. And/or/or not/ look in a special folder for other files/folders, and unpack those into the current folder.

That way, teachers could ask students to get x zipfile, open it with thonny (say while their new project is active), and then they have all the requirements installed, a sample file to open, and can get going. If the command for doing so was CLI as well, that's automate-able before class. etc.

In the mean time, one suggestion would be to integrate the virtual env creation/handling UI into the pip UI, so it is more visible.

[aivarannamaa]

If teachers want offline-functionality (i.e students don't have to spend 1-10 minutes pip installing things from the net after starting thonny) [...] archive a set of wheels into a zip

This is worth considering, but I hope the current ability to install wheel or zip files one-by-one is sufficent for most cases. Alternatively, if Thonny is centrally installed (or configured to use centrally installed back-end), the admin can add required packages right into main lib in order to save time, bandwidth and disk space.

But your idea got me thinking that it may be worthwhile to add "Install from requirements.txt" feature to the pip GUI. This way the teacher can easily prescribe required versions and doesn't have to correct mathplotlib to matplotlib for every other student :) There is an alternative of publishing a cource-specific meta package with required dependencies but this would be much bigger effort than creating the requirements.txt file.

[aivarannamaa]

About isolation ...

I'm not too much worried about conflicts between student's different projects, but I don't feel good about using (and affecting) the global user-site-packages. I wish students could mess up only Thonny's world, not the environment for other interpreters installed in the machine.

At the moment my plan for Thonny 4.0 is following:

• If Thonny is run with bundled Python, then it lives as the king in its own kingdom. Global user-site-packages will not be visible. The pip GUI (both for default backend and for plug-ins) will install packages into a private user-site-packages directory under THONNY_USER_DIR (or next to the program directory if running in portable mode (Create portable Windows variant of Thonny #697).
• If Thonny is run with external Python (eg. on Raspberry Pi) then it acts as dependant of the external environment -- pip GUI will be read-only both for plug-ins and default back-end and global user-site-packages will be visible. Plug-ins and extra packages need to be installed outside of Thonny (the user must decide whether to use sudo apt install python3-matplotlib, sudo pip install matplotlib or pip install --user matplotlib, ). I don't want Thonny to take the responsibility of affecting other programs in the system.
• If Thonny runs inside a virtual environment, then pip GUI (both for plug-ins and default backend) will install things without --user (obviously).
• If Thonny uses an external backend, then the rules for the back-end match the above principles -- if it's a virtual env then pip GUI installs directly, otherwise it is read-only.

This plan actually almost matches with the behavior of Thonny 2.1. The main difference is that the default automatic private virtual environment is now gone.

[Ivoz]

So I'm a noob at configuring Tk object placement, but here is a bad mockup of what I'd like to see in the install dialogue:

Essentially this lets you choose which "Python" you would like to install stuff to.

"Manage" would bring you a new dialogue for managing your python targets (and/or just to the configuration dialogue tab currently responsible for that)

"New Temporary Environment" is a suggested feature, it would allow you to create a new environment (say in system temp folder) to install a couple of packages to try out, and hopefully it would be deleted when Thonny quits (and when system temp folder is cleaned).

If you think this could go in a new feature request issue? I could put this text again in a new one

[aivarannamaa]

I expect most of the beginners don't need to change the back-end. Therefore I'm afraid any extra buttons and boxes would unnecessarily increase the confusion when using this dialog.

At the same time I like the idea of providing a header to the dialog, which reminds which back-end we're targeting (currently there is a Target block on the start page but it's easy to miss it). What about having something like this at the top of the dialog:

Current interpreter: A special virtual environment [change].

?

The nudge towards virtual environments could be done by adding following block to the start page (unless the back-end is already a virtual environment):

Consider using a virtual environment
Virtual environments allow you to maintain different sets of packages for your different projects. Click here to select or create a new virtual environment for your current project.

[Ivoz]

BTW I just copied the code to replicate the text that was displayed exactly as it is in the configuration tab; however maybe you'd want different text like

• Python 3.7.3 (C:\Program Files\Python)
• MicroPython
• Python 3.7.3 (Thonny's Install)

or similar?

If you would want to put the combo box/whatever behind an extra dialogue pop-up, like the ... button currently for existing packages, I think that could work OK.

[aivarannamaa]

however maybe you'd want different text like ...

Good idea, it's better if location of the interpreter is visible.

If you would want to put the combo box/whatever behind an extra dialogue pop-up

I was thinking of reusing the configuration dialog with "Interpreter" tab preselected.

[aoloe]

I'm starting using Thonny (for teaching) and I'm mostly comfortable with the current way to create a virtual environment.

There are two issues I'd like to mention:

• When selecting a "project" directory for creating a new venv, I would have expected Thonny to create a venv/ directory inside of it. Not to fill it with the venv files.

• The huge button is hard to see on my screen (it's easily confused with a text section with a border around it).
  It would be easier to tell the people to click on a "Create" button. Something like this:

  Known interpreters:
  [/usr/bin/python3                                               | v ]
  
  Locate a Python interpreter not in the list                [ Choose ]
  Create a new virtual environment                           [ Create ]
  

Keep up the good work and thanks for Thonny!

[aivarannamaa]

@aoloe, thank you for the tips!

The huge buttons may look weird indeed. I'll change them in 4.0

I need to think about the idea of forcing venv directory. It makes sense if you create a separate venv for each project, but if you use a venv for group of your projects, you may want to use a descriptive name for it.

[Ivoz]

I see nowadays that VS Code will will recognise if you have a .venv sub-directory in your project that is a virtualenv, it will ask if you wish to use that as the active python interpreter.

And Poetry, a very nice python library/application build&management tool, will also create a .venv virtualenv to "do its thing", if you set its configuration value virtualenvs.in-project to true (otherwise it makes one in a standard "cache" directory on the system).

So I am thinking there is becoming more of a defacto standard for this, which I personally like. I am using python -m venv .venv for any new casual project I make nowadays as well.

So a cool potential feature for Thonny could be to mirror this behaviour of VS Code: if you focus a folder in Thonny, and it finds a .venv sub-folder which looks like it has a valid python virtualenv inside, it could ask if you wish to [switch to/add] that interpreter as the current interpreter for thonny to use.

[aoloe]

(just a small addition from my side: if possible, don't rely on hidden directories... venv is good enough)

[aivarannamaa]

Thanks @Ivoz and @aoloe!

I have the same feeling that a venv per project can be promoted among Thonny users.

if you focus a folder in Thonny, and it finds a .venv sub-folder which looks like it has a valid python virtualenv inside, it could ask if you wish to [switch to/add] that interpreter as the current interpreter for thonny to use.

An alternative would be introducing a configuration flag for this -- when this flag is on, then each time the current directory is changed (either explicitly via the file browser or implicitly by running a script while "Auto cd" option is on), Thonny would select the nearest venv in current or ancestor directories. If none is found, then it switches to the interpreter running the UI.

More ideas:

• Introducing a new dialog for creating venv. Currently Thonny just uses the directory selection dialog. Currently the user needs to create the final directory (eg. .venv) herself, but the traditional way is changing to project directory and giving .venv as argument. So the new dialog would allow specifying the parent directory and venv name separately (with parent directory box prefilled with current directory and venv name box prefilled with .venv)
• Adding "Create virtual environment" to the new back-end switcher in the lower-right corner of Thonny's main window.
• Adding a custom editor for requirements.txt files. When the user saves a file with this name, then Thonny would add a pane to the bottom of the editor with buttons for creating and updating virtual environment in the same directory.

[aivarannamaa]

just a small addition from my side: if possible, don't rely on hidden directories... venv is good enough

I understand that hidden directories may cause confusion, but .venv seems to be much more popular than venv. Definitely there will be easy way to override the default, whichever it will be.

One way to remedy this problem, is by making Thonny's file browser always show certain hidden files and folders (eg. .gitignore, .venv, ...)

[aoloe]

... i've never seen an hidden venv, and the examples in the official documentation all feature visible virtual environments. (on top of it, i guess that on windows putting a dot in front probably feels strange and wont hide the directory)

i've just tested in pycharm, and it's calling it venv.

[Ivoz]

@aoloe a lot of the reasoning comes from that a project-specific virtualenv should be an ephemeral thing, able to be removed and recreated at will without harming your workflow. Similar to other cache folders (e.g I'm seeing .pytest_cache around these days) and many in your home directory.

As such its existence in a python project folder might be expected and just be "noise" rather than "signal" if its always visible. Although covering both cases to a basic extent, I imagine shouldn't be too much work anyway over the actual functionality.

[aivarannamaa]

I decided to stay with explicit creation of virtual environments, but it should be made easier #2096

Also, I decided to keep interpreter selection and settings separate from package management dialog, but duplicate "Manage packages" command at the interpreter switcher menu at the lower-right corner of the main window. I expect this menu to be most user's favorite way for switching interpreters and managing packages.

Isolation of the bundled Python will be implemented under #2038