Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

multiple parallel environments #1071

Closed
coldfix opened this issue Nov 15, 2017 · 17 comments

Comments

@coldfix
Copy link

commented Nov 15, 2017

Hi,

thanks for this wonderful tool!! It's well crafted and extremely useful, but I feel that there is exactly one feature missing to make it just perfect for a workflow I frequently encounter while developing python packages:

One often needs to switch python versions in order to test behaviour on different python versions. Technically, pipenv --python=XXX can be used to switch between different python versions. However, this replaces the current environment and is therefore inconveniently slow in practice.

I would really like to be able to install multiple environments in parallel for the same project and select which one to use on the pipenv command line. This can speed up the time required to switch python versions for the same project significantly and therefore we would have a powerful (IMO alround better) replacement for virtualenvwrapper and the like.

This could work by e.g. providing an environment name on the command line:

# setup environments
pipenv --name 35 --python 3.5 install
pipenv --name 36 --python 3.6 install

# run commands
pipenv --name 35 run python

One issue could be that of different lockfiles in different python versions..

If you would be fine with such an option and think that it is possible to implement for s.o. unacquainted with the code, I'm of course willing to help on the implementation.

This issue is related to #368 and #137 who I think were trying to ask the same question, but I'm unsure whether you rejected the idea altogether or were happy with an intermediate solution at the time.

Best, Thomas

@techalchemy

This comment has been minimized.

Copy link
Member

commented Nov 16, 2017

For automated testing many people use a pipeline similar to what you would find in our Travis config file. However you also have the option to use your own virtualenv when running pipenv. Most people don’t handle this by hand but if you do you probably want to make your own virtualenv for each or use pyenv

@techalchemy

This comment has been minimized.

Copy link
Member

commented Nov 16, 2017

Thanks for the suggestion but deterministic virtualenv locations is an inherent part of what pipenv is, so I don't see us implementing this. I'm gonna close this out for now!

@coldfix

This comment has been minimized.

Copy link
Author

commented Nov 16, 2017

Thanks for the answer.

For automated testing many people use a pipeline similar to what you would find in our Travis config file.

I was mainly thinking about interactive use, i.e. debugging, or stuff that's hard to test automatically (GUI, look&feel).

Thanks for the suggestion but deterministic virtualenv locations is an inherent part of what pipenv is, so I don't see us implementing this.

location can still be deterministic, based on path+name, so there is no real conflict here.

so I don't see us implementing this. I'm gonna close this out for now!

Fair enough, thanks anyway!

@AlecBenzer

This comment has been minimized.

Copy link

commented Mar 5, 2018

I also don't understand what conflict this would create with deterministic virtualenv locations.

@jtratner

This comment has been minimized.

Copy link
Collaborator

commented Mar 5, 2018

To be super clear, you can still get your own custom environments set up just by sourcing virtualenvs.

virtualenv 35 --python=python3.5
virtualenv 36 --python=python3.6
source 35/bin/activate && pipenv install
source 36/bin/activate && pipenv install
source 35/bin/activate && pipenv run <whatever>

a tiny bit of additional visual clutter to the commands but is pretty straightforward

@AlecBenzer

This comment has been minimized.

Copy link

commented Mar 5, 2018

Interesting, so if you run pipenv while already in a virtualenv, it will just use that virtualenv, and ignore the one it would use otherwise?

@techalchemy

This comment has been minimized.

Copy link
Member

commented Mar 5, 2018

Yup

@moloney

This comment has been minimized.

Copy link

commented May 23, 2018

The workaround kind of defeats the point of using pipenv...

The argument about deterministic virtualenv locations is bunk, the python version would just be another factor in automatically selecting the correct virtualenv.

I am not saying there isn't possibly some valid technical reason for rejecting this feature, but it would be nice to actually know what it is.

@uranusjr

This comment has been minimized.

Copy link
Member

commented May 25, 2018

@moloney Honest question out of curiosity: How would you design an interface if we are to start from the ground up for this one? How would you associate virtual environments to a given project without cluttering the usual workflow with --name arguments?

I ask because I’ve been looking into building my own home-baked Pipfile + virtual environment tool, and collecting feedbacks for a while. Any feedback would help me greatly toward my goal.

@moloney

This comment has been minimized.

Copy link

commented May 25, 2018

@uranusjr I should have been more precise. I agree that adding a --name option is probably too much. Instead of adding a --name option I would change the lookup behavior to use both the project and the python version to lookup the virtualenv to use.

I can understand not wanting to recreate the functionality of tox, but it is nice to be able to interactively switch between python versions and try things out without waiting for all the dependencies to reinstall.

@uranusjr

This comment has been minimized.

Copy link
Member

commented May 26, 2018

That makes sense. I’ll ponder on this a bit, and maybe I can even come up with a backward-compatible way to do this in Pipenv.

@Cito

This comment has been minimized.

Copy link

commented Jun 2, 2018

Another use case for this is when I want to work on the same project in a shared folder used by different virtual machines. Currently I'm using the approach suggested by @jtratner.

@NicolasDutronc

This comment has been minimized.

Copy link

commented Oct 26, 2018

I have another use case: I'm working on a project with pytorch. The issue here is that the installation of pytorch is os specific, thus when I switch between linux and mac, the pytorch version is incompatible. So I was looking for a way to create an environment for each os.

@nathanielcompton

This comment has been minimized.

Copy link

commented Jan 16, 2019

@uranusjr I realize this issue has been closed for some time now, but I'm wondering if there is any progress regarding a solution/feature? If so, would you please point me in the right direction?

@uranusjr

This comment has been minimized.

Copy link
Member

commented Jan 18, 2019

@nathanielcompton There is a (relatively recent) feature that adds support of an in-project .venv file to tell Pipenv to find the environment at a certain location. You can do things like this:

# Initialize the project.
mkdir my-project
cd my-project
echo "" > Pipfile

# Create some environments.
python3.5 -m venv .venv-3.5 --prompt=my-project
python3.6 -m venv .venv-3.6 --prompt=my-project

Now you can switch environments like this:

$ echo ./.venv-3.6 > .venv
$ pipenv --venv
/path/to/my/project/.venv-3.6
$ echo ./.venv-3.5 > .venv
$ pipenv --venv
/path/to/my/project/.venv-3.5

(Note: The ./ part is required. A string without path separators is interpreted as a directory name inside WORKON_HOME.)

So the functionality is there, all what’s left is to design a good UI around it. My suggestion would be to develop a standalone tool (called pipenv-switch, maybe?) and try figuring out how things can be wired together. Once it’s done, it’d be easy to propose a PEEP to try adding it into Pipenv proper.

@Jamim

This comment has been minimized.

Copy link
Contributor

commented Jan 18, 2019

Hello @uranusjr,
A feature that you mentioned above looks like the reinventing of symlinks.
Is it a workaround for Windows or are there any hidden benefits from using this approach on UNIX-like operating systems?

@uranusjr

This comment has been minimized.

Copy link
Member

commented Jan 18, 2019

Windows is certainly an important reason behind this design, where symlinking is not viable in general.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
10 participants
You can’t perform that action at this time.