Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add %pip and %conda magic functions #11524

Merged
merged 3 commits into from Dec 10, 2018

Conversation

@jakevdp
Copy link
Contributor

commented Dec 6, 2018

This PR adds %pip and %conda magic functions which automatically install packages into the currently-running kernel in an IPython or Jupyter notebook session.

screen shot 2018-12-05 at 9 41 51 pm

Why? Correctly installing Python packages within a running kernel has long been a point of confusion within the Jupyter ecosystem. While this might seem like an anti-pattern for the traditional local kernel use-case, the recent popularity of cloud-backed Jupyter notebook services has made this more relevant.

For some background on why !pip install or !conda install are incorrect in ways that can be very subtle and confusing, see my blog post from last year: Installing Python Packages From a Jupyter Notebook.

There are some existing issues requesting this sort of feature (see #9517 & #11481) as well as some prior art on pip and conda magic function implementations by @minrk (https://github.com/minrk/condamagic/) and @Carreau (https://github.com/Carreau/pip_magic), but the fact that these third-party functions must be manually installed limits their usefulness to people who are having trouble installing things 馃槃.

I'm hoping that including functionality like this within IPython itself will address the bulk of the confusion many of us have seen from users of the Jupyter stack.

@jakevdp

This comment has been minimized.

Copy link
Contributor Author

commented Dec 6, 2018

Needs some work on tests, but that can wait until there's some consensus on whether this feature should be added or not!

@minrk

This comment has been minimized.

Copy link
Member

commented Dec 6, 2018

I think this is a great thing to add, personally.

@dsblank

This comment has been minimized.

Copy link
Contributor

commented Dec 6, 2018

Yes! #11481 I'll take a look, and, going forward, will always try to help maintain keep refined. This is important.

@dsblank

This comment has been minimized.

Copy link
Contributor

commented Dec 6, 2018

Very clever on scanning the history, looking for the path! Never would have thought of that.

What will the advice be to users who don't know if a package can be install by conda or pip? I was originally thinking a general %install would try conda first, and then pip if not available.

@ccordoba12

This comment has been minimized.

Copy link
Member

commented Dec 6, 2018

We're also having problems with people using !pip and !conda in Spyder, so this is great! Thanks @jakevdp!

Could it be possible to intercept the use of !pip/!conda and print a message saying to people to use the magics instead? Or does the magics addition do that automatically?

@rgbkrk

This comment has been minimized.

Copy link
Member

commented Dec 6, 2018

Thank you so much @jakevdp

def _is_conda_environment():
"""Return True if the current Python executable is in a conda env"""
# TODO: does this need to change on windows?
conda_history = os.path.join(sys.prefix, 'conda-meta', 'history')

This comment has been minimized.

Copy link
@rgbkrk

rgbkrk Dec 6, 2018

Member

@dhirschfeld would this correctly detect being in a conda environment on Windows?

This comment has been minimized.

Copy link
@MSeal

MSeal Dec 6, 2018

Contributor

A better way might be to check if Anaconda is in sys.version. That should work cross-platform.

This comment has been minimized.

Copy link
@jakevdp

jakevdp Dec 6, 2018

Author Contributor

I think that only works if you happen to be using the Python built by Anaconda. For example:

$ conda create -n forge-python
$ conda activate forge-python
$ conda install python --yes -c conda-forge
$ python -c "import sys; print(sys.version)"
3.7.1 | packaged by conda-forge | (default, Nov 13 2018, 09:50:42) 
[Clang 9.0.0 (clang-900.0.37)]

This comment has been minimized.

Copy link
@MSeal

MSeal Dec 6, 2018

Contributor

Ahh yes you're right

'install', 'list', 'remove', 'uninstall', 'update', 'upgrade',
}
CONDA_COMMANDS_REQUIRING_YES = {
'install', 'remove', 'uninstall', 'update', 'upgrade',

This comment has been minimized.

Copy link
@rgbkrk

rgbkrk Dec 6, 2018

Member

Love this organization around args needing --y or the --prefix

This comment has been minimized.

Copy link
@mpacer

mpacer Jul 3, 2019

Member

It really is gloriously nice.

Usage:
%pip install [pkgs]
"""
self.shell.system(' '.join([sys.executable, '-m', 'pip', line]))

This comment has been minimized.

Copy link
@rgbkrk

rgbkrk Dec 6, 2018

Member

This works pretty well, the only hidden API surface to this is that this allows the full pip API. We likely wouldn't be able to extend this with new args, possibly for something ipython / jupyter specific (capturing metadata, accepting a flag to do so).

Needless to say, I'm cool with this implementation.

This comment has been minimized.

Copy link
@Carreau

Carreau Dec 6, 2018

Member

+1 We likely want to remind user they might need to restart the kernel.

This comment has been minimized.

Copy link
@jakevdp

jakevdp Dec 6, 2018

Author Contributor

Done in 594d000

@dhirschfeld

This comment has been minimized.

Copy link
Contributor

commented Dec 6, 2018

There might be an issue on Windows where you can silently corrupt your installation by trying to update a compiled extension when Windows is holding a reference to the DLL. Certainly, with pandas that used to often happen. I still kill all Python processes before doing any upgrades, but maybe that's no longer required and conda has worked around the original issue.

I'll try to check the current behaviour and report back...

@rgbkrk

This comment has been minimized.

Copy link
Member

commented Dec 6, 2018

@Carreau @MSeal self.shell.system is one of the calls that doesn't error when a shell command happens right? We should make sure this causes an error so that use of this magic will stop execution in nbconvert, papermill, and other run-all use cases.

@MSeal

This comment has been minimized.

Copy link
Contributor

commented Dec 6, 2018

I need to dig into that but yes one of the function magic options isn't erroring as one would expect.

@Carreau Carreau added this to the 7.3 milestone Dec 6, 2018

@Carreau

This comment has been minimized.

Copy link
Member

commented Dec 6, 2018

I guess we could also use subprocess.check_call or something. I didn't even remembered we had shell.system

@ipython ipython deleted a comment from MSeal Dec 7, 2018

@Carreau

This comment has been minimized.

Copy link
Member

commented Dec 10, 2018

I fixed the tests.
Let's get that in. I'll try to make a release close to end of month to keep the monthly release cadence.

@Carreau Carreau merged commit a8165da into ipython:master Dec 10, 2018

4 checks passed

codecov/patch 36.73% of diff hit (target 0%)
Details
codecov/project 68.33% (-0.01%) compared to eacadc1
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@rgbkrk

This comment has been minimized.

Copy link
Member

commented Dec 10, 2018

Thanks y'all!

@jakevdp jakevdp deleted the jakevdp:pip-conda-magic branch Dec 12, 2018

@ceceshao1

This comment has been minimized.

Copy link

commented Dec 17, 2018

Can't wait for this! Was just experiencing confusion today around this specific action

@dsblank

This comment has been minimized.

Copy link
Contributor

commented Jan 8, 2019

@Carreau Time for a 7.3 release?

@Carreau

This comment has been minimized.

Copy link
Member

commented Jan 8, 2019

Yes, on my todo list, but tiem is short and $DAYJOB is taking more time than planned after holidays.
Would you mind creating an issue that list the step that are remaining ? You can copy-past from the 7.2 one.

One of the biggest task is writing the what's new :-)

@dsblank

This comment has been minimized.

Copy link
Contributor

commented Jan 22, 2019

@Carreau Just saw your note. What "steps remaining" do you mean?

@Carreau

This comment has been minimized.

Copy link
Member

commented Jan 22, 2019

Basically a todo-list, with list of stem necessary before/during/after the release and list of issues /changes that need to be added in the what's new.
I did it yesterday (see #11571). I unfortunately risk to not be able to do any of that before this week-end, so any help welcome.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
10 participants
You can鈥檛 perform that action at this time.