Add %pip and %conda magic functions #11524

jakevdp · 2018-12-06T05:45:49Z

This PR adds %pip and %conda magic functions which automatically install packages into the currently-running kernel in an IPython or Jupyter notebook session.

Why? Correctly installing Python packages within a running kernel has long been a point of confusion within the Jupyter ecosystem. While this might seem like an anti-pattern for the traditional local kernel use-case, the recent popularity of cloud-backed Jupyter notebook services has made this more relevant.

For some background on why !pip install or !conda install are incorrect in ways that can be very subtle and confusing, see my blog post from last year: Installing Python Packages From a Jupyter Notebook.

There are some existing issues requesting this sort of feature (see #9517 & #11481) as well as some prior art on pip and conda magic function implementations by @minrk (https://github.com/minrk/condamagic/) and @Carreau (https://github.com/Carreau/pip_magic), but the fact that these third-party functions must be manually installed limits their usefulness to people who are having trouble installing things 😄.

I'm hoping that including functionality like this within IPython itself will address the bulk of the confusion many of us have seen from users of the Jupyter stack.

jakevdp · 2018-12-06T05:50:16Z

Needs some work on tests, but that can wait until there's some consensus on whether this feature should be added or not!

minrk · 2018-12-06T10:01:37Z

I think this is a great thing to add, personally.

dsblank · 2018-12-06T11:40:12Z

Yes! #11481 I'll take a look, and, going forward, will always try to help maintain keep refined. This is important.

dsblank · 2018-12-06T13:36:47Z

Very clever on scanning the history, looking for the path! Never would have thought of that.

What will the advice be to users who don't know if a package can be install by conda or pip? I was originally thinking a general %install would try conda first, and then pip if not available.

ccordoba12 · 2018-12-06T13:45:35Z

We're also having problems with people using !pip and !conda in Spyder, so this is great! Thanks @jakevdp!

Could it be possible to intercept the use of !pip/!conda and print a message saying to people to use the magics instead? Or does the magics addition do that automatically?

rgbkrk · 2018-12-06T13:51:58Z

Thank you so much @jakevdp

rgbkrk · 2018-12-06T13:56:37Z

IPython/core/magics/packaging.py

+def _is_conda_environment():
+    """Return True if the current Python executable is in a conda env"""
+    # TODO: does this need to change on windows?
+    conda_history = os.path.join(sys.prefix, 'conda-meta', 'history')


@dhirschfeld would this correctly detect being in a conda environment on Windows?

A better way might be to check if Anaconda is in sys.version. That should work cross-platform.

I think that only works if you happen to be using the Python built by Anaconda. For example:

$ conda create -n forge-python $ conda activate forge-python $ conda install python --yes -c conda-forge $ python -c "import sys; print(sys.version)" 3.7.1 | packaged by conda-forge | (default, Nov 13 2018, 09:50:42) [Clang 9.0.0 (clang-900.0.37)]

Ahh yes you're right

rgbkrk · 2018-12-06T13:59:19Z

IPython/core/magics/packaging.py

+    'install', 'list', 'remove', 'uninstall', 'update', 'upgrade',
+}
+CONDA_COMMANDS_REQUIRING_YES = {
+    'install', 'remove', 'uninstall', 'update', 'upgrade',


Love this organization around args needing --y or the --prefix

It really is gloriously nice.

rgbkrk · 2018-12-06T14:33:27Z

IPython/core/magics/packaging.py

+        Usage:
+          %pip install [pkgs]
+        """
+        self.shell.system(' '.join([sys.executable, '-m', 'pip', line]))


This works pretty well, the only hidden API surface to this is that this allows the full pip API. We likely wouldn't be able to extend this with new args, possibly for something ipython / jupyter specific (capturing metadata, accepting a flag to do so).

Needless to say, I'm cool with this implementation.

+1 We likely want to remind user they might need to restart the kernel.

Done in 594d000

dhirschfeld · 2018-12-06T20:19:11Z

There might be an issue on Windows where you can silently corrupt your installation by trying to update a compiled extension when Windows is holding a reference to the DLL. Certainly, with pandas that used to often happen. I still kill all Python processes before doing any upgrades, but maybe that's no longer required and conda has worked around the original issue.

I'll try to check the current behaviour and report back...

rgbkrk · 2018-12-06T21:43:37Z

@Carreau @MSeal self.shell.system is one of the calls that doesn't error when a shell command happens right? We should make sure this causes an error so that use of this magic will stop execution in nbconvert, papermill, and other run-all use cases.

MSeal · 2018-12-06T22:34:34Z

I need to dig into that but yes one of the function magic options isn't erroring as one would expect.

Carreau · 2018-12-06T23:47:45Z

I guess we could also use subprocess.check_call or something. I didn't even remembered we had shell.system

Carreau · 2018-12-10T23:16:42Z

I fixed the tests.
Let's get that in. I'll try to make a release close to end of month to keep the monthly release cadence.

rgbkrk · 2018-12-10T23:25:27Z

Thanks y'all!

ceceshao1 · 2018-12-17T23:16:29Z

Can't wait for this! Was just experiencing confusion today around this specific action

dsblank · 2019-01-08T13:45:17Z

@Carreau Time for a 7.3 release?

Carreau · 2019-01-08T15:28:22Z

Yes, on my todo list, but tiem is short and $DAYJOB is taking more time than planned after holidays.
Would you mind creating an issue that list the step that are remaining ? You can copy-past from the 7.2 one.

One of the biggest task is writing the what's new :-)

dsblank · 2019-01-22T13:23:19Z

@Carreau Just saw your note. What "steps remaining" do you mean?

Carreau · 2019-01-22T16:39:42Z

Basically a todo-list, with list of stem necessary before/during/after the release and list of issues /changes that need to be added in the what's new.
I did it yesterday (see #11571). I unfortunately risk to not be able to do any of that before this week-end, so any help welcome.

ENH: add pip and conda magics

8168be5

rgbkrk reviewed Dec 6, 2018

View reviewed changes

print note about kernel restart after installing packages

594d000

Carreau added this to the 7.3 milestone Dec 6, 2018

ipython deleted a comment from MSeal Dec 7, 2018

Update basic.py

c14d220

Carreau merged commit a8165da into ipython:master Dec 10, 2018

jakevdp deleted the pip-conda-magic branch December 12, 2018 16:39

tkf mentioned this pull request Dec 15, 2018

A Documented/Preferred way of installing PyPi dependencies JuliaPy/Conda.jl#50

Closed

weiji14 mentioned this pull request Feb 23, 2019

Add google colaboratory badge to main README.md weiji14/deepbedmap#61

Closed

jakevdp mentioned this pull request Jun 13, 2019

Install packages into the right environment #9517

Open

mdeff mentioned this pull request Mar 25, 2020

Suggest %pip or %conda when user types !pip or !conda #12204

Closed

hackalog mentioned this pull request Jan 16, 2022

Need "CI For Environments hackalog/easydata#234

Open

krassowski mentioned this pull request Jan 29, 2022

Can't load module from jupyter lab jupyterlab/jupyterlab#11959

Closed

psobolewskiPhD mentioned this pull request Feb 19, 2022

admin/update-min-aicsimageio-ver-and-move-to-gpl3 AllenCellModeling/napari-aicsimageio#36

Merged

5 tasks

Uh oh!

Add %pip and %conda magic functions #11524

Add %pip and %conda magic functions #11524

Uh oh!

Conversation

jakevdp commented Dec 6, 2018

Uh oh!

jakevdp commented Dec 6, 2018

Uh oh!

minrk commented Dec 6, 2018

Uh oh!

dsblank commented Dec 6, 2018

Uh oh!

dsblank commented Dec 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ccordoba12 commented Dec 6, 2018

Uh oh!

rgbkrk commented Dec 6, 2018

Uh oh!

rgbkrk Dec 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MSeal Dec 6, 2018

Choose a reason for hiding this comment

Uh oh!

jakevdp Dec 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MSeal Dec 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rgbkrk Dec 6, 2018

Choose a reason for hiding this comment

Uh oh!

mpacer Jul 3, 2019

Choose a reason for hiding this comment

Uh oh!

rgbkrk Dec 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Carreau Dec 6, 2018

Choose a reason for hiding this comment

Uh oh!

jakevdp Dec 6, 2018

Choose a reason for hiding this comment

Uh oh!

dhirschfeld commented Dec 6, 2018

Uh oh!

rgbkrk commented Dec 6, 2018

Uh oh!

MSeal commented Dec 6, 2018

Uh oh!

Carreau commented Dec 6, 2018

Uh oh!

Carreau commented Dec 10, 2018

Uh oh!

rgbkrk commented Dec 10, 2018

Uh oh!

ceceshao1 commented Dec 17, 2018

Uh oh!

dsblank commented Jan 8, 2019

Uh oh!

Carreau commented Jan 8, 2019

Uh oh!

dsblank commented Jan 22, 2019

Uh oh!

Carreau commented Jan 22, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

dsblank commented Dec 6, 2018 •

edited

Loading

rgbkrk Dec 6, 2018 •

edited

Loading

jakevdp Dec 6, 2018 •

edited

Loading

MSeal Dec 6, 2018 •

edited

Loading

rgbkrk Dec 6, 2018 •

edited

Loading