Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add %pip and %conda magic functions #11524

Merged
merged 3 commits into from Dec 10, 2018
Merged

Conversation

@jakevdp
Copy link
Contributor

@jakevdp jakevdp commented Dec 6, 2018

This PR adds %pip and %conda magic functions which automatically install packages into the currently-running kernel in an IPython or Jupyter notebook session.

screen shot 2018-12-05 at 9 41 51 pm

Why? Correctly installing Python packages within a running kernel has long been a point of confusion within the Jupyter ecosystem. While this might seem like an anti-pattern for the traditional local kernel use-case, the recent popularity of cloud-backed Jupyter notebook services has made this more relevant.

For some background on why !pip install or !conda install are incorrect in ways that can be very subtle and confusing, see my blog post from last year: Installing Python Packages From a Jupyter Notebook.

There are some existing issues requesting this sort of feature (see #9517 & #11481) as well as some prior art on pip and conda magic function implementations by @minrk (https://github.com/minrk/condamagic/) and @Carreau (https://github.com/Carreau/pip_magic), but the fact that these third-party functions must be manually installed limits their usefulness to people who are having trouble installing things 馃槃.

I'm hoping that including functionality like this within IPython itself will address the bulk of the confusion many of us have seen from users of the Jupyter stack.

@jakevdp
Copy link
Contributor Author

@jakevdp jakevdp commented Dec 6, 2018

Needs some work on tests, but that can wait until there's some consensus on whether this feature should be added or not!

Loading

@minrk
Copy link
Member

@minrk minrk commented Dec 6, 2018

I think this is a great thing to add, personally.

Loading

@dsblank
Copy link
Contributor

@dsblank dsblank commented Dec 6, 2018

Yes! #11481 I'll take a look, and, going forward, will always try to help maintain keep refined. This is important.

Loading

@dsblank
Copy link
Contributor

@dsblank dsblank commented Dec 6, 2018

Very clever on scanning the history, looking for the path! Never would have thought of that.

What will the advice be to users who don't know if a package can be install by conda or pip? I was originally thinking a general %install would try conda first, and then pip if not available.

Loading

@ccordoba12
Copy link
Member

@ccordoba12 ccordoba12 commented Dec 6, 2018

We're also having problems with people using !pip and !conda in Spyder, so this is great! Thanks @jakevdp!

Could it be possible to intercept the use of !pip/!conda and print a message saying to people to use the magics instead? Or does the magics addition do that automatically?

Loading

@rgbkrk
Copy link
Member

@rgbkrk rgbkrk commented Dec 6, 2018

Thank you so much @jakevdp

Loading

def _is_conda_environment():
"""Return True if the current Python executable is in a conda env"""
# TODO: does this need to change on windows?
conda_history = os.path.join(sys.prefix, 'conda-meta', 'history')
Copy link
Member

@rgbkrk rgbkrk Dec 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dhirschfeld would this correctly detect being in a conda environment on Windows?

Loading

Copy link
Contributor

@MSeal MSeal Dec 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A better way might be to check if Anaconda is in sys.version. That should work cross-platform.

Loading

Copy link
Contributor Author

@jakevdp jakevdp Dec 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that only works if you happen to be using the Python built by Anaconda. For example:

$ conda create -n forge-python
$ conda activate forge-python
$ conda install python --yes -c conda-forge
$ python -c "import sys; print(sys.version)"
3.7.1 | packaged by conda-forge | (default, Nov 13 2018, 09:50:42) 
[Clang 9.0.0 (clang-900.0.37)]

Loading

Copy link
Contributor

@MSeal MSeal Dec 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh yes you're right

Loading

'install', 'list', 'remove', 'uninstall', 'update', 'upgrade',
}
CONDA_COMMANDS_REQUIRING_YES = {
'install', 'remove', 'uninstall', 'update', 'upgrade',
Copy link
Member

@rgbkrk rgbkrk Dec 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love this organization around args needing --y or the --prefix

Loading

Copy link
Member

@mpacer mpacer Jul 3, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It really is gloriously nice.

Loading

Usage:
%pip install [pkgs]
"""
self.shell.system(' '.join([sys.executable, '-m', 'pip', line]))
Copy link
Member

@rgbkrk rgbkrk Dec 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This works pretty well, the only hidden API surface to this is that this allows the full pip API. We likely wouldn't be able to extend this with new args, possibly for something ipython / jupyter specific (capturing metadata, accepting a flag to do so).

Needless to say, I'm cool with this implementation.

Loading

Copy link
Member

@Carreau Carreau Dec 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 We likely want to remind user they might need to restart the kernel.

Loading

Copy link
Contributor Author

@jakevdp jakevdp Dec 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 594d000

Loading

@dhirschfeld
Copy link
Contributor

@dhirschfeld dhirschfeld commented Dec 6, 2018

There might be an issue on Windows where you can silently corrupt your installation by trying to update a compiled extension when Windows is holding a reference to the DLL. Certainly, with pandas that used to often happen. I still kill all Python processes before doing any upgrades, but maybe that's no longer required and conda has worked around the original issue.

I'll try to check the current behaviour and report back...

Loading

@rgbkrk
Copy link
Member

@rgbkrk rgbkrk commented Dec 6, 2018

@Carreau @MSeal self.shell.system is one of the calls that doesn't error when a shell command happens right? We should make sure this causes an error so that use of this magic will stop execution in nbconvert, papermill, and other run-all use cases.

Loading

@MSeal
Copy link
Contributor

@MSeal MSeal commented Dec 6, 2018

I need to dig into that but yes one of the function magic options isn't erroring as one would expect.

Loading

@Carreau Carreau added this to the 7.3 milestone Dec 6, 2018
@Carreau
Copy link
Member

@Carreau Carreau commented Dec 6, 2018

I guess we could also use subprocess.check_call or something. I didn't even remembered we had shell.system

Loading

@ipython ipython deleted a comment from MSeal Dec 7, 2018
@Carreau
Copy link
Member

@Carreau Carreau commented Dec 10, 2018

I fixed the tests.
Let's get that in. I'll try to make a release close to end of month to keep the monthly release cadence.

Loading

@Carreau Carreau merged commit a8165da into ipython:master Dec 10, 2018
4 checks passed
Loading
@rgbkrk
Copy link
Member

@rgbkrk rgbkrk commented Dec 10, 2018

Thanks y'all!

Loading

@ceceshao1
Copy link

@ceceshao1 ceceshao1 commented Dec 17, 2018

Can't wait for this! Was just experiencing confusion today around this specific action

Loading

@dsblank
Copy link
Contributor

@dsblank dsblank commented Jan 8, 2019

@Carreau Time for a 7.3 release?

Loading

@Carreau
Copy link
Member

@Carreau Carreau commented Jan 8, 2019

Yes, on my todo list, but tiem is short and $DAYJOB is taking more time than planned after holidays.
Would you mind creating an issue that list the step that are remaining ? You can copy-past from the 7.2 one.

One of the biggest task is writing the what's new :-)

Loading

@dsblank
Copy link
Contributor

@dsblank dsblank commented Jan 22, 2019

@Carreau Just saw your note. What "steps remaining" do you mean?

Loading

@Carreau
Copy link
Member

@Carreau Carreau commented Jan 22, 2019

Basically a todo-list, with list of stem necessary before/during/after the release and list of issues /changes that need to be added in the what's new.
I did it yesterday (see #11571). I unfortunately risk to not be able to do any of that before this week-end, so any help welcome.

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

10 participants