move the language extensions to a separate repo

[minrk]

I don't think it makes much sense for things like the R, Cython, and Octave extensions to ship with IPython. It ties them to IPython releases, which is a bad fit for their current state of development, and defeats most of the purpose of extensions.

I propose we add a new repo where extensions can reside. One part of this would be to add a 'repo' notion to %install_ext, so you could just do

%install_ext rmagic

and it would fetch the rmagic extension from GitHub, then this ipython-extensions repo could be treated as a repository for anyone to post their own extensions via pull request (a la PyPI).

[tkf]

Why not just upload each extension to PyPI as a separated package? For example if rmagic uses rpy2 it can simply depend on rpy2 which is installed behind the scenes. If rmagic needs some new rpy2 API then version number can be specified in setup.py. I think the same goes for Cython and Octave.

[minrk]

that's certainly another option, though using PyPI is not well suited to hosting single-files.

[tkf]

Why? There is many single-file modules in PyPI, no?

[rkern]

No, you cannot upload single-file modules to PyPI, only sdists and bdists of various flavors.

[tkf]

@rkern I mean you can package signle-file module using sdist.
http://docs.python.org/2/distutils/setupscript.html#listing-individual-modules

[bfroehle]

In terms of syntax, I'd suggest something like @install_ext xxyyzz/rmagic where xxyyzz is an alias (to url or git repo) known to IPython. This seems nicely extensible --- users could register other extension repositories. The IPython repository could be called IPython or contrib or whatever.

[ellisonbg]

I am +1 on both having an ipython-extensions repo and in making install_ext repo aware.

[takluyver]

As I've mentioned before, I'm not sure that a repo is the right approach here. I don't think it scales up well:

• If we hand out commit bits liberally, then before long there are dozens of people who can interfere with popular extensions like cythonmagic.
• If we're sparing with commit bits, it falls on us (or a limited team) to check and merge changes to any extensions, which rather defeats the point of them being extensions.

A github organisation, where different people can control different repos, might be better, but then creating new extensions becomes a burden.

I think we'd be better looking at how we can integrate this with something like PyPI, which is designed to host and deliver packages created by thousands of developers, rather than trying to use Github for something it's not intended for.

[minrk]

I like @bfroehle's idea of making it repo-aware, just for expressing shortcuts to repos, so I could post my extensions to minrk/ipython-extensions, and install them with something like %install_ext minrk/myext. That is, anyone can fork the repo, and anyone can put their own extensions in their own fork.

Even if all we end up adding is nice shortcut syntax for GitHub urls, that would be an improvement, and it would still allow us to split the language extensions into their own repo, which I think is the more important point here.

I don't see any reason for %install_ext to work with PyPI, because pip / easy_install already take care of that. Extensions that people want to distribute as regular Python packages seems like a solved problem to me. But posting and maintaining releases on PyPI is a shitty experience, and an extremely poor fit for simple single-file things like extensions, so I don't think it can be the answer in general.

[tkf]

I thought IPython devs want to provide beginner-friendly environment. Installing Rmagic and finding out that you need to update IPython and install rpy2 from stack trace is not beginner-friendly, IMHO.

[minrk]

you make a good point about the dependencies, and some extensions should be regular packages. But the current situation is definitely bad: extensions are tied to IPython versions for no good reason, so there isn't a mechanism to update the rmagic extension for 0.13.2 (other than downloading it from the master repo). It seems clear that extensions should live somewhere else, but it's not yet clear exactly where.

[ellisonbg]

What about using 1 repo per extension and hosting our language extensions
as repos in the ipython org?

On Tue, Apr 9, 2013 at 12:24 PM, Min RK notifications@github.com wrote:

you make a good point about the dependencies, and some extensions _should_be regular packages. But the current situation is definitely bad:
extensions are tied to IPython versions for no good reason, so there isn't
a mechanism to update the rmagic extension for 0.13.2 (other than
downloading it from the master repo). It seems clear that extensions should
live somewhere else, but it's not yet clear exactly where.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3150#issuecomment-16134397
.

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com

[Carreau]

Quick from my phone.

I like user/repo stuff especially While you are in early dev. It does not
prevent from hosting extension on a common mirror repo if needed and or on
pypi. Other project like component use that. You can register a "main" repo
on component web site, which allows to "search".

-1 on publishing extension under ipython org, if we do so we'll have to
administrate users. I would prefere another organization with a much easier
threshold to become admin.

Le mercredi 10 avril 2013, Brian E. Granger a écrit :

What about using 1 repo per extension and hosting our language extensions
as repos in the ipython org?

On Tue, Apr 9, 2013 at 12:24 PM, Min RK <notifications@github.com<javascript:_e({}, 'cvml', 'notifications@github.com');>>
wrote:

you make a good point about the dependencies, and some extensions
_should_be regular packages. But the current situation is definitely bad:
extensions are tied to IPython versions for no good reason, so there
isn't
a mechanism to update the rmagic extension for 0.13.2 (other than
downloading it from the master repo). It seems clear that extensions
should
live somewhere else, but it's not yet clear exactly where.

—
Reply to this email directly or view it on GitHub<
https://github.com/ipython/ipython/issues/3150#issuecomment-16134397>
.

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu <javascript:_e({}, 'cvml', 'bgranger@calpoly.edu');>and
ellisonbg@gmail.com <javascript:_e({}, 'cvml', 'ellisonbg@gmail.com');>

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3150#issuecomment-16152768
.

[minrk]

Just updating here - we aren't going to get to a decision about the external repo for 1.0, but do know that I plan to move these extensions as soon as there is a place for them.

Bumping to 2.0.

[mgaitan]

Just FYI , I'm maintaining fortran-magic, https://github.com/mgaitan/fortran_magic which is both pip and %install_ext enabled.

IMHO %install_ext is a handy shortcut, but we should avoid to add more features (handle dependencies, uninstall, etc) and "reinvent" the wheel (again): the packaging ecosystem in python is already sufficiently fragmented. So, +1 to use sdist+pypi.

[minrk]

An update: the R magic has moved to rpy2, though we are still shipping a version, just not accepting patches. I don't think any progress has been made on moving oct2py or cython out of IPython.

[rossant]

Will this be closed in 2.0?

[mgaitan]

Would be desirable that the %cython magic lives in the cython package itself? should we ask/raise an issue in cython's github repo?

[Carreau]

That's up to the Cython guys.
I'm just not sure that's their priority right now, and it might have them give commit right to someone they don't know well if none of the current dev are interested.

I also have a PR that fixes css leaks from annotate, that had not get any review in month, so I think they are pretty busy right now.

Envoyé de mon iPhone

Le 10 mars 2014 à 01:21, Martín Gaitán notifications@github.com a écrit :

Would be desirable that the %cython magic lives in the cython package itself? should we ask/raise an issue in cython's github repo?

—
Reply to this email directly or view it on GitHub.

[mgaitan]

Ok, I could move %cython magic to a new repo and then propose a PR removing it from IPython (but not right now -- maybe during the next week).
Should it be removed completely or "deprecated" (for instance, raise a message with instructions to install it from the new repo)?

[rossant]

Maybe it's not worth doing that for 2.0 unless all extensions can be removed from IPython for this release?

[ellisonbg]

I think the cython magic should be moved to the actual Cython package. That
will ensure that it remains in sync with the rest of Cython and is
available to all Cython users. Would love it if someone took the lead on
that :)

Cheers,

Brian

On Mon, Mar 10, 2014 at 6:58 AM, Cyrille Rossant
notifications@github.comwrote:

Maybe it's not worth doing that for 2.0 unless all extensions can be
removed from IPython for this release?

Reply to this email directly or view it on GitHubhttps://github.com//issues/3150#issuecomment-37184335
.

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com

[minrk]

closing as duplicate of #3803

[mgaitan]

@minrk I don't understand wich one you wanted to close, but both issues are still open