Join GitHub today
GitHub is home to over 20 million developers working together to host and review code, manage projects, and build software together.
add ability to bind to a ruby kernel #2643
Conversation
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment Hide comment
ellisonbg
Dec 3, 2012
Owner
The other kernels should be separate repos/projects under the ipython org. I think we should try to make these "packages" in whatever language they are written. So for the ruby kernel, we should be able to easily create a gem and install it like any other ruby package.
The other kernels should be separate repos/projects under the ipython org. I think we should try to make these "packages" in whatever language they are written. So for the ruby kernel, we should be able to easily create a gem and install it like any other ruby package. |
ellisonbg
reviewed
Dec 3, 2012
@@ -28,6 +28,7 @@ | ||
from IPython.config.configurable import LoggingConfigurable | ||
from IPython.utils.importstring import import_item | ||
+from IPython.zmq.rubykernel import RubyKernelManager |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment
Hide comment
ellisonbg
reviewed
Dec 3, 2012
+from IPython.zmq.kernelmanager import KernelManager | ||
+ | ||
+def launch_ruby_kernel(fname='cf.json', **kw): | ||
+ cmd = ['iruby_ruby', '~/iruby/lib/kernel.rb', fname] |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment
Hide comment
ellisonbg
Dec 3, 2012
Owner
Obviously, we will need to make these cmd
lines general before we can merge this. But I think it will be helped if we can create the ruby package and use ruby to find the location of kernel.rb. I guess we need to learn more about ruby packaging and command line installation...
ellisonbg
Dec 3, 2012
Owner
Obviously, we will need to make these cmd
lines general before we can merge this. But I think it will be helped if we can create the ruby package and use ruby to find the location of kernel.rb. I guess we need to learn more about ruby packaging and command line installation...
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment Hide comment
Carreau
Dec 3, 2012
Owner
So, we need to build a 'PopenKernelManager' I guess with a configurable command line....
This should handle pretty much every basic things that are not python...
Do we make some kind of "template" for the command line ? Like %(cfile)
replaced by connexion file ?
So, we need to build a 'PopenKernelManager' I guess with a configurable command line.... This should handle pretty much every basic things that are not python... |
Carreau
added some commits
Dec 4, 2012
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment Hide comment
Carreau
Dec 4, 2012
Owner
does this look better ?
I just have to add this in my config to run the ruby kernel :
[.ipython/profile_ruby/ipython_notebook_config.py]
c = get_config()
c.SubprocessKernelManager.kernel_launch_program =
['/.../ruby','~/.../kernel.rb','{connexion_file_name}']
c.MappingKernelManager.kernel_manager_class=
'IPython.zmq.subprocesskernel.SubprocessKernelManager'
and start notebook with --profile=ruby
I think it should be able to run pretty much anything...
does this look better ? I just have to add this in my config to run the ruby kernel : [.ipython/profile_ruby/ipython_notebook_config.py]
and start notebook with I think it should be able to run pretty much anything... |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment Hide comment
tkf
commented on IPython/zmq/subprocesskernel.py
in bb0b397
Dec 4, 2012
Is |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment Hide comment
probably not.... |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment Hide comment
tkf
commented on IPython/zmq/subprocesskernel.py
in bb0b397
Dec 4, 2012
I guess people normally use |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment Hide comment
Carreau
Dec 4, 2012
Owner
That the kind of variable I consider throwing away as it has no use outside of list comprehension. But I understand what you mean.
I've never find a good way to write those, my preference would have been to write it using map and lambda function, but you also need a useless variable to do so.
it would be great to have a way of telling 'this expression is a lambda function of implicite first ... nth variable called...'
map(expanduser(str.format( λ,foo='bar')), self.kernel_launch_program)
where f(λ,µ,...)
is automatically understand as lambda λ,µ: f(λ,µ,...)
with a convention for the order/name of parameters.
That the kind of variable I consider throwing away as it has no use outside of list comprehension. But I understand what you mean. it would be great to have a way of telling 'this expression is a lambda function of implicite first ... nth variable called...'
where |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment Hide comment
tkf
Dec 4, 2012
Variable leaks from list comprehension but this is Python 2.x's fraw, not feature. It is fixed in Python 3. So, if you write Python 3 compatible code then always you don't use the variable outside of the comprehension, meaning that it is always "throw away variable" by your criteria. I feel it is natural to consider the variable in list comprehension as used variable (if it is used).
I agree it would be nice if Python's anonymous function is shorter :)
tkf
replied
Dec 4, 2012
Variable leaks from list comprehension but this is Python 2.x's fraw, not feature. It is fixed in Python 3. So, if you write Python 3 compatible code then always you don't use the variable outside of the comprehension, meaning that it is always "throw away variable" by your criteria. I feel it is natural to consider the variable in list comprehension as used variable (if it is used). I agree it would be nice if Python's anonymous function is shorter :) |
Carreau
added some commits
Dec 4, 2012
Carreau
referenced this pull request
in JuliaLang/julia
Dec 4, 2012
Closed
zeromq/IPython notebook integration. #1675
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment Hide comment
minrk
Dec 13, 2012
Owner
The existing kernel manager already starts a subprocess, and the launch command could already be customized (via passing launch). For instance, the iruby project which you reference didn't need to touch IPython at all to work. It seems this subclass just adds one configurable as an easier way to do what was already possible. Why not just add this configurable to the base KernelManager class?
The existing kernel manager already starts a subprocess, and the launch command could already be customized (via passing launch). For instance, the iruby project which you reference didn't need to touch IPython at all to work. It seems this subclass just adds one configurable as an easier way to do what was already possible. Why not just add this configurable to the base KernelManager class? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment Hide comment
Carreau
Dec 13, 2012
Owner
I didn't looked deep enough in kernelmanager and missed the configurability.
And actually the original iruby added another kernel launcher to launch the ruby kernel.
I'll try to see if I can do what you suggest.
I didn't looked deep enough in kernelmanager and missed the configurability. And actually the original iruby added another kernel launcher to launch the ruby kernel. I'll try to see if I can do what you suggest. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment Hide comment
minrk
Dec 14, 2012
Owner
And actually the original iruby added another kernel launcher to launch the ruby kernel.
Yes, I wrote it :) - but the important thing is that IPython itself did not need to be modified - the kernel launch args were not configurable, but the launch function was, so it needed a subclass assigned via the KMClass configurable. But that subclass is totally unnecessary if the launch command is configurable in the base class. There is a benefit to adding the configurable to the base class rather than a subclass, because that also means it is inherited by other subclasses (Qt, Blocking, etc.).
The reason it isn't there is probably just that the base KM is not a Configurable, but that's just a historical artifact since it was written before the Config stuff - it absolutely should be a Configurable.
Yes, I wrote it :) - but the important thing is that IPython itself did not need to be modified - the kernel launch args were not configurable, but the launch function was, so it needed a subclass assigned via the KMClass configurable. But that subclass is totally unnecessary if the launch command is configurable in the base class. There is a benefit to adding the configurable to the base class rather than a subclass, because that also means it is inherited by other subclasses (Qt, Blocking, etc.). The reason it isn't there is probably just that the base KM is not a Configurable, but that's just a historical artifact since it was written before the Config stuff - it absolutely should be a Configurable. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment Hide comment
Carreau
Dec 14, 2012
Owner
Question to @minrk,
Making kernelmanager.launch_kernel
configurable will still have people be obliged to write their own launcher function in python to run a custom kernel, Am I wrong ?
Would it make more sens to make entry_point.base_launch_kernel
configurable, and/or to provide a way to have launching process so that you don't have to write any python ?
The reason would be that then, the different kernel wouldn't have to deal with writing/installing any python files/module.
I might be missing something in the way this works though...
Question to @minrk, Making Would it make more sens to make The reason would be that then, the different kernel wouldn't have to deal with writing/installing any python files/module. I might be missing something in the way this works though... |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment Hide comment
minrk
Dec 14, 2012
Owner
Ah, I forgot about the remnants still in entry_point. That means a little bit more needs to change than I thought.
But no, don't make launch_kernel
configurable, make the Popen list itself configurable,
and make appropriate changes to base_launch_kernel to allow that.
Ah, I forgot about the remnants still in entry_point. That means a little bit more needs to change than I thought. But no, don't make |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment Hide comment
minrk
Dec 14, 2012
Owner
a bit more:
base_launch_kernel does two things:
- turn some functional arguments into a Popen command
- add some wrapper stuff for pipes, etc.
It should be split so those are separate actions, and a custom Popen command skips the whole command assembly step.
a bit more: base_launch_kernel does two things:
It should be split so those are separate actions, and a custom Popen command skips the whole command assembly step. |
Carreau
referenced this pull request
in cosmin/IClojure
Jan 7, 2013
Open
IClojure as IPython kernel #27
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment Hide comment
ellisonbg
Jan 11, 2013
Owner
I just had a look at the following modules in IPython.zmq
:
entry_point.py
ipkernel.py
kernelapp.py
kernelmanager.py
At least from a first glance, they are all sort of a rats nest of functions and classes that call each other. It is tempting to start refactoring this in a clean way, but I worry that it is actually quite subtle because of the different ways that kernels are used. But I also remember that this code grew in a very organic way and hasn't been cleaned up since the beginning. @minrk do you have a sense of why things are such a mess - is that needed? Is part of this the complexity of IPython.parallel also using the same kernel?
I just had a look at the following modules in
At least from a first glance, they are all sort of a rats nest of functions and classes that call each other. It is tempting to start refactoring this in a clean way, but I worry that it is actually quite subtle because of the different ways that kernels are used. But I also remember that this code grew in a very organic way and hasn't been cleaned up since the beginning. @minrk do you have a sense of why things are such a mess - is that needed? Is part of this the complexity of IPython.parallel also using the same kernel? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment Hide comment
minrk
Jan 11, 2013
Owner
- I think kernelmanager.py should be fairly well contained. What's wrong with it? Maybe you want to split the channels into their own file? I don't see any particular reason to do that.
- entry_point shouldn't exist, as it's a vestige of the pre-Application way of launching things. Almost all of it is gone now, with just two utility functions left. They can go somewhere else, if there is a better name for it. I never did move them, unlike the rest of its original contents which I did move various other more appropriate places, because I couldn't come up with a better one.
- kernelapp and ipkernel are the way they are because there used to be two kinds of kernels - they should definitely be cleaned up, ipkernel especially. Probably move IPKernelApp to its own ipkernelapp or add it to kernelapp. And the extra launch functions at the bottom probably belong wherever the contents of entry_point end up.
It is totally appropriate to clean this up, but do note that from IPython.zmq.ipkernel import IPKernelApp
is currently in use as a public API. Be aware of PR #2724, because it should probably result in a zmq-free bases of Kernel / KernelManager, from with the zmq ones will inherit. Any attempt at cleanup will conflict severely with that work.
It is totally appropriate to clean this up, but do note that |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment Hide comment
minrk
Jan 11, 2013
Owner
Forgot to mention: No, there shouldn't be anything related to IPython.parallel in the complexity here. The last major step of merging the two kernels is to make IPEngineApp a minimal subclass of IPKernelApp.
Forgot to mention: No, there shouldn't be anything related to IPython.parallel in the complexity here. The last major step of merging the two kernels is to make IPEngineApp a minimal subclass of IPKernelApp. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment Hide comment
ellisonbg
Jan 11, 2013
Owner
Thanks Min that really helps. I guess with #2724 I will just wait. I can accomplish everything I need to with the current design, it is just a bit messy.
- I think the only problem with
kernelmanager.py
is how thelaunch_kernel
function is customized in a wonky way by passing alauncher
kwarg tostart_kernel
. It should probably be a configurable attribute. I have thought about separating the channels into separate code, but that is a minor point. - I would separate the files into three clean ones: put the actual kernel implementation in one file, put the applications in another, and put the launching code in another.
- Is there any need to have separate
KernelApp
andIPKernelApp
classes. It is not at all clear why the two are needed and it means thatIPKernelApp
has multiple inheritance.
In what way is from IPython.zmq.ipkernel import IPKernelApp
a public API? If it makes sense I would love to combine the two applications and put the result into its own file, to leave ipkernel
with just the kernel.
Thanks Min that really helps. I guess with #2724 I will just wait. I can accomplish everything I need to with the current design, it is just a bit messy.
In what way is |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment Hide comment
minrk
Jan 11, 2013
Owner
In what way is from IPython.zmq.ipkernel import IPKernelApp a public API?
If you want to start a kernel and customize it in any way whatsoever, you must make this import. It is equivalent to the embedded classes - IPython.embed
is fine if you don't want to change anything, but you have to import the class itself to do any customization. People are doing this (maybe not many).
I would separate the files into three clean ones: put the actual kernel implementation in one file, put the applications in another, and put the launching code in another.
Sensible
Is there any need to have separate KernelApp and IPKernelApp classes. It is not at all clear why the two are needed and it means that IPKernelApp has multiple inheritance.
The main reason was to separate IPython-shell-specific startup from just 'generic interpreter startup'. Again, the reason being that we used to have a pure Python Kernel. I think the main argument against merging them is that we may again, but that's not a particularly strong one. I would maybe just put IPKernelApp as-is after KernelApp in zmq.kernelapp, but merging the two is fine as well.
If you want to start a kernel and customize it in any way whatsoever, you must make this import. It is equivalent to the embedded classes -
Sensible
The main reason was to separate IPython-shell-specific startup from just 'generic interpreter startup'. Again, the reason being that we used to have a pure Python Kernel. I think the main argument against merging them is that we may again, but that's not a particularly strong one. I would maybe just put IPKernelApp as-is after KernelApp in zmq.kernelapp, but merging the two is fine as well. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment Hide comment
ellisonbg
Jan 11, 2013
Owner
OK great, I will try to help review #2724 and keep my eye on these things
for cleanup.
On Thu, Jan 10, 2013 at 9:58 PM, Min RK notifications@github.com wrote:
In what way is from IPython.zmq.ipkernel import IPKernelApp a public API?
If you want to start a kernel and customize it in any way whatsoever, you
must make this import. It is equivalent to the embedded classes -
IPython.embed is fine if you don't want to change anything, but you have
to import the class itself to do any customization. People are doing this
(maybe not many).I would separate the files into three clean ones: put the actual kernel
implementation in one file, put the applications in another, and put the
launching code in another.Sensible
Is there any need to have separate KernelApp and IPKernelApp classes. It
is not at all clear why the two are needed and it means that IPKernelApp
has multiple inheritance.The main reason was to separate IPython-shell-specific startup from just
'generic interpreter startup'. Again, the reason being that we used to have
a pure Python Kernel. I think the main argument against merging them is
that we may again, but that's not a particularly strong one. I would maybe
just put IPKernelApp as-is after KernelApp in zmq.kernelapp, but merging
the two is fine as well.—
Reply to this email directly or view it on GitHubhttps://github.com/ipython/ipython/pull/2643#issuecomment-12132833.
Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com
OK great, I will try to help review #2724 and keep my eye on these things On Thu, Jan 10, 2013 at 9:58 PM, Min RK notifications@github.com wrote:
Brian E. Granger |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment Hide comment
ellisonbg
Jan 17, 2013
Owner
I cleaned up the kernel manager code a lot in #2775 but didn't really change how the custom launcher is specified. Bu tin either case, I don't think the approach in this branch is the way to do. In the long run, I think we will want to build an additional argument into MultiKernelManager.start_kernel, namely, the name of the kernel to start: ipython, ruby, etc. This list should be where the configuration happens, Each kernel name should be associated with the required launcher. For now can we just get by with what we have until we really get to thinking about multi-kernel support an what that will look like?
I cleaned up the kernel manager code a lot in #2775 but didn't really change how the custom launcher is specified. Bu tin either case, I don't think the approach in this branch is the way to do. In the long run, I think we will want to build an additional argument into MultiKernelManager.start_kernel, namely, the name of the kernel to start: ipython, ruby, etc. This list should be where the configuration happens, Each kernel name should be associated with the required launcher. For now can we just get by with what we have until we really get to thinking about multi-kernel support an what that will look like? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment Hide comment
minrk
Jan 17, 2013
Owner
@ellisonbg that approach doesn't get users anywhere without updates to IPython itself. I think the point here is that we should have a configurable for launching custom kernels, either with
A) a custom launch_kernel function, as a DottedObjectName
or B) a custom Popen command-line list.
Now, maybe this should be a mapping by kernel-name, but in any case, we should expose the fact that the KernelManager as it already is can launch custom kernels. It's just not exposed to config.
It should be straightforward to add one or both of these, particularly now that KernelManager itself is Configurable, as it always should have been.
@ellisonbg that approach doesn't get users anywhere without updates to IPython itself. I think the point here is that we should have a configurable for launching custom kernels, either with A) a custom launch_kernel function, as a DottedObjectName Now, maybe this should be a mapping by kernel-name, but in any case, we should expose the fact that the KernelManager as it already is can launch custom kernels. It's just not exposed to config. It should be straightforward to add one or both of these, particularly now that KernelManager itself is Configurable, as it always should have been. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment Hide comment
ellisonbg
Jan 17, 2013
Owner
I wasn't very clear in my description. I agree that these things should
absolutely be a configurable attribute, not something that is merely passed
into the function in the wonky way that launcher currently is. And I think
that somethink like the API you are proposing makes sense. I just think we
should close this PR and open an issue to have that discussion about what
this should look like. I agree that the work shouldn't be too difficult,
but we also probably want to cleanup the entry_point/base_kernel_launcher
mess.
On Wed, Jan 16, 2013 at 10:18 PM, Min RK notifications@github.com wrote:
@ellisonbg https://github.com/ellisonbg that approach doesn't get users
anywhere without updates to IPython itself. I think the point here is that
we should have a configurable for launching custom kernels, either withA) a custom launch_kernel function, as a DottedObjectName
or B) a custom Popen command-line list.Now, maybe this should be a mapping by kernel-name, but in any case, we
should expose the fact that the KernelManager as it already is can launch
custom kernels. It's just not exposed to config.It should be straightforward to add one or both of these, particularly now
that KernelManager itself is Configurable, as it always should have been.—
Reply to this email directly or view it on GitHubhttps://github.com/ipython/ipython/pull/2643#issuecomment-12355928.
Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com
I wasn't very clear in my description. I agree that these things should On Wed, Jan 16, 2013 at 10:18 PM, Min RK notifications@github.com wrote:
Brian E. Granger |
Carreau
referenced this pull request
Jan 22, 2013
Closed
qtconsole should use kernel_info_request to choose pygments lexer #2827
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment Hide comment
FYI, perl kernel : |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment Hide comment
minrk
Jan 23, 2013
Owner
Actually you did, just not here. I was going to submit a new PR with the changes discussed here (split base_launch_kernel into its two separate actions, and expose Popen command as KM config, avoiding the need for subclass), and suggest closing this when I submit that, rather than closing without having a replacement to point to.
Actually you did, just not here. I was going to submit a new PR with the changes discussed here (split base_launch_kernel into its two separate actions, and expose Popen command as KM config, avoiding the need for subclass), and suggest closing this when I submit that, rather than closing without having a replacement to point to. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Show comment Hide comment
ellisonbg
Jan 23, 2013
Owner
Hehe, if I actually were old I could make some joke about my memory going
bad. Just forgot...sorry for wasting your time...
On Wed, Jan 23, 2013 at 1:06 PM, Min RK notifications@github.com wrote:
Actually you did, just not here. I was going to submit a new PR with the
changes discussed here (split base_launch_kernel into its two separate
actions, and expose Popen command as KM config, avoiding the need for
subclass), and suggest closing this when I submit that, rather than closing
without having a replacement to point to.—
Reply to this email directly or view it on GitHubhttps://github.com/ipython/ipython/pull/2643#issuecomment-12622293.
Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu and ellisonbg@gmail.com
Hehe, if I actually were old I could make some joke about my memory going On Wed, Jan 23, 2013 at 1:06 PM, Min RK notifications@github.com wrote:
Brian E. Granger |
Jan 26, 2013
This was referenced
replaced by #2854 |
Carreau commentedDec 3, 2012
This need the iruby kernel of my repo, (https://github.com/Carreau/iruby)
And to be modified as ruby interpreter and zmq-kernel path in IPython/zmq/rubykernel.py
Launch with (for example)
ipython notebook
--MappingKernelManager.kernel_manager_class='IPython.zmq.rubykernel.RubyKernelManager'
need zmq, uuid and hmac gem, and ruby 1.9.3
I would love to change the ruby interpreter path to
ruby1.9.3
but Even with latest ruby installed, it is not in my path, so I don't know how what the ruby way is.Also any idea of how we want to managed external kernel ? Do we put them in main IPython repo ? keep them separate ?