Deployment of Julia-ipython for web usage #1

ViralBShah · 2013-04-28T04:04:50Z

This is perhaps a bit of an umbrella issue. I would like to have julia-ipython be ready for the classes this Fall. With stuff such notebookcloud, hosting such a solution for general use on tryjulia.org through EC2 seems like a possibility. As discussed on the mailing list, the earlier web-repl used to get about 50 trials a day and a few hundred a day whenever there was a mention on hackernews etc.

It would be nice to have a list of all the things that have to be done, so that they can be tracked for such a solution to be taken live in the next few months.

So far, I thought JuliaLang/julia#2775 was a bottleneck, but that has been resolved.

stevengj · 2013-07-28T01:57:53Z

@ViralBShah, the IJulia stuff is getting close to being usable (I would also like to use it this fall). Can you look into what we have to do to get this on a server?

Keno · 2013-07-28T03:47:06Z

I've been doing some work with Docker containers. Potentially useful here.

ViralBShah · 2013-08-04T17:14:24Z

Does ipython provide anything for user/session management, so that you can deploy it behind a apache or nginx, which forks off a new session for every new user?

Carreau · 2013-08-04T20:24:50Z

Does ipython provide anything for user/session management, so that you can deploy it behind a apache or nginx, which forks off a new session for every new user?

Not really, currently we make the assumption that the person that start the server wil own all the kernels. There is a simple https/password (no username) mechanisme to prevent other users from logging into others' notebook.

There are tools like hydra and other I forgot the names that spawn one server/user + setup auth on a per user basis.

Real development for multi-user loggin in planed for next year (if nobody outside of core team tackle it :-) )

ViralBShah · 2013-08-05T03:23:52Z

@yarapavan Would it be possible for you to try this out, and see how far we can get in providing a hosted IJulia based on hydra or notebookcloud?

Carreau · 2013-08-05T07:55:09Z

@aculich was it you that showed me another tool to deploy IPython notebook ? Can't remember the name.

Carreau · 2013-08-05T08:00:34Z

Of course 5 min later, I find it : https://github.com/UnataInc/ipydra. I also drafted that some time ago.

yarapavan · 2013-08-05T13:21:41Z

@ViralBShah Interesting. Let me try out this week.

stevengj · 2013-08-09T03:16:29Z

@yarapavan, any progress?

yarapavan · 2013-08-09T04:37:31Z

@stevengj Planned for weekend. will have an update by Monday

yarapavan · 2013-08-11T16:12:43Z

Folks,

I've tried 3 different approaches this weekend.

NotebookCloud
Docker
ipydra (ipython-hydra)
StarCluser (not tested)

Here is a brief summary:

NotebookCloud: Demo Google App Engine URL: https://ijulia-notebookcloud.appspot.com/. Users login using Google Authentication, inputs AWS EC2 Access ID, Secret Key, existing keypair and a password for the notebook cloud. After successful authentication, launches an EC2 instance (http://aws.amazon.com/ec2/instance-types/) using iJulia AMI (ami-487a3c21). The web url for iJulia will be: https://:8888, protected by his password (in the web portal). AMI bundles Julia 0.2 nightly build (as of Aug 10, 2013) + iPython 1.0.0 on Ubuntu 13.04. I seem to hit few issues with iPython kernel restarting few times (AMI issues?). Had issues with (I)Julia core dumping in couple of instance types ( openblas issues? do not try t1.micro for Julia)
Docker.io: LXC + AUFS + cgroups = Docker. This provides better isolation and performance but a minimal ubuntu-based ipython deployment takes about 1-1.5 GB disk space. One can easily do port redirection to expose container ports via host PUBLIC port. EC2 was too slow for me to try docker and had to use my Eucalyptus private cloud to launch couple of iPython instances. Still have to build Julia. Interesting project to note here: https://github.com/miha-stopar/IPyDocker. My docker file for iPython 1.0.0 deployment is here -https://gist.github.com/yarapavan/6200580
iPydra: Inspired from iPython-hydra project, uses Flask and SQLAlchemy (Sqlite3 backend) to spawn iPython notebook servers. When you hit the URL: http://:5555, asks for a user name, and then setups a session, redirecting to any of the ports >=9500. Tested with inbuilt Flask server, which fails to start if child processes are running. It is simple and still buggy.

aculich · 2013-08-11T16:31:27Z

Thanks for the summary of everything you've investigated!

I need to loop all of you into a much wider discussion that we've been
having here
about JiffyLab https://github.com/ptone/jiffylab that has also included
folks
from D-Lab, LBL, Rackspace, Mozilla, and more. There's a lot of momentum
around this, so let's combine forces to move forward in a coordinated way
so we're not continuing to re-invent wheels and also provide a solution that
works well for the IPython Notebook, but that is also easily generalized for
other tools that follow a similar pattern: browser access to process running
in an LXC running on a virtual container.

On Sun, Aug 11, 2013 at 9:12 AM, Pavan notifications@github.com wrote:

Folks,

I've tried 3 different approaches this weekend. Here is a brief summary:

NotebookCloud

Docker

ipydra (ipython-hydra)
4.

StarCluser (not tested)
5.

NotebookCloud: Demo Google App Engine URL:
https://ijulia-notebookcloud.appspot.com/. Users login using Google
Authentication, inputs AWS EC2 Access ID, Secret Key, existing keypair and
a password for the notebook cloud. After successful authentication,
launches an EC2 instance (http://aws.amazon.com/ec2/instance-types/)
using iJulia AMI (ami-487a3c21). The web url for iJulia will be: https://:8888,
protected by his password (in the web portal). AMI bundles Julia 0.2
nightly build (as of Aug 10, 2013) + iPython 1.0.0 on Ubuntu 13.04. I seem
to hit few issues with iPython kernel restarting few times (AMI issues?).
Had issues with (I)Julia core dumping in couple of instance types (
openblas issues? do not try t1.micro for Julia)
6.

Docker.io: LXC + AUFS + cgroups = Docker. This provides better
isolation and performance but a minimal ubuntu-based ipython deployment
takes about 1-1.5 GB disk space. One can easily do port redirection to
expose container ports via host PUBLIC port. EC2 was too slow for me to try
docker and had to use my Eucalyptus private cloud to launch couple of
iPython instances. Still have to build Julia. Interesting project to note
here: https://github.com/miha-stopar/IPyDocker. My docker file for
iPython 1.0.0 deployment is here -
https://gist.github.com/yarapavan/6200580
7.

iPydra: Inspired from iPython-hydra project, uses Flask and SQLAlchemy
(Sqlite3 backend) to spawn iPython notebook servers. When you hit the URL:
http://:5555, asks for a user name, and then setups a session,
redirecting to any of the ports >=9500. Tested with inbuilt Flask server,
which fails to start if child processes are running. It is simple and still
buggy.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1#issuecomment-22460150
.

stevengj · 2013-08-11T17:06:37Z

I'd love to have something working for classes in the fall, ~100 students. What would be our best bet in the near term?

ViralBShah · 2013-08-11T17:21:44Z

I believe @alanedelman will also want something this fall for his classes. We really need to make this work.

Is JiffyLab likely to be ready in the next 2-3 weeks?

@yarapavan I think there are other issues open where micro instances are trouble for julia. Best to use a small or medium instance. If you need API keys, I have some for julia, and I can send those to you. Also, rather than compiling, why not use @staticfloat 's deb packages?

In your opinion of the options 1-3, which one is most likely that is likely to work in the near term?

ViralBShah · 2013-08-11T17:22:33Z

Also pinging @amitmurthy with whom I had discussed these ideas at one point.

yarapavan · 2013-08-11T18:01:39Z

@ViralBShah Thanks. Learnt the hard way, t1.micro doesn't work for Julia! The problem seems to be that the CPU availability varies dramatically and unpredictably (Ref: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts_micro_instances.html). Tried using @staticfloat 's packages but had issues occasionally in other instance types too. Perhaps, these issues are specific to VMs?

@stevengj, @ViralBShah IMO, NotebookCloud is easier to deploy if the backend environment is EC2 (uses python boto library) although Docker-based approach would be the right way to do in near future. Cannot say much about StarCluster (http://star.mit.edu/cluster/index.html) with out evaluating.

ViralBShah · 2013-08-11T20:42:31Z

The relevant issue was: JuliaLang/julia#3263

I believe @JeffBezanson (and perhaps even @stevengj) knows the guys who are doing StarCluster at MIT, and it may be worth investigating that.

stevengj · 2013-08-13T19:42:22Z

I don't know the StarCluster folks.

yarapavan · 2013-08-17T06:37:40Z

Did any one try this -> JQuery Terminal connecting to a 'real' shell running in a EC2/server host to parse/work over http/websockets, or have customized parsing of user commands on the client side.

ViralBShah · 2013-08-17T09:44:07Z

JQuery terminal seems really cool. However, IJulia offers graphical capabilities and among others, it is also good for teaching.

jverzani · 2013-08-27T20:00:35Z

Does anyone have a working solution here? I've been trying to get ipydra to work and am way over my head. After adjusting some environment variables through trial and error and copying a profile-julia to the right place, I can get an IJulia notebook to open, but can't do any work. The kernel dies and the apache log has:

ERROR: key not found: HOME
in getindex at env.jl:25
in path at pkg/dir.jl:9
in find_in_path at loading.jl:18
in find_in_node1_path at loading.jl:29
in require at loading.jl:45
in include at boot.jl:238
in include_from_node1 at loading.jl:96
in include at boot.jl:238
in include_from_node1 at loading.jl:96
in process_options at client.jl:274
in _start at client.jl:349
at /home/julia/.julia/IJulia/src/IJulia.jl:10
at /home/julia/.julia/IJulia/src/kernel.jl:1
2013-08-27 15:56:20.336 [NotebookApp] KernelRestarter: restarting kernel (1/5)
WARNING:root:kernel 4b541d33-c05a-45c0-90ce-031241c0a09d restarted

I guess I'm not quite sure how to set the HOME variable for the www-data user so that julia recognizes it (I'm using Ubuntu + apache2 + wsgi + ipydra)

staticfloat · 2013-08-27T20:04:52Z

Try editing /etc/apache2/envvars. You can put export HOME=<path> inside
of it, but ensure that whatever you set that to, www-data has permissions
to modify it.

HOME is typically set from /etc/passwd, (the second to last field, each
field separate by commas) and usually defaults to something like /var/www.
You could also change it from there, if you like, but the
/etc/apache2/envvars approach is more general for all environment variables.
-E

On Tue, Aug 27, 2013 at 4:00 PM, john verzani notifications@github.comwrote:

Does anyone have a working solution here? I've been trying to get ipydra
to work and am way over my head. After adjusting some environment variables
through trial and error and copying a profile-julia to the right place, I
can get an IJulia notebook to open, but can't do any work. The kernel dies
and the apache log has:

ERROR: key not found: HOME
in getindex at env.jl:25
in path at pkg/dir.jl:9
in find_in_path at loading.jl:18
in find_in_node1_path at loading.jl:29
in require at loading.jl:45
in include at boot.jl:238
in include_from_node1 at loading.jl:96
in include at boot.jl:238
in include_from_node1 at loading.jl:96
in process_options at client.jl:274
in _start at client.jl:349
at /home/julia/.julia/IJulia/src/IJulia.jl:10
at /home/julia/.julia/IJulia/src/kernel.jl:1
2013-08-27 15:56:20.336 [NotebookApp] KernelRestarter: restarting kernel
(1/5)
WARNING:root:kernel 4b541d33-c05a-45c0-90ce-031241c0a09d restarted

I guess I'm not quite sure how to set the HOME variable for the www-data
user so that julia recognizes it (I'm using Ubuntu + apache2 + wsgi +
ipydra)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1#issuecomment-23365959
.

staticfloat · 2013-08-27T20:06:48Z

Oh, wait, you're using uwsgi. Do you have a separate uwsgi master process
running? If so, the /etc/apache2/envvars approach won't work, because the
uwsgi process runs independently from apache, you'll have to modify HOME
for whatever user uwsgi is running as, (possibly www-data, possibly
something else, I'm not sure).

On Tue, Aug 27, 2013 at 4:04 PM, Elliot Saba staticfloat@gmail.com wrote:

Try editing /etc/apache2/envvars. You can put export HOME=<path> inside
of it, but ensure that whatever you set that to, www-data has permissions
to modify it.

HOME is typically set from /etc/passwd, (the second to last field, each
field separate by commas) and usually defaults to something like /var/www.
You could also change it from there, if you like, but the
/etc/apache2/envvars approach is more general for all environment variables.
-E

On Tue, Aug 27, 2013 at 4:00 PM, john verzani notifications@github.comwrote:

Does anyone have a working solution here? I've been trying to get ipydra
to work and am way over my head. After adjusting some environment variables
through trial and error and copying a profile-julia to the right place, I
can get an IJulia notebook to open, but can't do any work. The kernel dies
and the apache log has:

ERROR: key not found: HOME
in getindex at env.jl:25
in path at pkg/dir.jl:9
in find_in_path at loading.jl:18
in find_in_node1_path at loading.jl:29
in require at loading.jl:45
in include at boot.jl:238
in include_from_node1 at loading.jl:96
in include at boot.jl:238
in include_from_node1 at loading.jl:96
in process_options at client.jl:274
in _start at client.jl:349
at /home/julia/.julia/IJulia/src/IJulia.jl:10
at /home/julia/.julia/IJulia/src/kernel.jl:1
2013-08-27 15:56:20.336 [NotebookApp] KernelRestarter: restarting kernel
(1/5)
WARNING:root:kernel 4b541d33-c05a-45c0-90ce-031241c0a09d restarted

I guess I'm not quite sure how to set the HOME variable for the www-data
user so that julia recognizes it (I'm using Ubuntu + apache2 + wsgi +
ipydra)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1#issuecomment-23365959
.

jverzani · 2013-08-27T20:13:49Z

That HOME trick worked, though now it is dying for some other reason that
doesn't show up in the apache logs. (I'm not sure what the u is in uwsgi,
I'm using mod-wsgi with apache.) Thanks for the help.

On Tue, Aug 27, 2013 at 4:06 PM, Elliot Saba notifications@github.comwrote:

Oh, wait, you're using uwsgi. Do you have a separate uwsgi master process
running? If so, the /etc/apache2/envvars approach won't work, because the
uwsgi process runs independently from apache, you'll have to modify HOME
for whatever user uwsgi is running as, (possibly www-data, possibly
something else, I'm not sure).

On Tue, Aug 27, 2013 at 4:04 PM, Elliot Saba staticfloat@gmail.com
wrote:

Try editing /etc/apache2/envvars. You can put export HOME=<path>
inside
of it, but ensure that whatever you set that to, www-data has
permissions
to modify it.

HOME is typically set from /etc/passwd, (the second to last field, each
field separate by commas) and usually defaults to something like
/var/www.
You could also change it from there, if you like, but the
/etc/apache2/envvars approach is more general for all environment
variables.
-E

On Tue, Aug 27, 2013 at 4:00 PM, john verzani notifications@github.comwrote:

Does anyone have a working solution here? I've been trying to get
ipydra
to work and am way over my head. After adjusting some environment
variables
through trial and error and copying a profile-julia to the right place,
I
can get an IJulia notebook to open, but can't do any work. The kernel
dies
and the apache log has:

ERROR: key not found: HOME
in getindex at env.jl:25
in path at pkg/dir.jl:9
in find_in_path at loading.jl:18
in find_in_node1_path at loading.jl:29
in require at loading.jl:45
in include at boot.jl:238
in include_from_node1 at loading.jl:96
in include at boot.jl:238
in include_from_node1 at loading.jl:96
in process_options at client.jl:274
in _start at client.jl:349
at /home/julia/.julia/IJulia/src/IJulia.jl:10
at /home/julia/.julia/IJulia/src/kernel.jl:1
2013-08-27 15:56:20.336 [NotebookApp] KernelRestarter: restarting
kernel
(1/5)
WARNING:root:kernel 4b541d33-c05a-45c0-90ce-031241c0a09d restarted

I guess I'm not quite sure how to set the HOME variable for the
www-data
user so that julia recognizes it (I'm using Ubuntu + apache2 + wsgi +
ipydra)

—
Reply to this email directly or view it on GitHub<
https://github.com/JuliaLang/IJulia.jl/issues/1#issuecomment-23365959>
.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1#issuecomment-23366383
.

John Verzani
Chair, Department of Mathematics
College of Staten Island, CUNY
verzani@math.csi.cuny.edu

jverzani · 2013-08-30T00:29:28Z

Does anyone have a working solution here? Despite the help above, I still couldn't get ipydra + apache to work to show IJulia notebooks. Is it just me? I was hoping to have something in place for about 25 students by the end of next week.

ViralBShah · 2013-08-30T02:04:28Z

@yarapavan Were you able to get the jquery terminal to work? It would be great if we can get something to work for use in classes.

Keno · 2013-08-30T04:55:53Z

I'm working on something. Stay tuned.

marksale · 2013-10-03T23:52:17Z

Total Newby question:
Trying to get IJulia running, got the latest Julia (0.2.0, Aug 26), installed Anaconda Python 1.7, 64 bit, checked add to path and make default Python), installed the Ijulia package (Pkg.add("IJulia"). Launched the Ipython (ipython notebook --profile julia
It started (in a brower) just fine, but I don't seem to have any Julia functionality, e.g., sin(3) gives an error
NameError: name 'sin' is not defined
seems like it isn't running Julia
Any idea what I'm doing wrong?
thanks
Mark (msale2@yahoo.com)

stevengj · 2013-10-04T01:35:54Z

You are running Python, not Julia. See the troubleshooting section of my Tutorial notes. In any case, this issue is not the place to ask questions about installation.

ViralBShah · 2013-11-23T09:35:22Z

Relevant: https://github.com/amitmurthy/JDock

stevengj · 2015-11-27T18:57:40Z

Seems to be closed by JuliaBox.

stevengj mentioned this issue Aug 8, 2013

Replace web repl picture JuliaLang/www_old.julialang.org#43

Closed

dotfelixb mentioned this issue Jan 30, 2014

IJulia kernel dies instantly on OS X #138

Closed

dbartow mentioned this issue Mar 4, 2014

no method countna(Array{UTF8String,1}) - while using show() on a DF #155

Closed

FrancoisFayard mentioned this issue May 19, 2014

Kernel keep dying #178

Closed

stevengj closed this as completed Nov 27, 2015

chelseas mentioned this issue May 8, 2017

Slow/Frozen Jupyter Notebook #536

Closed

fpmenninger mentioned this issue Jul 9, 2018

ERROR: LoadError: Failed to precompile IJulia ; ERROR: LoadError: UndefVarError: uuid4 not defined #676

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deployment of Julia-ipython for web usage #1

Deployment of Julia-ipython for web usage #1

ViralBShah commented Apr 28, 2013

stevengj commented Jul 28, 2013

Keno commented Jul 28, 2013

ViralBShah commented Aug 4, 2013

Carreau commented Aug 4, 2013

ViralBShah commented Aug 5, 2013

Carreau commented Aug 5, 2013

Carreau commented Aug 5, 2013

yarapavan commented Aug 5, 2013

stevengj commented Aug 9, 2013

yarapavan commented Aug 9, 2013

yarapavan commented Aug 11, 2013

aculich commented Aug 11, 2013

stevengj commented Aug 11, 2013

ViralBShah commented Aug 11, 2013

ViralBShah commented Aug 11, 2013

yarapavan commented Aug 11, 2013

ViralBShah commented Aug 11, 2013

stevengj commented Aug 13, 2013

yarapavan commented Aug 17, 2013

ViralBShah commented Aug 17, 2013

jverzani commented Aug 27, 2013

staticfloat commented Aug 27, 2013

staticfloat commented Aug 27, 2013

jverzani commented Aug 27, 2013

jverzani commented Aug 30, 2013

ViralBShah commented Aug 30, 2013

Keno commented Aug 30, 2013

marksale commented Oct 3, 2013

stevengj commented Oct 4, 2013

ViralBShah commented Nov 23, 2013

stevengj commented Nov 27, 2015

Deployment of Julia-ipython for web usage #1

Deployment of Julia-ipython for web usage #1

Comments

ViralBShah commented Apr 28, 2013

stevengj commented Jul 28, 2013

Keno commented Jul 28, 2013

ViralBShah commented Aug 4, 2013

Carreau commented Aug 4, 2013

ViralBShah commented Aug 5, 2013

Carreau commented Aug 5, 2013

Carreau commented Aug 5, 2013

yarapavan commented Aug 5, 2013

stevengj commented Aug 9, 2013

yarapavan commented Aug 9, 2013

yarapavan commented Aug 11, 2013

aculich commented Aug 11, 2013

stevengj commented Aug 11, 2013

ViralBShah commented Aug 11, 2013

ViralBShah commented Aug 11, 2013

yarapavan commented Aug 11, 2013

ViralBShah commented Aug 11, 2013

stevengj commented Aug 13, 2013

yarapavan commented Aug 17, 2013

ViralBShah commented Aug 17, 2013

jverzani commented Aug 27, 2013

staticfloat commented Aug 27, 2013

staticfloat commented Aug 27, 2013

jverzani commented Aug 27, 2013

jverzani commented Aug 30, 2013

ViralBShah commented Aug 30, 2013

Keno commented Aug 30, 2013

marksale commented Oct 3, 2013

stevengj commented Oct 4, 2013

ViralBShah commented Nov 23, 2013

stevengj commented Nov 27, 2015