Can't start StarCluster after upgrading to IPython 0.13 #2171

Closed
capk opened this Issue Jul 20, 2012 · 9 comments

Comments

Projects
None yet
4 participants

capk commented Jul 20, 2012

I tried upgrading IPython on two different StarCluster AMIs (their default 64-bit Ubuntu 11.10 and the QIIME 1.5 image). In either case, when I start up my cluster the initialization script hangs at 'Waiting for JSON connector file...'. I logged in to the instances and, indeed, even though an ipcluster daemon was running, it hadn't written any JSON files to profile_default/security.

Maybe this is a difference between where IPython 0.12 and 0.13 put their connector files? Or maybe the updated ipcluster hangs for some reason?

Owner

minrk commented Jul 21, 2012

There's a bug in the SGE launchers in 0.13. I think it should work if you use master (pip install https://github.com/ipython/ipython/zipball/master).

capk commented Jul 21, 2012

Oh! That explains it, thanks. Any idea when the fix will make it out to PyPI-- I'll have a harder time convincing people to try starcluster+ipython if the latter requires a custom install.

Owner

minrk commented Jul 21, 2012

We intend to have 0.13.1 by the end of the summer.

The bug is in a configurable, so you can workaround it with the following in your ipcluster_config.py on starcluster:

c.SGEControllerLauncher.batch_template = """
#$ -V
#$ -S /bin/sh
#$ -N ipcontroller
ipcontroller --log-to-file --profile-dir="{profile_dir}" --cluster-id="{cluster_id}"
"""

c.SGEEngineSetLauncher.batch_template = """
#$ -V
#$ -t 1-{n}
#$ -S /bin/sh
#$ -N ipengine
ipengine --profile-dir="{profile_dir}" --cluster-id="{cluster_id}"
"""
Owner

minrk commented Jul 21, 2012

Also, you can check out our 0.13.1 backport branch to see what it will look like (only bugfixes), but know that that branch will probably be clobbered and rebased a number of times while we figure out how best to build a maintenance branch.

capk commented Jul 21, 2012

Thanks for the info.

An unrelated question: I find it quite tricky to get all the imports and
global variables distributed across the cluster. Picloud does a good job of
making a similar situation much more convenient: their cloud.map traces all
of a function's dependencies (obviously excluding native libraries) and
sends them over the wire along with the function. Any chance you all might
implement a smarter function pickler for IPython?

On Sat, Jul 21, 2012 at 5:08 PM, Min RK <
reply@reply.github.com

wrote:

Also, you can check out our 0.13.1 backport branch to see what it will
look like (only bugfixes), but know that that branch will probably be
clobbered and rebased a number of times while we figure out how best to
build a maintenance branch.


Reply to this email directly or view it on GitHub:
#2171 (comment)

Owner

fperez commented Jul 24, 2012

@capk, it would be useful if you ca file a new issue with the specific features you'd find useful as improvements. That will help anyone wanting to tackle this particular problem.

We've certainly thought about handling closures more robustly, just haven't had the time to dig into that.

ps - I'm closing this one as I don't think there's any issue left here to fix.

@fperez fperez closed this Jul 24, 2012

Contributor

jtriley commented Jul 24, 2012

@minrk Do you know if your fix would break 0.12? If it doesn't hurt 0.12 I'll just add the fix to the ipcluster plugin verbatim. Otherwise I'll add a special case for 0.13. I can figure this out by trial and error if you're not sure or busy.

@jtriley jtriley referenced this issue in jtriley/StarCluster Jul 24, 2012

Closed

add fix to ipcluster plugin for IPython 0.13 #127

Owner

minrk commented Jul 24, 2012

I am pretty sure the fix should have no effect on 0.12, but I would check to be sure.

minrk added a commit to minrk/StarCluster that referenced this issue Jul 25, 2012

add workaround for ipython/ipython#2171
changes only affect config for IPython 0.13

closes #127
Owner

minrk commented Jul 25, 2012

For further reference, starcluster itself now has this workaround applied for 0.13.

@minrk minrk added this to the 1.0 milestone Mar 26, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment