Can't start StarCluster after upgrading to IPython 0.13

[capk]

I tried upgrading IPython on two different StarCluster AMIs (their default 64-bit Ubuntu 11.10 and the QIIME 1.5 image). In either case, when I start up my cluster the initialization script hangs at 'Waiting for JSON connector file...'. I logged in to the instances and, indeed, even though an ipcluster daemon was running, it hadn't written any JSON files to profile_default/security.

Maybe this is a difference between where IPython 0.12 and 0.13 put their connector files? Or maybe the updated ipcluster hangs for some reason?

[minrk]

There's a bug in the SGE launchers in 0.13. I think it should work if you use master (pip install https://github.com/ipython/ipython/zipball/master).

[capk]

Oh! That explains it, thanks. Any idea when the fix will make it out to PyPI-- I'll have a harder time convincing people to try starcluster+ipython if the latter requires a custom install.

[minrk]

We intend to have 0.13.1 by the end of the summer.

The bug is in a configurable, so you can workaround it with the following in your ipcluster_config.py on starcluster:

c.SGEControllerLauncher.batch_template = """
#$ -V
#$ -S /bin/sh
#$ -N ipcontroller
ipcontroller --log-to-file --profile-dir="{profile_dir}" --cluster-id="{cluster_id}"
"""

c.SGEEngineSetLauncher.batch_template = """
#$ -V
#$ -t 1-{n}
#$ -S /bin/sh
#$ -N ipengine
ipengine --profile-dir="{profile_dir}" --cluster-id="{cluster_id}"
"""

[minrk]

Also, you can check out our 0.13.1 backport branch to see what it will look like (only bugfixes), but know that that branch will probably be clobbered and rebased a number of times while we figure out how best to build a maintenance branch.

[capk]

Thanks for the info.

An unrelated question: I find it quite tricky to get all the imports and
global variables distributed across the cluster. Picloud does a good job of
making a similar situation much more convenient: their cloud.map traces all
of a function's dependencies (obviously excluding native libraries) and
sends them over the wire along with the function. Any chance you all might
implement a smarter function pickler for IPython?

On Sat, Jul 21, 2012 at 5:08 PM, Min RK <
reply@reply.github.com

wrote:

Also, you can check out our 0.13.1 backport branch to see what it will
look like (only bugfixes), but know that that branch will probably be
clobbered and rebased a number of times while we figure out how best to
build a maintenance branch.

---

Reply to this email directly or view it on GitHub:
#2171 (comment)

[fperez]

@capk, it would be useful if you ca file a new issue with the specific features you'd find useful as improvements. That will help anyone wanting to tackle this particular problem.

We've certainly thought about handling closures more robustly, just haven't had the time to dig into that.

ps - I'm closing this one as I don't think there's any issue left here to fix.

[jtriley]

@minrk Do you know if your fix would break 0.12? If it doesn't hurt 0.12 I'll just add the fix to the ipcluster plugin verbatim. Otherwise I'll add a special case for 0.13. I can figure this out by trial and error if you're not sure or busy.

[minrk]

I am pretty sure the fix should have no effect on 0.12, but I would check to be sure.

[minrk]

For further reference, starcluster itself now has this workaround applied for 0.13.