Ubuntu 13.04 based AMIs are broken #408

Open
ohthetrees opened this Issue Jul 16, 2014 · 21 comments

8 participants

@ohthetrees

Ubuntu has withdrawn support for 13.04, and now it seems that all starcluster ubuntu 13.04 based AMI's are broken, and without the ability to install or upgrade software via apt-get.

I respectfully request that the starcluster folks create some 14.04 based AMIs, and retire the 13.04 based AMIs. I took a crack at it upgrading to 14.04, but failed, it was a bit over my head.

@darnells

I am having the same problem as @ohthetrees. I cannot use the pkginstaller plugin because apt-get fails since Canonical removed the 13.04 security repository.

Error:
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/raring-security/main/source/Sources 404 Not Found [IP: 91.189.91.15 80]

Security repository missing for 13.04 Raring Ringtail (EOL 2014-01-27)
http://91.189.91.15/ubuntu/dists/

@brspurri

Same issue here. Anyone have any advice?

@darnells

@brspurri: As for myself, I am rolling back to use the previous 12.04 StarCluster AMI. It is going to take me some time to confirm that my project again works correctly on 12.04; however, I expect that it should be straight forward.

Like @ohthetrees stated before, I too respectfully request a 14.04 based AMI.

@vasisht

you can run 'do-release-upgrade' to upgrade the current AMI and generate a new AMI.

Note that portmap has been replaced by rpcbind
#409

@cancan101

Are there any instructions on what I need to do to make my own 14.04 AMI (along the lines of these cookbooks)?

@ohthetrees
@cancan101

I think that you can get by with a pretty vanilla AMI. I recall having to install SGE or this and NFS.

You can then use this to allow installing packages like NFS when starting the cluster.

Optionally, I installed:

  • OpenBLAS
@andrewcstewart

The current stagnation in AMIs reflects the problem with using pre-baked images vs automated provisioning of vanilla images. Unfortunately it's beginning to look as though StarCluster is in danger of becoming abandonware, so updated AMIs are tied to active development of SC in general.. If StarCluster AMis were provisioned through something like Ansible, it would be simpler for the community to keep generating more AMIs.

There are some projects like ElastiCluster (https://elasticluster.readthedocs.org/en/latest/) that take that approach, though it doesn't seem like an incredibly mature product either.

@cancan101

I think that you are getting caught up on needing a "StarCluster AMI". You should be able to get by with any Ubuntu AMI. Just install SGE (you should be able to use package manager or the plugin I linked to) and nfs (again you can use the apt-get plugin).

You might want to patch this out assuming that however you installed SGE does not install to/opt/sge6-fresh.

@andrewcstewart

Are you saying that I could just use, say, Amazon's vanilla ubuntu 14.04 AMI as a starter and just include SGE and NFS in the PkgInstaller plugin? I was under the impression that SC has some helper scripts and similar things baked into the public SC AMis. (starcluster listpublic)

Would you mind providing an example of what the pkginstaller plugin section should look like then, along with any relevant portions of the cluster config section?

@darnells
@andrewcstewart

Actually i'll go ahead and take stab at it and maybe you can correct where I'm wrong:

[plugin pkginstaller]
SETUP_CLASS = starcluster.plugins.pkginstaller.PackageInstaller
PACKAGES = nfs-kernel-server, nfs-common, sge #?

Where does one disable the default SGE plugin then as noted in your link?

@cancan101

Mine looks like:

[plugin cluster_install]
setup_class = starcluster.plugins.pkginstaller.PackageInstaller
packages = rpcbind, nfs-common, nfs-kernel-server, rxvt, protobuf-compiler, python-protobuf, fftw3-dev

And the list of configs:

PLUGINS = bash-runner-master, bash-runner, cluster_install, default, sgeinstaller, msg-pack-installer, installer-ipython, ipcluster, setup-spearmint, setup-swap, bash-runner2, sge
@cancan101

I install SGE using:

[plugin sgeinstaller]
setup_class = starcluster.plugins.sgeinstaller.SGEInstaller
path = https://dl.dropboxusercontent.com/...

but there are packages for it. You just have to deal with starcluster expecting a very specific install directory.

@andrewcstewart

Thanks for the info. I was able to get a working Ubuntu 14.04 AMI from the author of the youtube video linked above.

@cancan101 I think you're on the right path for devising an automated 'no-bake' solution. There's still a bit of setup involved there for the non-poweruser, which I imagine is the intended audience of StarCluster.

@cancan101

If you have a list of the commands you ran to build the image, feel free to post them here. I would be curious if I am missing something given that I haven't rebuilt an AMI in a few months.

@andrewcstewart
@ppau1

@andrewcstewart,

I just instanced it, re-imaged it, and copied it to my region.

It would be great if you could give a more detailed explanation of the steps involved to accomplish this?

@andrewcstewart
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment