An example build on AWS using latest generation GPU instances

I have been unsuccessfully trying to get Caffe to install on latest GPU instances, is it possible to provide a public AMI that has caffe pre-installed?

[shelhamer]

Make sure you pick a GPU instance that has compute capability >= 3.0.

The wiki has a reference to an AMI but I'm not sure that it's up-to-date: https://github.com/BVLC/caffe/wiki/Setting-up-Caffe-on-Ubuntu-14.04

@cdoersch could you comment on any instance details from your recent installation?

[cdoersch]

I've only gotten it to run on g2.2xlarge instances, which are the newest GPU's on EC2. I was using the starcluster HVM AMI which is ubuntu 12.04. Confusingly, it comes with its own version of cuda and the nvidia driver that's too old to run caffe. I find that from stock ubuntu, this does the trick:

wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1204/x86_64/cuda-repo-ubuntu1204_6.5-14_amd64.deb
dpkg -i cuda-repo-ubuntu1204_6.5-14_amd64.deb
apt-get update
apt-get install -y cuda

Otherwise, I just followed the directions from the caffe website.

[kloudkl]

There are a few docker files for Caffe.

[mmoghimi]

I followed the instructions to setup caffe on aws but still have issues related to CUDA.

@cdoersch do you an ami that you can share?

[cdoersch]

I unfortunately don't have one at the moment; there's some additional customizations on the machine I'm using (not to mention that I'm currently running out of AWS funds). If you have a more specific issue, post it and I may be able to help.

[mmoghimi]

@cdoersch here is the error message.
http://pastebin.com/hfeEMVty

[cdoersch]

Looks like your GPU isn't recognized. What's the result of nvidia-smi -a?

[mmoghimi]

modprobe: ERROR: could not insert 'nvidia_340': Unknown symbol in module, or unknown parameter (see dmesg)
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

[cdoersch]

Did you get any errors when you ran apt-get install -y cuda? On my system, that installs the 340 driver.

[mmoghimi]

I just launched a new instance and did everything from scratch.
apt-get install cuda installs 340 but I get error when I'm trying to run make runtest
then I uninstalled
sudo apt-get remove --purge nvidia-340 nvidia-modprobe nvidia and installed them from the .run file NVIDIA-Linux-x86_64-340.46.run and installed it.

Still doesn't work.

E1016 03:36:38.640488 13775 common.cpp:98] Cannot create Curand generator. Curand won't be available.
F1016 03:36:38.640584 13775 benchmark.cpp:87] Check failed: error == cudaSuccess (38 vs. 0) no CUDA-capable device is detected

Any thoughts?

[cdoersch]

That apt-get remove would have gotten rid of cuda too, wouldn't it?

For me, cuda installs into /usr/local/cuda-6.5 when done via apt-get install cuda. Make sure it's there.

Which version of ubuntu are you using? If it's a more recent version, you may have an nvidia-340 in the ubuntu repositories, which may cause conflicts with what you would get from nvidia's repositories.

Also, this isn't a caffe problem, it's a problem with your install of cuda. I wouldn't bother running caffe again until you can get nvidia-smi -a to output your graphics card model. That program will hopefully give more helpful error messages.

[kmatzen]

@AKSHAYUBHAT, I have an AMI that works with caffe on g2.2xlarge. We can chat offline if you want access, but I'm pretty sure it's just the HVM Ubuntu 14.04 AMI with cuda 6.5 and docker installed. Then I use my kmatzen/caffe or kmatzen/caffe-debug docker image. Both are available on hub.docker.com.

sudo docker run -t -i --privileged -v /mnt/datastore:/datastore kmatzen/caffe /bin/bash

https://registry.hub.docker.com/u/kmatzen/caffe/dockerfile/
https://registry.hub.docker.com/u/kmatzen/caffe-debug/dockerfile/

I also have a docker image called kmatzen/caffe-base that includes just the dependencies. The Dockerfile can be found in my repo:
https://github.com/kmatzen/caffe/blob/mesos/docker/base/Dockerfile

One thing you might want to change is that this Dockerfile references my mesos-base docker image. You could just change it to reference the ubuntu:14.04 docker image.

[shelhamer]

@kmatzen your explanation and docker images could help a lot of new users -- if you have a chance, please add a section (perhaps under installation) to the wiki https://github.com/BVLC/caffe/wiki.