-
Notifications
You must be signed in to change notification settings - Fork 358
How to switch on GPU mode on multiple nodes #109
Comments
Please check your solver configuration file Andy On Wed, Jul 13, 2016 at 10:57 AM, GnosisYu notifications@github.com wrote:
|
@anfeng I have modified the solver configuration to GPU solver_mode. But it didn't work |
I have another question, how to improve the GPU usage rate, when I submit task (standalone mode), Attached GPUs : 2 GPU is always 9% when task run, is there any config parameter can improve this? BTW, I find when I set spark.task.cpus a large value, the much time the task cost. |
It is possible that GPUs were not available to you, or some of the cluster settings are incorrect. Let's say a node has only one gnu, but two containers are scheduled on it, each of which requires one GPU. One of the containers may not get a GPU if the GPU can not be shared. Please look for "GPU not available" or similar wording in the log file. We tested standalone mode on Amazon EC2 a while back. It worked fine. Regarding GPU usage, there are two possibilities:
For (2), you can increase batch size to see if GPU utilization improves. |
@guchensmile , my case is similar to yours. Do you check the utility of CPU? Actually, starting training did increase utility of GPU a little(5%). Instead, the utility of CPU increased tremendously(100% for 4 cores on each node). So my assumption is that CaffeOnSpark needs CPU to allocate the job and the low utility of GPU is resulted from bottleneck of CPU capability. @junshi15 is this possible? |
@GnosisYu as you said, when I set spark.task.cpus 2, the utility of CPU is almost 200% on both 2 node. @junshi15 Thanks for your reply, I use dataset in https://github.com/yahoo/CaffeOnSpark/wiki/GetStarted_yarn |
@guchensmile . I am pretty sure that GPU was used during the trainning. When I increased the batch size, the utility of GPU increased proportionally. |
I built a spark cluster with several nodes to implement CaffeOnSpark. Each node is equipped with GPU. I compiled CaffeOnSpark with GPU mode and also changed configuration in the solver. However, while I followed tutorial on GetStarted_local and deployed mnist training on multiple nodes (standalone mode), the computation was carried out by CPU instead of GPU. On the contrary, training on single node through CaffeOnSpark used GPU successfully.
So my question is how to switch on GPU mode on multiple nodes.
The text was updated successfully, but these errors were encountered: