Does Keras support using multiple GPUs?

[henry0312]

Theano has supported multiple GPUs since v0.8.0.
(cf. Using multiple GPUs — Theano 0.8.0 documentation )
Does Keras also support using multiple GPUs?

For example, can I run the below task?

1. Learn a sequential model A on gpu0
2. Learn a sequential model B on gpu1
3. Merge A and B on gpu0

[fchollet]

Yes, can run Keras models on multiple GPUs. This is only possible with the TensorFlow backend for the time being, because the Theano feature is still rather new. We are looking at adding support for multi-gpu in Theano in the near future (it should be fairly straightforward).

With the TensorFlow backend, you can achieve this the same way as you would in pure TensorFlow: by using the with tf.device(d) scope when defining Keras layers.

[henry0312]

We are looking at adding support for multi-gpu in Theano in the near future (it should be fairly straightforward).

I'm looking forward to it 😃
Thank you.

[lemuriandezapada]

tf.device() scope?
Can you expand on this?
I haven't seen it in the api

[jeffzhengye]

Any example to use multiple gpus with TF?

[phalexo]

Hm. Theano has libgpuarray, which allows one to push shared variables to different devices. This will not do all the work for you of recombining weight matrices but with a little effort you could use multiple GPUs.

[nouiz]

There is platoon, a project on top of Theano for data parallelism. Should
be easy to use. We currently focus more on days parallelism then model
parallelism in Theano. But both are possible.

Fred
Le 23 avr. 2016 17:24, "phalexo" notifications@github.com a écrit :

Hm. Theano has libgpuarray, which allows one to push shared variables to
different devices. This will not do all the work for you of recombining
weight matrices but with a little effort you could use multiple GPUs.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#2436 (comment)

[fchollet]

I have looked into Platoon and it seemed like it was pretty much compatible
with Keras out of the box except for a couple lines of code. Easy to adapt,
in any case...

On 25 April 2016 at 05:46, Frédéric Bastien notifications@github.com
wrote:

There is platoon, a project on top of Theano for data parallelism. Should
be easy to use. We currently focus more on days parallelism then model
parallelism in Theano. But both are possible.

Fred
Le 23 avr. 2016 17:24, "phalexo" notifications@github.com a écrit :

Hm. Theano has libgpuarray, which allows one to push shared variables to
different devices. This will not do all the work for you of recombining
weight matrices but with a little effort you could use multiple GPUs.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#2436 (comment)

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#2436 (comment)

[phalexo]

The way libgpuarray work is by mapping variables to different GPUs, and
then function automatically generates code to transfer data between GPUs as
needed.
On Apr 25, 2016 16:13, "François Chollet" notifications@github.com wrote:

I have looked into Platoon and it seemed like it was pretty much compatible
with Keras out of the box except for a couple lines of code. Easy to adapt,
in any case...

On 25 April 2016 at 05:46, Frédéric Bastien notifications@github.com
wrote:

There is platoon, a project on top of Theano for data parallelism. Should
be easy to use. We currently focus more on days parallelism then model
parallelism in Theano. But both are possible.

Fred
Le 23 avr. 2016 17:24, "phalexo" notifications@github.com a écrit :

Hm. Theano has libgpuarray, which allows one to push shared variables
to
different devices. This will not do all the work for you of recombining
weight matrices but with a little effort you could use multiple GPUs.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#2436 (comment)

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#2436 (comment)

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#2436 (comment)

[jtanios]

I have looked into Platoon and it seemed like it was pretty much compatible
with Keras out of the box except for a couple lines of code. Easy to adapt,
in any case...

What's the priority of adding multi GPU support for the theano backend?

[phalexo]

I think it would expand user base for Keras. I have several Titan X in the
same box. Please, take a look at libgpuarray as well.
On Jun 22, 2016 19:54, "themummy" notifications@github.com wrote:

I have looked into Platoon and it seemed like it was pretty much compatible
with Keras out of the box except for a couple lines of code. Easy to adapt,
in any case...

What's the priority of adding multi GPU support for the theano backend?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#2436 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AEY95aBPElrTcVv0ZzPFyDgcDKMaw-0iks5qOcsXgaJpZM4IMTcS
.

[tetmin]

How does this actually work in tensorflow? There is a brief tutorial here: http://blog.keras.io/keras-as-a-simplified-interface-to-tensorflow-tutorial.html, I understand the concept of running the model replicas on seperate GPU devices & then merging the weights, but how do we actually run this? instead of model.fit do we call merged.fit on the result of the merged models?

[pengpaiSH]

@tetmin I have the same confusion as yours. Although the blog shows how to predict model in different GPUs, it is still unclear how to train the same model across different GPUs in a single machine, i.e. I need data parallelism and don't know how to implement it in Keras with TensorFlow as backend.

[rudaoshi]

Agreed with @pengpaiSH and @tetmin . Hope there would be more details.