New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switching GPUs before training when tensorflow is used #1602

Open
snurkabill opened this Issue Jan 30, 2016 · 23 comments

Comments

Projects
None yet
@snurkabill
Contributor

snurkabill commented Jan 30, 2016

Hi,

I would like to ask, how one can change used GPU when keras is used? I have to GPUs on my machine and I would like to run two separate scripts on them, parametrized by GPU id.

Is it possible?

Use case in TF is to run session with device("/gpu:x") where "x" is GPU's id.

@grahamannett

This comment has been minimized.

Show comment
Hide comment
@grahamannett

grahamannett Feb 3, 2016

Contributor

I'm curious about this as well, since theano seems to need the flags/theanorc/configure a device at start, and not listed here: http://keras.io/faq/#how-can-i-run-keras-on-gpu and I tried to get it to work with (prior to keras fitting model)

tf.Session() as sess:
    with tf.device("/gpu:1"):

but it didn't work.

Contributor

grahamannett commented Feb 3, 2016

I'm curious about this as well, since theano seems to need the flags/theanorc/configure a device at start, and not listed here: http://keras.io/faq/#how-can-i-run-keras-on-gpu and I tried to get it to work with (prior to keras fitting model)

tf.Session() as sess:
    with tf.device("/gpu:1"):

but it didn't work.

@parag2489

This comment has been minimized.

Show comment
Hide comment
@parag2489

parag2489 Feb 3, 2016

Contributor

This article running Keras on multiple GPUs may give you some direction, it's untested though.

Contributor

parag2489 commented Feb 3, 2016

This article running Keras on multiple GPUs may give you some direction, it's untested though.

@jfsantos

This comment has been minimized.

Show comment
Hide comment
@jfsantos

jfsantos Mar 7, 2016

Contributor

The referenced article is only valid for Theano, and setting a device on .theanorc or running Python with THEANO_FLAGS=device=gpu0 also works. Any ideas on how to make this on Tensorflow? I have a machine with 2 GPUs and would like to run two experiments in parallel, one in each GPU, but Tensorflow always picks the first GPU by default.

Contributor

jfsantos commented Mar 7, 2016

The referenced article is only valid for Theano, and setting a device on .theanorc or running Python with THEANO_FLAGS=device=gpu0 also works. Any ideas on how to make this on Tensorflow? I have a machine with 2 GPUs and would like to run two experiments in parallel, one in each GPU, but Tensorflow always picks the first GPU by default.

@parag2489

This comment has been minimized.

Show comment
Hide comment
@parag2489

parag2489 Mar 7, 2016

Contributor

I think this link explains about using multiple GPUs, their placement etc. in Tensorflow.

Contributor

parag2489 commented Mar 7, 2016

I think this link explains about using multiple GPUs, their placement etc. in Tensorflow.

@jfsantos

This comment has been minimized.

Show comment
Hide comment
@jfsantos

jfsantos Mar 7, 2016

Contributor

Following the link @parag2489 posted, I did the following: created my model inside a context created using with tf.device('/gpu:1'): and changed the session to allow soft placement (in case the model uses any Tensorflow operations that do not have GPU implementations). Here's an example:

import keras.backend.tensorflow_backend as K

with K.tf.device('/gpu:1'):
    K._set_session(K.tf.Session(config=K.tf.ConfigProto(allow_soft_placement=True, log_device_placement=True)))
    model = Sequential()
   # define the rest of your model here...
    model.compile(loss='mse', optimizer='adam')

(note: you should set log_device_placement=False, I'm using it just for debugging)

This seems to work properly, but I did not test with all operations to make sure.

@fchollet Any comments on this? Should we add a function to configure the Tensorflow backend to specify a device? I think we could maybe hide device selection easily using the variable and placeholder functions.

Contributor

jfsantos commented Mar 7, 2016

Following the link @parag2489 posted, I did the following: created my model inside a context created using with tf.device('/gpu:1'): and changed the session to allow soft placement (in case the model uses any Tensorflow operations that do not have GPU implementations). Here's an example:

import keras.backend.tensorflow_backend as K

with K.tf.device('/gpu:1'):
    K._set_session(K.tf.Session(config=K.tf.ConfigProto(allow_soft_placement=True, log_device_placement=True)))
    model = Sequential()
   # define the rest of your model here...
    model.compile(loss='mse', optimizer='adam')

(note: you should set log_device_placement=False, I'm using it just for debugging)

This seems to work properly, but I did not test with all operations to make sure.

@fchollet Any comments on this? Should we add a function to configure the Tensorflow backend to specify a device? I think we could maybe hide device selection easily using the variable and placeholder functions.

@grahamannett

This comment has been minimized.

Show comment
Hide comment
@grahamannett

grahamannett Mar 8, 2016

Contributor

Seems like there could be a function for every backend that allows the user to set gpu/cpu stuff through keras and not through theano/tensorflow/etc

Contributor

grahamannett commented Mar 8, 2016

Seems like there could be a function for every backend that allows the user to set gpu/cpu stuff through keras and not through theano/tensorflow/etc

@snurkabill

This comment has been minimized.

Show comment
Hide comment
@snurkabill

snurkabill Mar 8, 2016

Contributor

solved by:
#1918

Contributor

snurkabill commented Mar 8, 2016

solved by:
#1918

@jfsantos

This comment has been minimized.

Show comment
Hide comment
@jfsantos

jfsantos Mar 8, 2016

Contributor

@grahamannett If we design such an API, we have to keep in mind that you can allocate different parts of a model to different devices (but at the same time, do not make it extremely complicated). It is straightforward to do in Tensorflow, and there is a currently experimental way to do it in Theano as well.

Contributor

jfsantos commented Mar 8, 2016

@grahamannett If we design such an API, we have to keep in mind that you can allocate different parts of a model to different devices (but at the same time, do not make it extremely complicated). It is straightforward to do in Tensorflow, and there is a currently experimental way to do it in Theano as well.

@fchollet

This comment has been minimized.

Show comment
Hide comment
@fchollet

fchollet Mar 8, 2016

Collaborator

Any PR to add such functionality to the backend would be really welcome!

On 7 March 2016 at 18:46, João Felipe Santos notifications@github.com
wrote:

@grahamannett https://github.com/grahamannett If we design such an API,
we have to keep in mind that you can allocate different parts of a model to
different devices (but at the same time, do not make it extremely
complicated). It is straightforward to do in Tensorflow, and there is a
currently experimental way
http://deeplearning.net/software/theano/tutorial/using_multi_gpu.html
to do it in Theano as well.


Reply to this email directly or view it on GitHub
#1602 (comment).

Collaborator

fchollet commented Mar 8, 2016

Any PR to add such functionality to the backend would be really welcome!

On 7 March 2016 at 18:46, João Felipe Santos notifications@github.com
wrote:

@grahamannett https://github.com/grahamannett If we design such an API,
we have to keep in mind that you can allocate different parts of a model to
different devices (but at the same time, do not make it extremely
complicated). It is straightforward to do in Tensorflow, and there is a
currently experimental way
http://deeplearning.net/software/theano/tutorial/using_multi_gpu.html
to do it in Theano as well.


Reply to this email directly or view it on GitHub
#1602 (comment).

@jfsantos

This comment has been minimized.

Show comment
Hide comment
@jfsantos

jfsantos Mar 8, 2016

Contributor

@fchollet Do you think we should specify devices on a per-layer basis? The API would get a little bloated, though. What I would really like to do is something similar to what is done in Tensorflow, where you can specify a "device context" and anything declared within that scope is allocated to that device. Any ideas?

Contributor

jfsantos commented Mar 8, 2016

@fchollet Do you think we should specify devices on a per-layer basis? The API would get a little bloated, though. What I would really like to do is something similar to what is done in Tensorflow, where you can specify a "device context" and anything declared within that scope is allocated to that device. Any ideas?

@fchollet

This comment has been minimized.

Show comment
Hide comment
@fchollet

fchollet Mar 8, 2016

Collaborator

Why not just what TF is doing? It seems like a good API.

By the way, note that the variables used by a layer are instantiated when
that layer gets connected to the next one. Something to keep in mind with
regard to scopes.

On 8 March 2016 at 05:16, João Felipe Santos notifications@github.com
wrote:

@fchollet https://github.com/fchollet Do you think we should specify
devices on a per-layer basis? The API would get a little bloated, though.
What I would really like to do is something similar to what is done in
Tensorflow, where you can specify a "device context" and anything declared
within that scope is allocated to that device. Any ideas?


Reply to this email directly or view it on GitHub
#1602 (comment).

Collaborator

fchollet commented Mar 8, 2016

Why not just what TF is doing? It seems like a good API.

By the way, note that the variables used by a layer are instantiated when
that layer gets connected to the next one. Something to keep in mind with
regard to scopes.

On 8 March 2016 at 05:16, João Felipe Santos notifications@github.com
wrote:

@fchollet https://github.com/fchollet Do you think we should specify
devices on a per-layer basis? The API would get a little bloated, though.
What I would really like to do is something similar to what is done in
Tensorflow, where you can specify a "device context" and anything declared
within that scope is allocated to that device. Any ideas?


Reply to this email directly or view it on GitHub
#1602 (comment).

@jfsantos

This comment has been minimized.

Show comment
Hide comment
@jfsantos

jfsantos Mar 8, 2016

Contributor

I think we could do something similar to what Tensorflow is doing and adapt the Theano backend to do the same thing. My idea is to add an implementation for the Theano backend that replaces the device parameter in Theano calls by the device specified by our scopes.

Contributor

jfsantos commented Mar 8, 2016

I think we could do something similar to what Tensorflow is doing and adapt the Theano backend to do the same thing. My idea is to add an implementation for the Theano backend that replaces the device parameter in Theano calls by the device specified by our scopes.

@leocnj

This comment has been minimized.

Show comment
Hide comment
@leocnj

leocnj Dec 28, 2016

@snurkabill If possible, will you please post a recipe showing how to use this newly added function? Thanks.

leocnj commented Dec 28, 2016

@snurkabill If possible, will you please post a recipe showing how to use this newly added function? Thanks.

@snurkabill

This comment has been minimized.

Show comment
Hide comment
@snurkabill

snurkabill Jan 3, 2017

Contributor

@leocnj just use keras syntax in "with("/:gpu42")" statement. It's same as TF has, but propagated via keras

Contributor

snurkabill commented Jan 3, 2017

@leocnj just use keras syntax in "with("/:gpu42")" statement. It's same as TF has, but propagated via keras

@wolfrevoda

This comment has been minimized.

Show comment
Hide comment
@wolfrevoda

wolfrevoda Mar 15, 2017

use a instruct of CUDA_VISIBLE_DEVICES to choice which gpu can be seen and decide which gpu can be used indirectly

wolfrevoda commented Mar 15, 2017

use a instruct of CUDA_VISIBLE_DEVICES to choice which gpu can be seen and decide which gpu can be used indirectly

@ktamiola

This comment has been minimized.

Show comment
Hide comment
@ktamiola

ktamiola Mar 23, 2017

@jfsantos I have tried to follow:

import keras.backend.tensorflow_backend as K

with K.tf.device('/gpu:1'):
    K._set_session(K.tf.Session(config=K.tf.ConfigProto(allow_soft_placement=True, log_device_placement=True)))
    model = Sequential()
   # define the rest of your model here...
    model.compile(loss='mse', optimizer='adam')

however I get the following error:

  2
      3 with K.tf.device('/gpu:1'):
----> 4     K._set_session(K.tf.Session(config=K.tf.ConfigProto(allow_soft_placement=True, log_device_placement=True)))
      5     model = Sequential()
      6    # define the rest of your model here...

AttributeError: 'module' object has no attribute '_set_session'

Keras 2.0.2 / Tensorflow 0.12.1

ktamiola commented Mar 23, 2017

@jfsantos I have tried to follow:

import keras.backend.tensorflow_backend as K

with K.tf.device('/gpu:1'):
    K._set_session(K.tf.Session(config=K.tf.ConfigProto(allow_soft_placement=True, log_device_placement=True)))
    model = Sequential()
   # define the rest of your model here...
    model.compile(loss='mse', optimizer='adam')

however I get the following error:

  2
      3 with K.tf.device('/gpu:1'):
----> 4     K._set_session(K.tf.Session(config=K.tf.ConfigProto(allow_soft_placement=True, log_device_placement=True)))
      5     model = Sequential()
      6    # define the rest of your model here...

AttributeError: 'module' object has no attribute '_set_session'

Keras 2.0.2 / Tensorflow 0.12.1

@jfsantos

This comment has been minimized.

Show comment
Hide comment
@jfsantos

jfsantos Mar 27, 2017

Contributor

@ktamiola Keras 2.0 was rewritten, so that API has probably changed. If you want to use only one GPU, use the CUDA_VISIBLE_DEVICES approach instead.

Contributor

jfsantos commented Mar 27, 2017

@ktamiola Keras 2.0 was rewritten, so that API has probably changed. If you want to use only one GPU, use the CUDA_VISIBLE_DEVICES approach instead.

@tanjosh

This comment has been minimized.

Show comment
Hide comment
@tanjosh

tanjosh Apr 14, 2017

@jfsantos I am getting same error as @ktamiola and I have to use 2nd GPU.

tanjosh commented Apr 14, 2017

@jfsantos I am getting same error as @ktamiola and I have to use 2nd GPU.

@Vishruit

This comment has been minimized.

Show comment
Hide comment
@Vishruit

Vishruit May 1, 2017

@ktamiola @tanjosh You can use K.set_session(...). Have tried it works!

But I am still looking for "data parellelism" in Keras... :/ Any luck?

Vishruit commented May 1, 2017

@ktamiola @tanjosh You can use K.set_session(...). Have tried it works!

But I am still looking for "data parellelism" in Keras... :/ Any luck?

@skjerns

This comment has been minimized.

Show comment
Hide comment
@skjerns

skjerns Jun 7, 2017

I am trying to run two models on two GPUs at the same time inside the same script.

Seems like CUDA_VISIBLE_DEVICES only works when used before starting my script, but I want to switch GPUs during training/validation.

I've tried

with K.tf.device('/gpu:{}'.format(gpu)):
    K.set_session(K.tf.Session(config=K.tf.ConfigProto(allow_soft_placement=True, log_device_placement=True)))
    model = Sequential(name='ann')
    model.add(Dense 50, input_shape=input_shape, activation='elu', kernel_initializer='he_normal'))
    model.add(BatchNormalization())
    model.add(Dense(n_classes, activation = 'softmax'))
    model.compile(loss='categorical_crossentropy', optimizer=Adam(), metrics=[keras.metrics.categorical_accuracy])

But I get a lot of errors:

WARNING:tensorflow:Tried to colocate 
gradients_1/batch_normalization_4/moments/sufficient_statistics/count_grad/Const_1 with an op 
batch_normalization_4/moments/sufficient_statistics/count that had a different device: /device:CPU:0 vs 
/device:GPU:0. Ignoring colocation property.

should the soft placement not prevent those warnings? can I just ignore them? the model seems to run fine.

skjerns commented Jun 7, 2017

I am trying to run two models on two GPUs at the same time inside the same script.

Seems like CUDA_VISIBLE_DEVICES only works when used before starting my script, but I want to switch GPUs during training/validation.

I've tried

with K.tf.device('/gpu:{}'.format(gpu)):
    K.set_session(K.tf.Session(config=K.tf.ConfigProto(allow_soft_placement=True, log_device_placement=True)))
    model = Sequential(name='ann')
    model.add(Dense 50, input_shape=input_shape, activation='elu', kernel_initializer='he_normal'))
    model.add(BatchNormalization())
    model.add(Dense(n_classes, activation = 'softmax'))
    model.compile(loss='categorical_crossentropy', optimizer=Adam(), metrics=[keras.metrics.categorical_accuracy])

But I get a lot of errors:

WARNING:tensorflow:Tried to colocate 
gradients_1/batch_normalization_4/moments/sufficient_statistics/count_grad/Const_1 with an op 
batch_normalization_4/moments/sufficient_statistics/count that had a different device: /device:CPU:0 vs 
/device:GPU:0. Ignoring colocation property.

should the soft placement not prevent those warnings? can I just ignore them? the model seems to run fine.

@raginisharma14

This comment has been minimized.

Show comment
Hide comment
@raginisharma14

raginisharma14 Aug 28, 2017

@Vishruit Could you please paste the code you tried because k.set_session is not working for me.

raginisharma14 commented Aug 28, 2017

@Vishruit Could you please paste the code you tried because k.set_session is not working for me.

@stale

This comment has been minimized.

Show comment
Hide comment
@stale

stale bot Nov 26, 2017

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.

stale bot commented Nov 26, 2017

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.

@stale stale bot added the stale label Nov 26, 2017

@RaphaelRoyerRivard

This comment has been minimized.

Show comment
Hide comment
@RaphaelRoyerRivard

RaphaelRoyerRivard Jul 24, 2018

I tried with K.tf.device('/cpu:0'): but got a AttributeError: module 'keras.backend.tensorflow_backend' has no attribute 'device'. Any chance this feature is still available in the latest version of Keras?
I would like to be able to run predictions on the CPU and fit on the GPU.

RaphaelRoyerRivard commented Jul 24, 2018

I tried with K.tf.device('/cpu:0'): but got a AttributeError: module 'keras.backend.tensorflow_backend' has no attribute 'device'. Any chance this feature is still available in the latest version of Keras?
I would like to be able to run predictions on the CPU and fit on the GPU.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment