Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tf.ConfigProto usage on TF 2.0 #25446

Closed
samsamoa opened this issue Feb 2, 2019 · 17 comments
Closed

tf.ConfigProto usage on TF 2.0 #25446

samsamoa opened this issue Feb 2, 2019 · 17 comments
Assignees
Labels
TF 2.0 Issues relating to TensorFlow 2.0 type:others issues not falling in bug, perfromance, support, build and install or feature

Comments

@samsamoa
Copy link

samsamoa commented Feb 2, 2019

On TensorFlow 1.X, there are various important parameters set by passing tf.ConfigProto to tf.Session(config=...) or tf.enable_eager_execution(config=...). For example, to use NCCL, it is useful to set the visible GPUs for a session with config.gpu_options.visible_device_list.

My understanding is that TensorFlow 2.0 no longer has a way to set this configuration — both tf.Session and tf.enable_eager_execution are gone. Is there an alternate way to set this config?

Related StackOverflow question
CC @girving @allenlavoie

@dynamicwebpaige dynamicwebpaige added TF 2.0 Issues relating to TensorFlow 2.0 type:others issues not falling in bug, perfromance, support, build and install or feature labels Feb 2, 2019
@dynamicwebpaige
Copy link
Contributor

Potentially related: #25138.

@jvishnuvardhan jvishnuvardhan added the stat:awaiting tensorflower Status - Awaiting response from tensorflower label Feb 4, 2019
@jvishnuvardhan jvishnuvardhan removed the stat:awaiting tensorflower Status - Awaiting response from tensorflower label Feb 4, 2019
@martinwicke
Copy link
Member

Gaurav is working on a proper replacement, it's not ready yet, for now please use compat.v1.

@girving
Copy link
Contributor

girving commented Feb 5, 2019

We don't know how to use compat.v1 to do this. If there is a way, would it be possible to answer the Stackoverflow question so that it would be possible for us to try TF 2? The implication from @allenlavoie was that no workaround existed.

@dynamicwebpaige
Copy link
Contributor

@girving - If you use the upgrade script on your original TensorFlow model, tf.Session(config=...) should change to tf.compat.v1.Session(config=...) and work as expected.

@martinwicke
Copy link
Member

martinwicke commented Feb 5, 2019 via email

@samsamoa
Copy link
Author

samsamoa commented Feb 5, 2019

If I want to experiment with writing "true TF 2.0 code", will tf.compat.v1.enable_eager_execution(config=...) have the desired effect? In other words, should I expect that we'll end up with a drop-in replacement for the above command?

@martinwicke
Copy link
Member

martinwicke commented Feb 5, 2019 via email

@dynamicwebpaige dynamicwebpaige added this to In progress in TensorFlow 2.0 Feb 5, 2019
@jaingaurav
Copy link
Contributor

@samsamoa: I've landed a fix to #25138. I'm continuing to work on additional config options. Could you please share which specific options are of interest to you?

@girving
Copy link
Contributor

girving commented Feb 25, 2019

@jaingaurav I use config.device_count['GPU'] and config.gpu_options.visible_device_list currently.

@cottrell
Copy link

cottrell commented Apr 3, 2019

@samsamoa: I've landed a fix to #25138. I'm continuing to work on additional config options. Could you please share which specific options are of interest to you?

allow_growth appears to be an essential hack to fix some cudnn issue on rtx20xx devices.

config.gpu_options.allow_growth = True
config.log_device_placement = True # to log device placement (on which device the operation ran)

@jaingaurav
Copy link
Contributor

A number of new API were added in tf.config namespace to support this use case. Please let me know if there is anything we missed regarding this specific issue.

@opcecco
Copy link

opcecco commented Dec 5, 2019

Looks like you can do this in TF 2.0 now like this:

gpu_devices = tf.config.experimental.list_physical_devices('GPU')
for device in gpu_devices:
    tf.config.experimental.set_memory_growth(device, True)

@leimao
Copy link

leimao commented Dec 13, 2019

Check https://www.tensorflow.org/guide/gpu#limiting_gpu_memory_growth

@mehdishafiei
Copy link

@opcecco you are a lifesaver.
Thank you.

@Kokonut133
Copy link

Looks like you can do this in TF 2.0 now like this:

gpu_devices = tf.config.experimental.list_physical_devices('GPU')
for device in gpu_devices:
    tf.config.experimental.set_memory_growth(device, True)

I have placed this right before defining a keras Model() instance and compiling it. My taskmanager yet still shows that indepdently all video card memory is taken. What am I doing wrong?

@RoboEvangelist
Copy link

how do we add this in C++?

@martinwicke
Copy link
Member

The C++ API still uses sessions, so use the session config as you would in 1.x.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
TF 2.0 Issues relating to TensorFlow 2.0 type:others issues not falling in bug, perfromance, support, build and install or feature
Projects
None yet
Development

No branches or pull requests