It's possible to do this today, but the interface isn't very intuitive. TL;DR: The GPUOptions in the session creation will be ignored, and you have to pass them when you create the tf.train.Server objects to which these settings apply. (The reason for this is that the GPU device is created when you create the server, not when you create the session. When you use non-distributed TensorFlow, the device is created when you create the session.)

To set this option then, you have to set the tf.ServerDef.default_session_config.gpu_options.per_process_gpu_memory_fraction field when you create the server. Currently you can only do that if you build the tf.train.ServerDef yourself. I'll add a interface to let you override the tf.ConfigProto on its own while still using the Pythonic sugar for defining a cluster, and use that to close this issue.

@mrry What if I didn't define any tf.train.Server object? And my code is sth like sess = sv.prepare_or_wait_for_session(config=tf.ConfigProto(gpu_options=gpu_options)). So where should I pass the config object to? TIA!