How to train multi-gpu on tensorflow using nccl library?

[kazemSafari]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
  Yes

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
  Linux Ubuntu 16.04

• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
  Local custom build PC

• TensorFlow installed from (source or binary):
  binary

• TensorFlow version (use command below):
  1.10

• Python version:
  3.5

• Bazel version (if compiling from source):
  No

• GCC/Compiler version (if compiling from source):
  gcc (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609

• CUDA/cuDNN version:
  9.0/7.2

• GPU model and memory:
  Two GTX 1080 Ti each with 11G of memory

• Exact command to reproduce:
  IDK

Describe the problem:

Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.

I was wondering if you could provide a tutorial on how to train a simple CNN on multiple gpus on MNIST, or Cifar dataset that also explains how to use the nccl library.
The only tutorials available are the following: https://github.com/tensorflow/models/blob/master/tutorials/image/cifar10/cifar10_multi_gpu_train.py#L101
http://blog.s-schoener.com/2017-12-15-parallel-tensorflow-intro/

https://proteusmaster.urcf.drexel.edu/urcfwiki/index.php/Job_Script_Example_09_TensorFlow_MNIST_Multi-GPU-CNN
None of which explains how to use nccl.

I also search for an answer on many websites but the only one i found is this from openai in Chinese:
http://openresearch.ai/t/nccl-efficient-tensorflow-multigpu-training/159

Although 1/6 of the population of earth does speak Chinese, the other 5/6 does not! So i was wondering if someone in tensorflow team that knows English and perhaps Chinese could help with this issue. Thank you. I would also appreciate if you do me a favor and do not refer or pass me to stackoverflow. Thanks again.

[kazemSafari]

@tfboyd I am wondering if there is a way to do this without using the estimator api. Since in estimator api there is no flexibility and one cannot use it for deep research projects.

Based on the complaint I have seen about tensorflow, i guess people in the community do not have any voice and tensorflow does what it does regardless of the needs of the community.

[tfboyd]

From your comment linked below did not mention you could not use Estimator. I understand your frustration. We spend a lot of time listening to the community and sometimes we fail to get it correct. A lot of people want easy multi-gpu and the best way to do that was via Estimator or Keras that can then handle all the little details. For some top researchers, mostly in performance, they would use the tf_cnn_benchmarks approaches and build from them to add custom logic to some some really neat tricks. The issue with that is the code is hard to follow as it is a sandbox to test new ideas. The site you linked is not OpenAI (Elan Musk) and while I did not take time to read it closely I think the code snippets are from tf_cnn_benchmarks. Reasons we wrapped it:

@anj-s
Can you provide any info on using distribution strategies without Estimator or Keras?

Best of luck finding what you need.

[kazemSafari]

@tfboyd Thank you so much for your answer. I certainly can write my code with Keras. But based on what i have read on Tensorflows latest documentation in order to be able to use all_reduce I have to go back to Estimators, which is what i would prefer to avoid. I would greatly appreciate if you can help me find a way to tune the code in https://github.com/tensorflow/models/blob/master/tutorials/image/cifar10/cifar10_multi_gpu_train.py so that I could use the all_reduce functionality as well even if it might add some complicated code.

The Chinese link i shared above has the following code snippets:

def allreduce_grads(all_grads, average=True):
    from tensorflow.contrib import nccl
    nr_tower = len(all_grads)
    if nr_tower == 1:
        return all_grads
    new_all_grads = []  # N x K
    for grads in zip(*all_grads):
        summed = nccl.all_sum(grads)

        grads_for_devices = []  # K
        for g in summed:
            with tf.device(g.device):
                # tensorflow/benchmarks didn't average gradients
               if average:
                   g = tf.multiply(g, 1.0 / nr_tower, name='allreduce_avg')
            grads_for_devices.append(g)
        new_all_grads.append(grads_for_devices)

    # transpose to K x N
    ret = list(zip(*new_all_grads))
    return ret](url)

[ppwwyyxx]

The link is not Chinese (it's Korean).

I wrote the snippet above originally in tensorpack: https://github.com/tensorpack/tensorpack/blob/81a4fc332eeae7e230e0736958014a0958fca822/tensorpack/graph_builder/utils.py#L140-L170.
You can probably use it for multigpu training very easily: just do tf.gradients on each GPU, then call this function to reduce them. It is used in this file.

[tfboyd]

The link I provided also showed Keras, it was at the top of the document I linked for you. @ppwwyyxx Apologies for guessing the snippets were from the benchmark code. Big thanks for sharing as always.

[anj-s]

@tfboyd You can try using DistributionStrategy APIs directly and your code snippet should look something like this:

  with my_distribution.scope():
    iterator = my_distribution.distribute_dataset(
        dataset).make_one_shot_iterator()
    tower_train_ops = my_distribution.call_for_each_tower(
        tower_fn, iterator.get_next())
    train_op = tf.group(my_distribution.unwrap(tower_train_ops))

If you use MirroredStrategy for distributing your model on multiple GPUs on a single machine DistributionStrategy handles aggregating gradients using allreduce.

From the above conversation it seems like @kazemSafari wants to implement the allreduce functionality. Hence I am not sure if using DistributionStrategy API directly will help.

[kazemSafari]

@tfboyd Thank you so much again. I saw MirroredStrategy is indeed available in Keras on its own. I will let you know if I have more questions.

@anj-s Hi thank you for your explanation. I was wondering if I you could provide a simple example using MirroredStrategy say on a simple dataset like mnist with a simple one layer network that doesn't use estimator or Keras and just using tensorflow.

[kazemSafari]

@ppwwyyxx Thank you for sharing the code. Sorry about my comment. I was trying to follow the code you shared on mnist with a simple mlp:

import tensorflow as tf
import numpy as np

def get_shape(tensor):
    return tensor.get_shape().as_list()


# Source:
# https://stackoverflow.com/questions/38559755/how-to-get-current-available-gpus-in-tensorflow
def get_available_gpus():
    """
        Returns a list of the identifiers of all visible GPUs.
    """
    from tensorflow.python.client import device_lib
    local_device_protos = device_lib.list_local_devices()
    return [x.name for x in local_device_protos if x.device_type == 'GPU']


PS_OPS = [
    'Variable', 'VariableV2', 'AutoReloadVariable', 'MutableHashTable',
    'MutableHashTableOfTensors', 'MutableDenseHashTable'
]


# see https://github.com/tensorflow/tensorflow/issues/9517
def assign_to_device(device, ps_device):
    """Returns a function to place variables on the ps_device.

    Args:
        device: Device for everything but variables
        ps_device: Device to put the variables on. Example values are /GPU:0 and /CPU:0.

    If ps_device is not set then the variables will be placed on the default device.
    The best device for shared varibles depends on the platform as well as the
    model. Start with CPU:0 and then test GPU:0 to see if there is an
    improvement.
    """
    def _assign(op):
        node_def = op if isinstance(op, tf.NodeDef) else op.node_def
        if node_def.op in PS_OPS:
            return ps_device
        else:
            return device
    return _assign


def device_options():
    config = tf.ConfigProto(log_device_placement=False, allow_soft_placement=True)
    with tf.Session(config=config) as sess:
        # your code here
        pass


# Source:
# https://github.com/tensorflow/models/blob/master/tutorials/image/cifar10/cifar10_multi_gpu_train.py#L101
def average_gradients(tower_grads):
    """Calculate the average gradient for each shared variable across all towers.
    Note that this function provides a synchronization point across all towers.
    Args:
    tower_grads: List of lists of (gradient, variable) tuples. The outer list ranges
        over the devices. The inner list ranges over the different variables.
    Returns:
            List of pairs of (gradient, variable) where the gradient has been averaged
            across all towers.
    """
    average_grads = []
    for grad_and_vars in zip(*tower_grads):

        # Note that each grad_and_vars looks like the following:
        #   ((grad0_gpu0, var0_gpu0), ... , (grad0_gpuN, var0_gpuN))
        grads = [g for g, _ in grad_and_vars]
        grad = tf.reduce_mean(grads, 0)

        # Keep in mind that the Variables are redundant because they are shared
        # across towers. So .. we will just return the first tower's pointer to
        # the Variable.
        v = grad_and_vars[0][1]
        grad_and_var = (grad, v)
        average_grads.append(grad_and_var)
    return average_grads


# Source
# https://github.com/tensorpack/tensorpack/blob/81a4fc332eeae7e230e0736958014a0958fca822/tensorpack/graph_builder/utils.py#L140-L170
#################################
def split_grad_list(grad_list):
    """
    Args:
        grad_list: K x N x 2
    Returns:
        K x N: gradients
        K x N: variables
    """
    g = []
    v = []
    for tower in grad_list:
        g.append([x[0] for x in tower])
        v.append([x[1] for x in tower])
    print("g: ")
    print(g)
    print("v: ")
    print(v)
    return g, v


def merge_grad_list(all_grads, all_vars):
    """
    Args:
        all_grads (K x N): gradients
        all_vars(K x N): variables
    Return:
        K x N x 2: list of list of (grad, var) pairs
    """
    all_towers = [list(zip(gs, vs)) for gs, vs in zip(all_grads, all_vars)]
    print("all_towers: ")
    print(all_towers)
    return all_towers


**def allreduce_grads(all_grads, average=True):
    """
    All-reduce average the gradients among K devices. Results are broadcasted to all devices.
    Args:
        all_grads (K x N): List of list of gradients. N is the number of variables.
        average (bool): average gradients or not.
    Returns:
        K x N: same as input, but each grad is replaced by the average over K devices.
    """
    from tensorflow.contrib import nccl
    nr_tower = len(all_grads)
    if nr_tower == 1:
        return all_grads
    new_all_grads = []  # N x K
    for grads in zip(*all_grads):
        print('grads:')
        print(grads)
        summed = nccl.all_sum(grads)

        grads_for_devices = []  # K
        for g in summed:
            with tf.device(g.device):
                # tensorflow/benchmarks didn't average gradients
                if average:
                    g = tf.multiply(g, 1.0 / nr_tower)
            grads_for_devices.append(g)
        new_all_grads.append(grads_for_devices)

    # transpose to K x N
    ret = list(zip(*new_all_grads))
    return ret**
###########################


# Source
# https://github.com/tensorpack/tensorpack/blob/81a4fc332eeae7e230e0736958014a0958fca822/tensorpack/graph_builder/training.py#L240
def apply_gradients(opt, grads):
    devices = get_available_gpus()
    # num_gpus = len(devices)
    raw_devices = ['/gpu:{}'.format(k) for k in devices]
    # optimizer using NCCL
    train_ops = []
    with tf.name_scope('apply_gradients'):
        for idx, grad_and_vars in enumerate(grads):
            with tf.device(raw_devices[idx]):
                # apply_gradients may create variables. Make them LOCAL_VARIABLES
                # with override_to_local_variable(enable=idx > 0):
                train_ops.append(opt.apply_gradients(
                    grad_and_vars, name='apply_grad_{}'.format(idx)))
    train_op = tf.group(*train_ops, name='train_op')
    return train_op


def create_parallel_optimization(X, Y, input_flattened_dim, num_classes, optimizer, controller="/cpu:0"):
    devices = get_available_gpus()
    num_gpus = len(devices)
    # Place all ops on CPU by default
    # with tf.device('/cpu:0'):
    tower_grads = []
    losses = []
    # tf Graph input
    # X = tf.placeholder(tf.float32, [None, num_input])
    # Y = tf.placeholder(tf.float32, [None, num_classes])

    # Split data between GPUs
    X_s = tf.split(X, num_gpus)
    Y_s = tf.split(Y, num_gpus)
    # optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
    # Get the current variable scope so we can reuse all variables we need once we get
    # to the second iteration of the loop below
    with tf.variable_scope(tf.get_variable_scope()) as outer_scope:
        for i, id in enumerate(devices):
            name = 'tower_{}'.format(i)
            # Use the assign_to_device function to ensure that variables are created on the
            # controller.
            with tf.device(assign_to_device(id, controller)), tf.name_scope(name):

                _x = X_s[i]
                _y = Y_s[i]

                logits = mnist_mlp(_x, input_flattened_dim, num_classes)

                # Define loss and optimizer (with train logits, for dropout to take effect)
                loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(
                    logits=logits, labels=_y))

                with tf.name_scope("compute_gradients"):
                    # `compute_gradients` returns a list of (gradient, variable) pairs
                    grads = optimizer.compute_gradients(loss)
                    tower_grads.append(grads)

                losses.append(loss)
                outer_scope.reuse_variables()

    print(tower_grads[0])

    all_grads, all_vars = split_grad_list(tower_grads)
    all_grads = allreduce_grads(all_grads)
    tower_grads = merge_grad_list(all_grads, all_vars)

    train_op = apply_gradients(optimizer, tower_grads)
    # return confusion_mat, equality_op, accuracy_op, train_op
    return train_op


def mnist_mlp(x, input_flattened_dim, num_classes):

    W = tf.get_variable(name='W1',
                        shape=[input_flattened_dim, num_classes],
                        initializer=tf.keras.initializers.glorot_uniform(seed=None),
                        dtype=tf.float32)
    b = tf.get_variable(name='b1',
                        shape=[num_classes],
                        initializer=tf.zeros_initializer(),
                        dtype=tf.float32)
    # Output layer, class prediction
    logits = tf.nn.relu(tf.matmul(x, W) + b)

    return logits


mnist = tf.keras.datasets.mnist

(x_train, y_train),(x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

print(x_train.shape)

x_train = np.reshape(x_train, [x_train.shape[0], -1])
print(x_train.shape)
print(y_train.shape)


input_flattened_dim = 784
num_classes = 10
learning_rate = .0001
max_steps = 100
batch_size = 200
X = tf.placeholder(tf.float32, [None, input_flattened_dim])
Y = tf.placeholder(tf.float32, [None, num_classes])


train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train)).repeat(None).batch(batch_size)
print(train_dataset)
iterator = train_dataset.make_one_shot_iterator()
# extract an element
next_element = iterator.get_next()


optimizer = tf.train.AdamOptimizer(learning_rate)
train_op = create_parallel_optimization(X, Y, input_flattened_dim, num_classes, optimizer, controller="/cpu:0")


with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    for i in range(max_steps):
        batch_x, batch_y = sess.run(next_element)
        sess.run(train_op, feed_dict={X: batch_x, Y: batch_y})

But I get the following error:

(60000, 28, 28)
(60000, 784)
(60000,)
<BatchDataset shapes: ((?, 784), (?,)), types: (tf.float64, tf.uint8)>
2018-10-04 17:02:35.991165: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-10-04 17:02:36.080553: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:897] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-10-04 17:02:36.080933: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties: 
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.607
pciBusID: 0000:01:00.0
totalMemory: 10.91GiB freeMemory: 10.13GiB
2018-10-04 17:02:36.154460: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:897] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-10-04 17:02:36.154821: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 1 with properties: 
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.607
pciBusID: 0000:02:00.0
totalMemory: 10.92GiB freeMemory: 10.76GiB
2018-10-04 17:02:36.155558: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0, 1
2018-10-04 17:02:36.480714: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-10-04 17:02:36.480738: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0 1 
2018-10-04 17:02:36.480743: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N Y 
2018-10-04 17:02:36.480746: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 1:   Y N 
2018-10-04 17:02:36.480964: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/device:GPU:0 with 9790 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2018-10-04 17:02:36.555074: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/device:GPU:1 with 10403 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
[(<tf.Tensor 'tower_0/compute_gradients/gradients/tower_0/MatMul_grad/tuple/control_dependency_1:0' shape=(784, 10) dtype=float32>, <tf.Variable 'W1:0' shape=(784, 10) dtype=float32_ref>), (<tf.Tensor 'tower_0/compute_gradients/gradients/tower_0/add_grad/tuple/control_dependency_1:0' shape=(10,) dtype=float32>, <tf.Variable 'b1:0' shape=(10,) dtype=float32_ref>)]
g: 
[[<tf.Tensor 'tower_0/compute_gradients/gradients/tower_0/MatMul_grad/tuple/control_dependency_1:0' shape=(784, 10) dtype=float32>, <tf.Tensor 'tower_0/compute_gradients/gradients/tower_0/add_grad/tuple/control_dependency_1:0' shape=(10,) dtype=float32>], [<tf.Tensor 'tower_1/compute_gradients/gradients/tower_1/MatMul_grad/tuple/control_dependency_1:0' shape=(784, 10) dtype=float32>, <tf.Tensor 'tower_1/compute_gradients/gradients/tower_1/add_grad/tuple/control_dependency_1:0' shape=(10,) dtype=float32>]]
v: 
[[<tf.Variable 'W1:0' shape=(784, 10) dtype=float32_ref>, <tf.Variable 'b1:0' shape=(10,) dtype=float32_ref>], [<tf.Variable 'W1:0' shape=(784, 10) dtype=float32_ref>, <tf.Variable 'b1:0' shape=(10,) dtype=float32_ref>]]
grads:
(<tf.Tensor 'tower_0/compute_gradients/gradients/tower_0/MatMul_grad/tuple/control_dependency_1:0' shape=(784, 10) dtype=float32>, <tf.Tensor 'tower_1/compute_gradients/gradients/tower_1/MatMul_grad/tuple/control_dependency_1:0' shape=(784, 10) dtype=float32>)
grads:
(<tf.Tensor 'tower_0/compute_gradients/gradients/tower_0/add_grad/tuple/control_dependency_1:0' shape=(10,) dtype=float32>, <tf.Tensor 'tower_1/compute_gradients/gradients/tower_1/add_grad/tuple/control_dependency_1:0' shape=(10,) dtype=float32>)
all_towers: 
[[(<tf.Tensor 'Mul:0' shape=(784, 10) dtype=float32>, <tf.Variable 'W1:0' shape=(784, 10) dtype=float32_ref>), (<tf.Tensor 'Mul_2:0' shape=(10,) dtype=float32>, <tf.Variable 'b1:0' shape=(10,) dtype=float32_ref>)], [(<tf.Tensor 'Mul_1:0' shape=(784, 10) dtype=float32>, <tf.Variable 'W1:0' shape=(784, 10) dtype=float32_ref>), (<tf.Tensor 'Mul_3:0' shape=(10,) dtype=float32>, <tf.Variable 'b1:0' shape=(10,) dtype=float32_ref>)]]
2018-10-04 17:02:38.038636: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0, 1
2018-10-04 17:02:38.038690: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-10-04 17:02:38.038695: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0 1 
2018-10-04 17:02:38.038699: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N Y 
2018-10-04 17:02:38.038701: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 1:   Y N 
2018-10-04 17:02:38.038823: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/device:GPU:0 with 9790 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2018-10-04 17:02:38.038988: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/device:GPU:1 with 10403 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1080 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
Traceback (most recent call last):
  File "/home/kazem/PycharmProjects/Prasad/multi_gpu_example.py", line 270, in <module>
    train_op = create_parallel_optimization(X, Y, input_flattened_dim, num_classes, optimizer, controller="/cpu:0")
  File "/home/kazem/PycharmProjects/Prasad/multi_gpu_example.py", line 220, in create_parallel_optimization
    train_op = apply_gradients(optimizer, tower_grads)
  File "/home/kazem/PycharmProjects/Prasad/multi_gpu_example.py", line 164, in apply_gradients
    with tf.device(raw_devices[idx]):
  File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/home/kazem/tf1.10_p3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 4085, in device
    device_function = pydev.merge_device(device_name_or_function)
  File "/home/kazem/tf1.10_p3/lib/python3.5/site-packages/tensorflow/python/framework/device.py", line 284, in merge_device
    spec = DeviceSpec.from_string(spec or "")
  File "/home/kazem/tf1.10_p3/lib/python3.5/site-packages/tensorflow/python/framework/device.py", line 230, in from_string
    return DeviceSpec().parse_from_string(spec)
  File "/home/kazem/tf1.10_p3/lib/python3.5/site-packages/tensorflow/python/framework/device.py", line 165, in parse_from_string
    self.device_index = int(y[1])
**ValueError: invalid literal for int() with base 10:** ''

Could you provide me with a simple example of using all_reduce. Thanks again.

[ppwwyyxx]

Your error is from the invalid device name generated in your code at this line:

    raw_devices = ['/gpu:{}'.format(k) for k in devices]

In your code, devices already contain device names therefore you should not add /gpu: to it.

[kazemSafari]

@ppwwyyxx Thanks. It works in the simple example. but i couldn't figure out how to use batch normalization or print accuracy. It also seems to scale poorly in performance. So I guess i have to go the route of using Keras. Thanks again.

[ppwwyyxx]

There are other things to do to scale in addition to using nccl. That's why there are tensorflow/benchmarks, distribution strategy (and tensorpack as well) to do the other necessary work to scale.

Going back to the original issue, I also hope to see some examples that use distribution strategy without Estimator or Keras.

[kazemSafari]

@ppwwyyxx Could provide me with some information about the things and factors i need to know to help scale up?
or maybe refer me to a place where i can learn more about them and how to implement them as well. Since i was training an RGAN for a company i was interning at this summer and we found out that it is very challenging to scale up performance even though we used tfrecords.

[tensorflowbutler]

Nagging Assignee @anj-s: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

[tfboyd]

@kazemSafari heads up there is an API being reviewed to give access to multi-gpu API without other wrappers. You may want to look at it to see how it may fit your needs/wants.

tensorflow/community#25