how to train using multi-gpu? #15
Assignees
Labels
Comments
|
Hi, TensorLayer naturally supported all features of TensorFlow. |
|
@GuangmingZhu this is a script to run multi-gpu hope it can help you. the |
|
@zsdonghao Thank you! How can I put variables on cpu when I define layers using tensorlayer APIs, since tensorlayer APIs define the variables within the APIs? |
|
@GuangmingZhu As TensorLayer is fully transparent to TensorFlow, you can just simply define your layers under This tutorial can help you understand how to put variables on cpu. |
|
@zsdonghao Great implementation！but, "tensorlayer(1.2.5)/layers.py", |
|
@narrator-wong Thank you for reporting bug. I just make a new commit. The idea is the same with |
|
i think the problem have been solved? |
|
I think the method in tutorial_cifar10_tfrecord.py is not correct. The multiple gpu tutorial put the inference, loss and the gradient all on |
|
@auroua please read the code carefully, |
|
I write a simple code follow the tutorial_cifar10_tfrecord.py. import tensorflow as tf
import tensorlayer as tl
with tf.device('/cpu:0'):
def inference():
x = tf.placeholder(tf.float32, shape=[None, 784], name='x')
network = tl.layers.InputLayer(x, name='input')
network = tl.layers.DropoutLayer(network, keep=0.8, name='drop1')
network = tl.layers.DenseLayer(network, n_units=800, act=tf.nn.relu, name='relu1')
network = tl.layers.DropoutLayer(network, keep=0.5, name='drop2')
network = tl.layers.DenseLayer(network, n_units=800, act=tf.nn.relu, name='relu2')
network = tl.layers.DropoutLayer(network, keep=0.5, name='drop3')
network = tl.layers.DenseLayer(network, n_units=10, act=tf.identity, name='output')
return network
if __name__ == '__main__':
with tf.device('/gpu:0'):
network = inference()
network.print_layers()
sess = tf.Session(config=tf.ConfigProto(
allow_soft_placement=True,
log_device_placement=True))
tl.layers.initialize_global_variables(sess)The following is the code output: The parameters and inference are all placed on |
|
I modified the class DenseLayer(Layer):
"""
The :class:`DenseLayer` class is a fully connected layer.
Parameters
----------
layer : a :class:`Layer` instance
The `Layer` class feeding into this layer.
n_units : int
The number of units of the layer.
act : activation function
The function that is applied to the layer activations.
W_init : weights initializer
The initializer for initializing the weight matrix.
b_init : biases initializer or None
The initializer for initializing the bias vector. If None, skip biases.
W_init_args : dictionary
The arguments for the weights tf.get_variable.
b_init_args : dictionary
The arguments for the biases tf.get_variable.
name : a string or None
An optional name to attach to this layer.
Examples
--------
>>> network = tl.layers.InputLayer(x, name='input_layer')
>>> network = tl.layers.DenseLayer(
... network,
... n_units=800,
... act = tf.nn.relu,
... W_init=tf.truncated_normal_initializer(stddev=0.1),
... name ='relu_layer'
... )
>>> Without TensorLayer, you can do as follow.
>>> W = tf.Variable(
... tf.random_uniform([n_in, n_units], -1.0, 1.0), name='W')
>>> b = tf.Variable(tf.zeros(shape=[n_units]), name='b')
>>> y = tf.nn.relu(tf.matmul(inputs, W) + b)
Notes
-----
If the input to this layer has more than two axes, it need to flatten the
input by using :class:`FlattenLayer` in this case.
"""
def __init__(
self,
layer=None,
n_units=100,
act=tf.identity,
W_init=tf.truncated_normal_initializer(stddev=0.1),
b_init=tf.constant_initializer(value=0.0),
W_init_args={},
b_init_args={},
name='dense_layer',
):
Layer.__init__(self, name=name)
self.inputs = layer.outputs
if self.inputs.get_shape().ndims != 2:
raise Exception("The input dimension must be rank 2, please reshape or flatten it")
n_in = int(self.inputs.get_shape()[-1])
self.n_units = n_units
print(" [TL] DenseLayer %s: %d %s" % (self.name, self.n_units, act.__name__))
with tf.variable_scope(name) as vs:
with tf.device('/cpu:0'):
W = tf.get_variable(name='W', shape=(n_in, n_units), initializer=W_init, dtype=D_TYPE, **W_init_args)
if b_init is not None:
try:
with tf.device('/cpu:0'):
b = tf.get_variable(name='b', shape=(n_units), initializer=b_init, dtype=D_TYPE, **b_init_args)
except: # If initializer is a constant, do not specify shape.
with tf.device('/cpu:0'):
b = tf.get_variable(name='b', initializer=b_init, dtype=D_TYPE, **b_init_args)
self.outputs = act(tf.matmul(self.inputs, W) + b)
else:
self.outputs = act(tf.matmul(self.inputs, W))
# Hint : list(), dict() is pass by value (shallow), without them, it is
# pass by reference.
self.all_layers = list(layer.all_layers)
self.all_params = list(layer.all_params)
self.all_drop = dict(layer.all_drop)
self.all_layers.extend([self.outputs])
if b_init is not None:
self.all_params.extend([W, b])
else:
self.all_params.extend([W])This is the modified code: def inference():
x = tf.placeholder(tf.float32, shape=[None, 784], name='x')
network = tl.layers.InputLayer(x, name='input')
network = tl.layers.DropoutLayer(network, keep=0.8, name='drop1')
network = DenseLayer(network, n_units=800, act=tf.nn.relu, name='relu1')
network = tl.layers.DropoutLayer(network, keep=0.5, name='drop2')
network = DenseLayer(network, n_units=800, act=tf.nn.relu, name='relu2')
network = tl.layers.DropoutLayer(network, keep=0.5, name='drop3')
network = DenseLayer(network, n_units=10, act=tf.identity, name='output')
return network
if __name__ == '__main__':
with tf.device('/gpu:0'):
network = inference()
network.print_layers()
sess = tf.Session(config=tf.ConfigProto(
allow_soft_placement=True,
log_device_placement=True))
tl.layers.initialize_global_variables(sess)This is the placement result: |
|
Shall it be merged in the main repository @zsdonghao or please close the issue. |
|
@auroua @DEKHTIARJonathan I think |
zsdonghao
pushed a commit
that referenced
this issue
May 4, 2019
fix a minor bug in test_auto_naming.py
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Tensorflow supports multi-gpu training, has tensorlayer supported multi-gpu training already?
The text was updated successfully, but these errors were encountered: