API - Layers

tensorlayer.layers

Name Scope and Sharing Parameters

These functions help you to reuse parameters for different inference (graph), and get a list of parameters by given name. About TensorFlow parameters sharing click here.

Get variables with name

get_variables_with_name

Get layers with name

get_layers_with_name

Enable layer name reuse

set_name_reuse

Print variables

print_all_variables

Initialize variables

initialize_global_variables

Understanding the Basic Layer

All TensorLayer layers have a number of properties in common:

layer.outputs : a Tensor, the outputs of current layer.

layer.all_params : a list of Tensor, all network variables in order.

layer.all_layers : a list of Tensor, all network outputs in order.

layer.all_drop : a dictionary of {placeholder : float}, all keeping probabilities of noise layers.

All TensorLayer layers have a number of methods in common:

layer.print_params() : print network variable information in order (after tl.layers.initialize_global_variables(sess)). alternatively, print all variables by tl.layers.print_all_variables().

layer.print_layers() : print network layer information in order.

layer.count_params() : print the number of parameters in the network.

A network starts with the input layer and is followed by layers stacked in order. A network is essentially a Layer class. The key properties of a network are network.all_params, network.all_layers and network.all_drop. The all_params is a list which store pointers to all network parameters in order. For example, the following script define a 3 layer network, then:

all_params = [W1, b1, W2, b2, W_out, b_out]

To get specified variable information, you can use network.all_params[2:3] or get_variables_with_name(). all_layers is a list which stores the pointers to the outputs of all layers, see the example as follow:

all_layers = [drop(?,784), relu(?,800), drop(?,800), relu(?,800), drop(?,800)], identity(?,10)]

where ? reflects a given batch size. You can print the layer and parameters information by using network.print_layers() and network.print_params(). To count the number of parameters in a network, run network.count_params().

sess = tf.InteractiveSession()

x = tf.placeholder(tf.float32, shape=[None, 784], name='x')
y_ = tf.placeholder(tf.int64, shape=[None, ], name='y_')

network = tl.layers.InputLayer(x, name='input_layer')
network = tl.layers.DropoutLayer(network, keep=0.8, name='drop1')
network = tl.layers.DenseLayer(network, n_units=800,
                                act = tf.nn.relu, name='relu1')
network = tl.layers.DropoutLayer(network, keep=0.5, name='drop2')
network = tl.layers.DenseLayer(network, n_units=800,
                                act = tf.nn.relu, name='relu2')
network = tl.layers.DropoutLayer(network, keep=0.5, name='drop3')
network = tl.layers.DenseLayer(network, n_units=10,
                                act = tl.activation.identity,
                                name='output_layer')

y = network.outputs
y_op = tf.argmax(tf.nn.softmax(y), 1)

cost = tl.cost.cross_entropy(y, y_)

train_params = network.all_params

train_op = tf.train.AdamOptimizer(learning_rate, beta1=0.9, beta2=0.999,
                            epsilon=1e-08, use_locking=False).minimize(cost, var_list = train_params)

tl.layers.initialize_global_variables(sess)

network.print_params()
network.print_layers()

In addition, network.all_drop is a dictionary which stores the keeping probabilities of all noise layers. In the above network, they represent the keeping probabilities of dropout layers.

In case for training, you can enable all dropout layers as follow:

feed_dict = {x: X_train_a, y_: y_train_a}
feed_dict.update( network.all_drop )
loss, _ = sess.run([cost, train_op], feed_dict=feed_dict)
feed_dict.update( network.all_drop )

In case for evaluating and testing, you can disable all dropout layers as follow.

feed_dict = {x: X_val, y_: y_val}
feed_dict.update(dp_dict)
print("   val loss: %f" % sess.run(cost, feed_dict=feed_dict))
print("   val acc: %f" % np.mean(y_val ==
                        sess.run(y_op, feed_dict=feed_dict)))

For more details, please read the MNIST examples in the example folder.

Layer list

get_variables_with_name get_layers_with_name set_name_reuse print_all_variables initialize_global_variables

Layer

InputLayer OneHotInputLayer Word2vecEmbeddingInputlayer EmbeddingInputlayer AverageEmbeddingInputlayer

DenseLayer ReconLayer DropoutLayer GaussianNoiseLayer DropconnectDenseLayer

Conv1dLayer Conv2dLayer DeConv2dLayer Conv3dLayer DeConv3dLayer

UpSampling2dLayer DownSampling2dLayer AtrousConv1dLayer AtrousConv2dLayer AtrousDeConv2dLayer

Conv1d Conv2d DeConv2d DeConv3d DepthwiseConv2d SeparableConv1d SeparableConv2d DeformableConv2d GroupConv2d

PadLayer PoolLayer ZeroPad1d ZeroPad2d ZeroPad3d MaxPool1d MeanPool1d MaxPool2d MeanPool2d MaxPool3d MeanPool3d GlobalMaxPool1d GlobalMeanPool1d GlobalMaxPool2d GlobalMeanPool2d GlobalMaxPool3d GlobalMeanPool3d

SubpixelConv1d SubpixelConv2d

SpatialTransformer2dAffineLayer transformer batch_transformer

BatchNormLayer LocalResponseNormLayer InstanceNormLayer LayerNormLayer

ROIPoolingLayer

TimeDistributedLayer

RNNLayer BiRNNLayer

ConvRNNCell BasicConvLSTMCell ConvLSTMLayer

advanced_indexing_op retrieve_seq_length_op retrieve_seq_length_op2 retrieve_seq_length_op3 target_mask_op DynamicRNNLayer BiDynamicRNNLayer

Seq2Seq

FlattenLayer ReshapeLayer TransposeLayer

LambdaLayer

ConcatLayer ElementwiseLayer ElementwiseLambdaLayer

ExpandDimsLayer TileLayer

StackLayer UnStackLayer

SlimNetsLayer

BinaryDenseLayer BinaryConv2d TernaryDenseLayer TernaryConv2d DorefaDenseLayer DorefaConv2d SignLayer ScaleLayer

PReluLayer PRelu6Layer PTRelu6Layer

MultiplexerLayer

flatten_reshape clear_layers_name initialize_rnn_state list_remove_repeat merge_networks

Customizing Layers

A Simple Layer

To implement a custom layer in TensorLayer, you will have to write a Python class that subclasses Layer and implement the outputs expression.

The following is an example implementation of a layer that multiplies its input by 2:

class DoubleLayer(Layer):
    def __init__(
        self,
        layer = None,
        name ='double_layer',
    ):
        # check layer name (fixed)
        Layer.__init__(self, layer=layer, name=name)

        # the input of this layer is the output of previous layer (fixed)
        self.inputs = layer.outputs

        # operation (customized)
        self.outputs = self.inputs * 2

        # update layer (customized)
        self.all_layers.append(self.outputs)

Your Dense Layer

Before creating your own TensorLayer layer, let's have a look at the Dense layer. It creates a weight matrix and a bias vector if not exists, and then implements the output expression. At the end, for a layer with parameters, we also append the parameters into all_params.

class MyDenseLayer(Layer):
  def __init__(
      self,
      layer = None,
      n_units = 100,
      act = tf.nn.relu,
      name ='simple_dense',
  ):
      # check layer name (fixed)
      Layer.__init__(self, layer=layer, name=name)

      # the input of this layer is the output of previous layer (fixed)
      self.inputs = layer.outputs

      # print out info (customized)
      print("  MyDenseLayer %s: %d, %s" % (self.name, n_units, act))

      # operation (customized)
      n_in = int(self.inputs._shape[-1])
      with tf.variable_scope(name) as vs:
          # create new parameters
          W = tf.get_variable(name='W', shape=(n_in, n_units))
          b = tf.get_variable(name='b', shape=(n_units))
          # tensor operation
          self.outputs = act(tf.matmul(self.inputs, W) + b)

      # update layer (customized)
      self.all_layers.extend( [self.outputs] )
      self.all_params.extend( [W, b] )

Modifying Pre-train Behaviour

Greedy layer-wise pretraining is an important task for deep neural network initialization, while there are many kinds of pre-training methods according to different network architectures and applications.

For example, the pre-train process of Vanilla Sparse Autoencoder can be implemented by using KL divergence (for sigmoid) as the following code, but for Deep Rectifier Network, the sparsity can be implemented by using the L1 regularization of activation output.

# Vanilla Sparse Autoencoder
beta = 4
rho = 0.15
p_hat = tf.reduce_mean(activation_out, reduction_indices = 0)
KLD = beta * tf.reduce_sum( rho * tf.log(tf.div(rho, p_hat))
        + (1- rho) * tf.log((1- rho)/ (tf.sub(float(1), p_hat))) )

There are many pre-train methods, for this reason, TensorLayer provides a simple way to modify or design your own pre-train method. For Autoencoder, TensorLayer uses ReconLayer.__init__() to define the reconstruction layer and cost function, to define your own cost function, just simply modify the self.cost in ReconLayer.__init__(). To creat your own cost expression please read Tensorflow Math. By default, ReconLayer only updates the weights and biases of previous 1 layer by using self.train_params = self.all _params[-4:], where the 4 parameters are [W_encoder, b_encoder, W_decoder, b_decoder], where W_encoder, b_encoder belong to previous DenseLayer, W_decoder, b_decoder belong to this ReconLayer. In addition, if you want to update the parameters of previous 2 layers at the same time, simply modify [-4:] to [-6:].

ReconLayer.__init__(...):
    ...
    self.train_params = self.all_params[-4:]
    ...
  self.cost = mse + L1_a + L2_w

Basic layer

Layer

Input Layers

Input Layer

InputLayer

One-hot Input Layer

OneHotInputLayer

Word2Vec Embedding Layer

Word2vecEmbeddingInputlayer

Embedding Input layer

EmbeddingInputlayer

Average Embedding Input layer

AverageEmbeddingInputlayer

Activation Layers

PReLU Layer

PReluLayer

PReLU6 Layer

PRelu6Layer

PTReLU6 Layer

PTRelu6Layer

Convolutional Layers

Simplified Convolutions

For users don't familiar with TensorFlow, the following simplified functions may easier for you. We will provide more simplified functions later, but if you are good at TensorFlow, the professional APIs may better for you.

Conv1d

Conv2d

Simplified Deconvolutions

For users don't familiar with TensorFlow, the following simplified functions may easier for you. We will provide more simplified functions later, but if you are good at TensorFlow, the professional APIs may better for you.

DeConv2d

DeConv3d

Expert Convolutions

Conv1dLayer

Conv2dLayer

Conv3dLayer

Expert Deconvolutions

DeConv2dLayer

DeConv3dLayer

Atrous (De)Convolutions

AtrousConv1dLayer

AtrousConv2dLayer

AtrousDeConv2dLayer

Binary (De)Convolutions

BinaryConv2d

Deformable Convolutions

DeformableConv2d

Depthwise Convolutions

DepthwiseConv2d

DoReFa Convolutions

DorefaConv2d

Group Convolutions

GroupConv2d

Separable Convolutions

SeparableConv1d

SeparableConv2d

SubPixel Convolutions

SubpixelConv1d

SubpixelConv2d

Ternary Convolutions

TernaryConv2d

Dense layer

Binary Dense Layer

BinaryDenseLayer

Dense Layer

DenseLayer

DoReFa Dense Layer

DorefaDenseLayer

Drop Connect Dense Layer

DropconnectDenseLayer

Ternary Dense Layer

TernaryDenseLayer

Dropout Layers

DropoutLayer

Extend Layers

Expand Dims Layer

ExpandDimsLayer

Tile layer

TileLayer

External Libraries Layers

TF-Slim Layer

TF-Slim models can be connected into TensorLayer. All Google's Pre-trained model can be used easily , see Slim-model.

SlimNetsLayer

Keras Layer

Yes ! Keras models can be connected into TensorLayer! see tutorial_keras.py .

KerasLayer

Estimator Layer

EstimatorLayer

Flow Control Layer

MultiplexerLayer

Image Resampling Layers

2D UpSampling

UpSampling2dLayer

2D DownSampling

DownSampling2dLayer

Lambda Layers

Lambda Layer

LambdaLayer

ElementWise Lambda Layer

ElementwiseLambdaLayer

Merge Layers

Concat Layer

ConcatLayer

ElementWise Layer

ElementwiseLayer

Noise Layer

GaussianNoiseLayer

Normalization layer

For local response normalization as it does not have any weights and arguments, you can also apply tf.nn.lrn on network.outputs.

Batch Normalization

BatchNormLayer

Local Response Normalization

LocalResponseNormLayer

Instance Normalization

InstanceNormLayer

Layer Normalization

LayerNormLayer

Object Detection Layer

ROIPoolingLayer

Padding Layers

Pad Layer (Expert API)

Padding layer for any modes.

PadLayer

1D Zero padding

ZeroPad1d

2D Zero padding

ZeroPad2d

3D Zero padding

ZeroPad3d

Padding Layers

Pool Layer (Expert API)

Pooling layer for any dimensions and any pooling functions.

PoolLayer

1D Max pooling

MaxPool1d

1D Mean pooling

MeanPool1d

2D Max pooling

MaxPool2d

2D Mean pooling

MeanPool2d

3D Max pooling

MaxPool3d

3D Mean pooling

MeanPool3d

1D Global Max pooling

GlobalMaxPool1d

1D Global Mean pooling

GlobalMeanPool1d

2D Global Max pooling

GlobalMaxPool2d

2D Global Mean pooling

GlobalMeanPool2d

3D Global Max pooling

GlobalMaxPool3d

3D Global Mean pooling

GlobalMeanPool3d

Quantized Nets

This is an experimental API package for building Quantized Neural Networks. We are using matrix multiplication rather than add-minus and bit-count operation at the moment. Therefore, these APIs would not speed up the inferencing, for production, you can train model via TensorLayer and deploy the model into other customized C/C++ implementation (We probably provide users an extra C/C++ binary net framework that can load model from TensorLayer).

Note that, these experimental APIs can be changed in the future

Sign

SignLayer

Scale

ScaleLayer

Recurrent Layers

Fixed Length Recurrent layer

All recurrent layers can implement any type of RNN cell by feeding different cell function (LSTM, GRU etc).

RNN layer

RNNLayer

Bidirectional layer

BiRNNLayer

Recurrent Convolutional layer

Conv RNN Cell

ConvRNNCell

Basic Conv LSTM Cell

BasicConvLSTMCell

Conv LSTM layer

ConvLSTMLayer

Advanced Ops for Dynamic RNN

These operations usually be used inside Dynamic RNN layer, they can compute the sequence lengths for different situation and get the last RNN outputs by indexing.

Output indexing

advanced_indexing_op

Compute Sequence length 1

retrieve_seq_length_op

Compute Sequence length 2

retrieve_seq_length_op2

Compute Sequence length 3

retrieve_seq_length_op3

Get Mask

target_mask_op

Dynamic RNN layer

RNN layer

DynamicRNNLayer

Bidirectional layer

BiDynamicRNNLayer

Sequence to Sequence

Simple Seq2Seq

Seq2Seq

Shape layer

Flatten layer

FlattenLayer

Reshape layer

ReshapeLayer

Transpose layer

TransposeLayer

Spatial Transformer

2D Affine Transformation

SpatialTransformer2dAffineLayer

2D Affine Transformation function

transformer

Batch 2D Affine Transformation function

batch_transformer

Stack layer

StackLayer

Unstack layer

UnStackLayer

Time Distributed Layer