diff --git a/README.md b/README.md index 9d7b6378d..e0811f093 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ TensorLayer is a deep learning and reinforcement learning library on top of [Ten # News * [18 Jan] [《深度学习:一起玩转TensorLayer》](http://www.broadview.com.cn/book/5059) (Deep Learning using TensorLayer) -* [17 Dec] Experimentally release APIs for distributed training (by [TensorPort](https://tensorport.com)). See [tiny example](https://github.com/zsdonghao/tensorlayer/blob/master/example/tutorial_mnist_distributed.py). +* [17 Dec] Release experimental APIs for distributed training (by [TensorPort](https://tensorport.com)). See [tiny example](https://github.com/zsdonghao/tensorlayer/blob/master/example/tutorial_mnist_distributed.py). * [17 Nov] Release data augmentation APIs for object detection, see [tl.prepro](http://tensorlayer.readthedocs.io/en/latest/modules/prepro.html#object-detection). * [17 Nov] Support [Convolutional LSTM](https://arxiv.org/abs/1506.04214), see [ConvLSTMLayer](http://tensorlayer.readthedocs.io/en/latest/modules/layers.html#conv-lstm-layer). * [17 Nov] Support [Deformable Convolution](https://arxiv.org/abs/1703.06211), see [DeformableConv2dLayer](http://tensorlayer.readthedocs.io/en/latest/modules/layers.html#d-deformable-conv). @@ -41,13 +41,15 @@ TensorLayer is a deep learning and reinforcement learning library on top of [Ten # Installation -TensorLayer has package pre-requisites including TensorFlow, numpy, matplotlib and nltk (optional). For GPU support, CUDA and cuDNN are required. Please check [documentation](http://tensorlayer.readthedocs.io/en/latest/user/installation.html) for detailed instructions. The simplest way to install TensorLayer in your python program is: +TensorLayer has pre-requisites including TensorFlow, numpy, matplotlib and nltk (optional). For GPU support, CUDA and cuDNN are required. +The simplest way to install TensorLayer is: ```bash [for master version] pip install git+https://github.com/zsdonghao/tensorlayer.git (Recommended) [for stable version] pip install tensorlayer ``` +Please check [documentation](http://tensorlayer.readthedocs.io/en/latest/user/installation.html) for detailed instructions. # Examples and Tutorials @@ -121,18 +123,21 @@ TensorLayer provides two set of Convolutional layer APIs, see [(Professional)](h --> -# Library Features +# Features -As deep learning practitioners, we have been looking for a library that can serve for various development phases. This library shall be easy for beginners by providing rich neural network reference implementations. Later, it can be extended to address *real-world problems* by controlling training backends to exhibit low-level cognitive behaviours. In the end, it shall be able to serve in challenging *production environments*. TensorLayer is developed with the following goals: +## Design Philosophy -- *Simplicity* : TensorLayer lifts the low-level dataflow abstraction of TensorFlow to **high-level** deep learning modules. -A user often find it easy to bootstrap with TensorLayer, and then dive into low-level implementation only if need. +As deep learning practitioners, we have been looking for a library that can serve for various development phases. This library is easy for beginners by offering rich neural network +examples and tutorials. Later, its APIs can be extended to address *real-world problems* by allowing users to carefully manipulating TensorFlow. In the end, it is able to serve in challenging *production environments*. TensorLayer is a novel library that aims to satisfy development requirements throughout these phases. It has three key features: + +- *Simplicity* : TensorLayer lifts the low-level dataflow abstraction of TensorFlow to **high-level** deep learning modules. It also +provides users with massive examples and tutorials to help bootstrap. - *Transparency* : TensorLayer provides access to the **native APIs** of TensorFlow. This helps users achieve flexible controls within the training engine. -- *Performance* : TensorLayer provides **zero-cost** abstraction (see Benchmark below). It can run on distributed and heterogeneous TensorFlow platforms with full power. +- *Performance* : TensorLayer provides **zero-cost** abstraction. It can run on distributed and heterogeneous TensorFlow platforms with full power. -## Low Runtime Overhead +## Negligible Overhead -TensorLayer has negligible overhead. We show this by benchmarking classic deep learning +TensorLayer has negligible performance overhead. We benchmark classic deep learning models using TensorLayer and native TensorFlow implementations on a Titan X Pascal GPU. Here are the training speeds of respective tasks: @@ -141,14 +146,18 @@ on a Titan X Pascal GPU. Here are the training speeds of respective tasks: | TensorLayer | 2528 images/s | 18063 words/s | 58167 words/s | | TensorFlow | 2530 images/s | 18075 words/s | 58181 words/s | -## Comparing TensorLayer with Keras and TFLearn +## Compared with Keras and TFLearn -A frequent question regarding TensorLayer is what is the different with other libraries like Keras and Tflearn. -These libraries are comfortable to start with. They provide imperative abstractions to lower adoption barrier; -but in turn mask the underlying engine from users. Though good for bootstrap, -it becomes hard to leverage the advanced features of TensorFlow, which is quite necessary in tackling real-world problems. +Similar to TensorLayer, Keras and TFLearn are also popular TensorFlow wrapper libraries. +These libraries are comfortable to start with. They provide high-level abstractions; +but in turn mask the underlying engine features from users. Though good for bootstrap, +it becomes hard to manipulate the low-level powerful features of TensorFlow, which we find quite necessary +in tackling many real-world problems. Without compromise in simplicity, TensorLayer APIs are generally more flexible and transparent. -Users often find it easy to start with the examples and tutorials of TensorLayer, then dive into the TensorFlow low-level APIs only if need through the transparent API design. +Users often find it easy to start with the examples and tutorials of TensorLayer, and +then dive into the TensorFlow low-level APIs only if need through the transparent API design. +TensorLayer does not intend to create library lock-in. Users can easily import models from Keras, TFSlim and TFLearn into +a TensorLayer environment. # Documentation @@ -229,7 +238,7 @@ We provide many helper functions (like `fit()` , `test()`) that is similar to Ke # Academic and Industry Users -TensorLayer is in an active development stage and has received numerous contributions from an open community. +TensorLayer is in an active development stage and has an open community. It has been widely used by researchers from Imperial College London, Carnegie Mellon University, Stanford University, Tsinghua University, UCLA, Linköping University and etc., as well as engineers from Google, Microsoft, Alibaba, Tencent, Penguins Innovate, ReFULE4, Bloomberg, GoodAILab and many others. diff --git a/docs/modules/visualize.rst b/docs/modules/visualize.rst index 8eac5426d..269d7743c 100644 --- a/docs/modules/visualize.rst +++ b/docs/modules/visualize.rst @@ -17,7 +17,7 @@ to visualize the model, activations etc. Here we provide more functions for data frame images2d tsne_embedding - + draw_weights Save and read images ---------------------- @@ -65,3 +65,7 @@ Images by matplotlib Visualize embeddings -------------------- .. autofunction:: tsne_embedding + +Visualize weights +-------------------- +.. autofunction:: draw_weights diff --git a/example/tutorial_mnist.py b/example/tutorial_mnist.py index 413027bda..fb4c52ad9 100644 --- a/example/tutorial_mnist.py +++ b/example/tutorial_mnist.py @@ -144,7 +144,7 @@ def main_test_layers(model='relu'): print(" val acc: %f" % (val_acc / n_batch)) try: # You can visualize the weight of 1st hidden layer as follow. - tl.vis.W(network.all_params[0].eval(), second=10, saveable=True, shape=[28, 28], name='w1_' + str(epoch + 1), fig_idx=2012) + tl.vis.draw_weights(network.all_params[0].eval(), second=10, saveable=True, shape=[28, 28], name='w1_' + str(epoch + 1), fig_idx=2012) # You can also save the weight of 1st hidden layer to .npz file. # tl.files.save_npz([network.all_params[0]] , name='w1'+str(epoch+1)+'.npz') except: @@ -370,7 +370,7 @@ def main_test_stacked_denoise_AE(model='relu'): print(" val acc: %f" % (val_acc / n_batch)) try: # visualize the 1st hidden layer during fine-tune - tl.vis.W(network.all_params[0].eval(), second=10, saveable=True, shape=[28, 28], name='w1_' + str(epoch + 1), fig_idx=2012) + tl.vis.draw_weights(network.all_params[0].eval(), second=10, saveable=True, shape=[28, 28], name='w1_' + str(epoch + 1), fig_idx=2012) except: print("You should change vis.W(), if you want to save the feature images for different dataset") diff --git a/tensorlayer/cost.py b/tensorlayer/cost.py index e5a09b748..07421bb4b 100644 --- a/tensorlayer/cost.py +++ b/tensorlayer/cost.py @@ -513,7 +513,7 @@ def li_regularizer(scale, scope=None): logging.info('Scale of 0 disables regularizer.') return lambda _, name=None: None - def li(weights, name=None): + def li(weights): """Applies li regularization to weights.""" with tf.name_scope('li_regularizer') as scope: my_scale = ops.convert_to_tensor(scale, dtype=weights.dtype.base_dtype, name='scale') @@ -526,7 +526,7 @@ def li(weights, name=None): return li -def lo_regularizer(scale, scope=None): +def lo_regularizer(scale): """Lo regularization removes the neurons of current layer. The `o` represents `outputs` Returns a function that can be used to apply group lo regularization to weights. The implementation follows `TensorFlow contrib `__. @@ -535,8 +535,6 @@ def lo_regularizer(scale, scope=None): ---------- scale : float A scalar multiplier `Tensor`. 0.0 disables the regularizer. - scope: str - An optional scope name for this function. Returns ------- @@ -576,7 +574,7 @@ def lo(weights, name='lo_regularizer'): return lo -def maxnorm_regularizer(scale=1.0, scope=None): +def maxnorm_regularizer(scale=1.0): """Max-norm regularization returns a function that can be used to apply max-norm regularization to weights. More about max-norm, see `wiki-max norm `_. @@ -586,8 +584,6 @@ def maxnorm_regularizer(scale=1.0, scope=None): ---------- scale : float A scalar multiplier `Tensor`. 0.0 disables the regularizer. - scope: str - An optional scope name for this function. Returns --------- @@ -627,7 +623,7 @@ def mn(weights, name='max_regularizer'): return mn -def maxnorm_o_regularizer(scale, scope): +def maxnorm_o_regularizer(scale): """Max-norm output regularization removes the neurons of current layer. Returns a function that can be used to apply max-norm regularization to each column of weight matrix. The implementation follows `TensorFlow contrib `__. @@ -636,8 +632,6 @@ def maxnorm_o_regularizer(scale, scope): ---------- scale : float A scalar multiplier `Tensor`. 0.0 disables the regularizer. - scope: str - An optional scope name for this function. Returns --------- @@ -677,7 +671,7 @@ def mn_o(weights, name='maxnorm_o_regularizer'): return mn_o -def maxnorm_i_regularizer(scale, scope=None): +def maxnorm_i_regularizer(scale): """Max-norm input regularization removes the neurons of previous layer. Returns a function that can be used to apply max-norm regularization to each row of weight matrix. The implementation follows `TensorFlow contrib `__. @@ -686,8 +680,6 @@ def maxnorm_i_regularizer(scale, scope=None): ---------- scale : float A scalar multiplier `Tensor`. 0.0 disables the regularizer. - scope: str - An optional scope name for this function. Returns --------- @@ -725,6 +717,3 @@ def mn_i(weights, name='maxnorm_i_regularizer'): return standard_ops_fn(my_scale, standard_ops.reduce_sum(standard_ops.reduce_max(standard_ops.abs(weights), 1)), name=scope) return mn_i - - -# diff --git a/tensorlayer/db.py b/tensorlayer/db.py index 98eef0be5..15ff35b1a 100644 --- a/tensorlayer/db.py +++ b/tensorlayer/db.py @@ -418,119 +418,3 @@ def __str__(self): _s = "[TensorDB] Info:\n" _t = _s + " " + str(self.db) return _t - - # def save_bulk_data(self, data=None, filename='filename'): - # """ Put bulk data into TensorDB.datafs, return file ID. - # When you have a very large data, you may like to save it into GridFS Buckets - # instead of Collections, then when you want to load it, XXXX - # - # Parameters - # ----------- - # data : serialized data. - # filename : string, GridFS Buckets. - # - # References - # ----------- - # - MongoDB find, xxxxx - # """ - # s = time.time() - # f_id = self.datafs.put(data, filename=filename) - # print("[TensorDB] save_bulk_data: {} took: {}s".format(filename, round(time.time()-s, 2))) - # return f_id - # - # def save_collection(self, data=None, collect_name='collect_name'): - # """ Insert data into MongoDB Collections, return xx. - # - # Parameters - # ----------- - # data : serialized data. - # collect_name : string, MongoDB collection name. - # - # References - # ----------- - # - MongoDB find, xxxxx - # """ - # s = time.time() - # rl = self.db[collect_name].insert_many(data) - # print("[TensorDB] save_collection: {} took: {}s".format(collect_name, round(time.time()-s, 2))) - # return rl - # - # def find(self, args={}, collect_name='collect_name'): - # """ Find data from MongoDB Collections. - # - # Parameters - # ----------- - # args : dictionary, arguments for finding. - # collect_name : string, MongoDB collection name. - # - # References - # ----------- - # - MongoDB find, xxxxx - # """ - # s = time.time() - # - # pc = self.db[collect_name].find(args) # pymongo.cursor.Cursor object - # flist = pc.distinct('f_id') - # fldict = {} - # for f in flist: # you may have multiple Buckets files - # # fldict[f] = pickle.loads(self.datafs.get(f).read()) - # # s2 = time.time() - # tmp = self.datafs.get(f).read() - # # print(time.time()-s2) - # fldict[f] = pickle.loads(tmp) - # # print(time.time()-s2) - # # exit() - # # print(round(time.time()-s, 2)) - # data = [fldict[x['f_id']][x['id']] for x in pc] - # data = np.asarray(data) - # print("[TensorDB] find: {} get: {} took: {}s".format(collect_name, pc.count(), round(time.time()-s, 2))) - # return data - - -class DBLogger: - """ """ - - def __init__(self, db, model): - self.db = db - self.model = model - - def on_train_begin(self, logs={}): - print("start") - - def on_train_end(self, logs={}): - print("end") - - def on_epoch_begin(self, epoch, logs={}): - self.epoch = epoch - self.et = time.time() - return - - def on_epoch_end(self, epoch, logs={}): - self.et = time.time() - self.et - print("ending") - print(epoch) - logs['epoch'] = epoch - logs['time'] = datetime.utcnow() - logs['stepTime'] = self.et - logs['acc'] = np.asscalar(logs['acc']) - print(logs) - - w = self.model.Params - fid = self.db.save_params(w, logs) - logs.update({'params': fid}) - self.db.valid_log(logs) - - def on_batch_begin(self, batch, logs={}): - self.t = time.time() - self.losses = [] - self.batch = batch - - def on_batch_end(self, batch, logs={}): - self.t2 = time.time() - self.t - logs['acc'] = np.asscalar(logs['acc']) - #logs['loss']=np.asscalar(logs['loss']) - logs['step_time'] = self.t2 - logs['time'] = datetime.utcnow() - logs['epoch'] = self.epoch - logs['batch'] = self.batch - self.db.train_log(logs) diff --git a/tensorlayer/files.py b/tensorlayer/files.py index d1eb14749..1dabaa8fd 100644 --- a/tensorlayer/files.py +++ b/tensorlayer/files.py @@ -117,7 +117,7 @@ def load_mnist_labels(path, filename): return X_train, y_train, X_val, y_val, X_test, y_test -def load_cifar10_dataset(shape=(-1, 32, 32, 3), path='data', plotable=False, second=3): +def load_cifar10_dataset(shape=(-1, 32, 32, 3), path='data', plotable=False): """Load CIFAR-10 dataset. It consists of 60000 32x32 colour images in 10 classes, with @@ -137,8 +137,6 @@ def load_cifar10_dataset(shape=(-1, 32, 32, 3), path='data', plotable=False, sec The path that the data is downloaded to, defaults is ``data/cifar10/``. plotable : boolean Whether to plot some image examples, False as default. - second : int - If ``plotable`` is True, it is the display time. Examples -------- @@ -1791,4 +1789,4 @@ def npz_to_W_pdf(path=None, regx='w1pre_[0-9]+\.(npz)'): for f in file_list: W = load_npz(path, f)[0] logging.info("%s --> %s" % (f, f.split('.')[0] + '.pdf')) - visualize.W(W, second=10, saveable=True, name=f.split('.')[0], fig_idx=2012) + visualize.draw_weights(W, second=10, saveable=True, name=f.split('.')[0], fig_idx=2012) diff --git a/tensorlayer/iterate.py b/tensorlayer/iterate.py index d7db2a0a0..5fd68643e 100644 --- a/tensorlayer/iterate.py +++ b/tensorlayer/iterate.py @@ -269,156 +269,3 @@ def ptb_iterator(raw_data, batch_size, num_steps): x = data[:, i * num_steps:(i + 1) * num_steps] y = data[:, i * num_steps + 1:(i + 1) * num_steps + 1] yield (x, y) - - -# def minibatches_for_sequence2D(inputs, targets, batch_size, sequence_length, stride=1): -# """ -# Input a group of example in 2D numpy.array and their labels. -# Return the examples and labels by the given batchsize, sequence_length. -# Use for RNN. -# -# Parameters -# ---------- -# inputs : numpy.array -# (X) The input features, every row is a example. -# targets : numpy.array -# (y) The labels of inputs, every row is a example. -# batchsize : int -# The batch size must be a multiple of sequence_length: int(batch_size % sequence_length) == 0 -# sequence_length : int -# The sequence length -# stride : int -# The stride step -# -# Examples -# -------- -# >>> sequence_length = 2 -# >>> batch_size = 4 -# >>> stride = 1 -# >>> X_train = np.asarray([[1,2,3],[4,5,6],[7,8,9],[10,11,12],[13,14,15],[16,17,18],[19,20,21],[22,23,24]]) -# >>> y_train = np.asarray(['0','1','2','3','4','5','6','7']) -# >>> print('X_train = %s' % X_train) -# >>> print('y_train = %s' % y_train) -# >>> for batch in minibatches_for_sequence2D(X_train, y_train, batch_size=batch_size, sequence_length=sequence_length, stride=stride): -# >>> inputs, targets = batch -# >>> print(inputs) -# >>> print(targets) -# ... [[ 1. 2. 3.] -# ... [ 4. 5. 6.] -# ... [ 4. 5. 6.] -# ... [ 7. 8. 9.]] -# ... [1 2] -# ... [[ 4. 5. 6.] -# ... [ 7. 8. 9.] -# ... [ 7. 8. 9.] -# ... [ 10. 11. 12.]] -# ... [2 3] -# ... ... -# ... [[ 16. 17. 18.] -# ... [ 19. 20. 21.] -# ... [ 19. 20. 21.] -# ... [ 22. 23. 24.]] -# ... [6 7] -# """ -# print('len(targets)=%d batch_size=%d sequence_length=%d stride=%d' % (len(targets), batch_size, sequence_length, stride)) -# assert len(inputs) == len(targets), '1 feature vector have 1 target vector/value' #* sequence_length -# # assert int(batch_size % sequence_length) == 0, 'batch_size % sequence_length must == 0\ -# # batch_size is number of examples rather than number of targets' -# -# # print(inputs.shape, len(inputs), len(inputs[0])) -# -# n_targets = int(batch_size/sequence_length) -# # n_targets = int(np.ceil(batch_size/sequence_length)) -# X = np.empty(shape=(0,len(inputs[0])), dtype=np.float32) -# y = np.zeros(shape=(1, n_targets), dtype=np.int32) -# -# for idx in range(sequence_length, len(inputs), stride): # go through all example during 1 epoch -# for n in range(n_targets): # for num of target -# X = np.concatenate((X, inputs[idx-sequence_length+n:idx+n])) -# y[0][n] = targets[idx-1+n] -# # y = np.vstack((y, targets[idx-1+n])) -# yield X, y[0] -# X = np.empty(shape=(0,len(inputs[0]))) -# # y = np.empty(shape=(1,0)) -# -# -# def minibatches_for_sequence4D(inputs, targets, batch_size, sequence_length, stride=1): # -# """ -# Input a group of example in 4D numpy.array and their labels. -# Return the examples and labels by the given batchsize, sequence_length. -# Use for RNN. -# -# Parameters -# ---------- -# inputs : numpy.array -# (X) The input features, every row is a example. -# targets : numpy.array -# (y) The labels of inputs, every row is a example. -# batchsize : int -# The batch size must be a multiple of sequence_length: int(batch_size % sequence_length) == 0 -# sequence_length : int -# The sequence length -# stride : int -# The stride step -# -# Examples -# -------- -# >>> sequence_length = 2 -# >>> batch_size = 2 -# >>> stride = 1 -# >>> X_train = np.asarray([[1,2,3],[4,5,6],[7,8,9],[10,11,12],[13,14,15],[16,17,18],[19,20,21],[22,23,24]]) -# >>> y_train = np.asarray(['0','1','2','3','4','5','6','7']) -# >>> X_train = np.expand_dims(X_train, axis=1) -# >>> X_train = np.expand_dims(X_train, axis=3) -# >>> for batch in minibatches_for_sequence4D(X_train, y_train, batch_size=batch_size, sequence_length=sequence_length, stride=stride): -# >>> inputs, targets = batch -# >>> print(inputs) -# >>> print(targets) -# ... [[[[ 1.] -# ... [ 2.] -# ... [ 3.]]] -# ... [[[ 4.] -# ... [ 5.] -# ... [ 6.]]]] -# ... [1] -# ... [[[[ 4.] -# ... [ 5.] -# ... [ 6.]]] -# ... [[[ 7.] -# ... [ 8.] -# ... [ 9.]]]] -# ... [2] -# ... ... -# ... [[[[ 19.] -# ... [ 20.] -# ... [ 21.]]] -# ... [[[ 22.] -# ... [ 23.] -# ... [ 24.]]]] -# ... [7] -# """ -# print('len(targets)=%d batch_size=%d sequence_length=%d stride=%d' % (len(targets), batch_size, sequence_length, stride)) -# assert len(inputs) == len(targets), '1 feature vector have 1 target vector/value' #* sequence_length -# # assert int(batch_size % sequence_length) == 0, 'in LSTM, batch_size % sequence_length must == 0\ -# # batch_size is number of X_train rather than number of targets' -# assert stride >= 1, 'stride must be >=1, at least move 1 step for each iternation' -# -# n_example, n_channels, width, height = inputs.shape -# print('n_example=%d n_channels=%d width=%d height=%d' % (n_example, n_channels, width, height)) -# -# n_targets = int(np.ceil(batch_size/sequence_length)) # 实际为 batchsize/sequence_length + 1 -# print(n_targets) -# X = np.zeros(shape=(batch_size, n_channels, width, height), dtype=np.float32) -# # X = np.zeros(shape=(n_targets, sequence_length, n_channels, width, height), dtype=np.float32) -# y = np.zeros(shape=(1,n_targets), dtype=np.int32) -# # y = np.empty(shape=(0,1), dtype=np.float32) -# # time.sleep(2) -# for idx in range(sequence_length, n_example-n_targets+2, stride): # go through all example during 1 epoch -# for n in range(n_targets): # for num of target -# # print(idx+n, inputs[idx-sequence_length+n : idx+n].shape) -# X[n*sequence_length : (n+1)*sequence_length] = inputs[idx+n-sequence_length : idx+n] -# # X[n] = inputs[idx-sequence_length+n:idx+n] -# y[0][n] = targets[idx+n-1] -# # y = np.vstack((y, targets[idx-1+n])) -# # y = targets[idx: idx+n_targets] -# yield X, y[0] diff --git a/tensorlayer/layers/convolution.py b/tensorlayer/layers/convolution.py index 8e6a46709..37b1b4bef 100644 --- a/tensorlayer/layers/convolution.py +++ b/tensorlayer/layers/convolution.py @@ -46,7 +46,6 @@ def __init__( stride=1, dilation_rate=1, padding='SAME', - # use_cudnn_on_gpu=None, data_format='NWC', W_init=tf.truncated_normal_initializer(stddev=0.02), b_init=tf.constant_initializer(value=0.0), @@ -1165,7 +1164,6 @@ def conv1d( dilation_rate=1, act=tf.identity, padding='SAME', - # use_cudnn_on_gpu=None, data_format="NWC", W_init=tf.truncated_normal_initializer(stddev=0.02), b_init=tf.constant_initializer(value=0.0), diff --git a/tensorlayer/layers/core.py b/tensorlayer/layers/core.py index a71e54262..1b7b453b3 100644 --- a/tensorlayer/layers/core.py +++ b/tensorlayer/layers/core.py @@ -1063,7 +1063,8 @@ def pretrain(self, sess, x, X_train, X_val, denoise_name=None, n_epoch=100, batc logging.info(" val loss: %f" % (val_loss / n_batch)) if save: try: - visualize.W(self.train_params[0].eval(), second=10, saveable=True, shape=[28, 28], name=save_name + str(epoch + 1), fig_idx=2012) + visualize.draw_weights( + self.train_params[0].eval(), second=10, saveable=True, shape=[28, 28], name=save_name + str(epoch + 1), fig_idx=2012) files.save_npz([self.all_params[0]], name=save_name + str(epoch + 1) + '.npz') except: raise Exception( diff --git a/tensorlayer/layers/recurrent.py b/tensorlayer/layers/recurrent.py index 5f8ae2f0d..0b0ca7157 100644 --- a/tensorlayer/layers/recurrent.py +++ b/tensorlayer/layers/recurrent.py @@ -135,7 +135,6 @@ def __init__( n_steps=5, initial_state=None, return_last=False, - # is_reshape = True, return_seq_2d=False, name='rnn_layer', ): @@ -474,11 +473,10 @@ def output_size(self): """Integer or TensorShape: size of outputs produced by this cell.""" raise NotImplementedError("Abstract method") - def zero_state(self, batch_size, dtype): + def zero_state(self, batch_size): """Return zero-filled state tensor(s). Args: batch_size: int, float, or unit Tensor representing the batch size. - dtype: the data type to use for the state. Returns: tensor of shape '[batch_size x shape[0] x shape[1] x num_features] filled with zeros @@ -900,7 +898,6 @@ def target_mask_op(data, pad_val=0): # HangSheng: return tensor for mask,if inp raise ValueError("target_mask_op: handling data_shape_size %s hasn't been implemented!" % (data_shape_size)) -# Dynamic RNN class DynamicRNNLayer(Layer): """ The :class:`DynamicRNNLayer` class is a dynamic recurrent layer, see ``tf.nn.dynamic_rnn``. @@ -1531,7 +1528,6 @@ def __init__( initial_state_decode=None, dropout=None, n_layer=1, - # return_last = False, return_seq_2d=False, name='seq2seq', ): @@ -1554,6 +1550,7 @@ def __init__( cell_fn=cell_fn, cell_init_args=cell_init_args, n_hidden=n_hidden, + initializer=initializer, initial_state=initial_state_encode, dropout=dropout, n_layer=n_layer, @@ -1568,6 +1565,7 @@ def __init__( cell_fn=cell_fn, cell_init_args=cell_init_args, n_hidden=n_hidden, + initializer=initializer, initial_state=(network_encode.final_state if initial_state_decode is None else initial_state_decode), dropout=dropout, n_layer=n_layer, diff --git a/tensorlayer/nlp.py b/tensorlayer/nlp.py index 79faaa76e..2ddfc7bb5 100755 --- a/tensorlayer/nlp.py +++ b/tensorlayer/nlp.py @@ -442,24 +442,25 @@ def simple_read_words(filename="nietzsche.txt"): def read_words(filename="nietzsche.txt", replace=['\n', '']): - """File to list format context. Note that, this script can not handle punctuations. + """Read list format context from a file. + For customized read_words method, see ``tutorial_generate_text.py``. Parameters - ----------- + ---------- filename : str - A file path (like .txt file) + a file path. replace : list of str - Replace original string by target string, e.g. `[original string, target string]`. To disable replace use `['', '']`. + replace original string by target string. Returns - -------- + ------- list of str - The context in a list, split by space by default, and use ```` to represent ``\n``, e.g. ``[... 'how', 'useful', 'it', "'s" ... ]``. + The context in a list (split using space). - References - --------------- - - `tensorflow.models.rnn.ptb.reader `_ + See Also + -------- + - `tensorflow.models.rnn.ptb.reader `__ """ with tf.gfile.GFile(filename, "r") as f: @@ -477,7 +478,7 @@ def read_analogies_file(eval_file='questions-words.txt', word2id={}): Parameters ---------- - eval_data : str + eval_file : str The file name. word2id : dictionary a dictionary that maps word to ID. @@ -485,7 +486,7 @@ def read_analogies_file(eval_file='questions-words.txt', word2id={}): Returns -------- numpy.array - A `[n_examples, 4]` numpy array containing the analogy question's word IDs. + A ``[n_examples, 4]`` numpy array containing the analogy question's word IDs. Examples --------- diff --git a/tensorlayer/prepro.py b/tensorlayer/prepro.py index b7aeb2385..a9ef4bd8d 100644 --- a/tensorlayer/prepro.py +++ b/tensorlayer/prepro.py @@ -102,23 +102,17 @@ def threading_data(data=None, fn=None, thread_count=None, **kwargs): """ - ## plot function info - # for name, value in kwargs.items(): - # logging.info('{0} = {1}'.format(name, value)) - # exit() - # define function for threading def apply_fn(results, i, data, kwargs): results[i] = fn(data, **kwargs) - ## start multi-threaded reading. - if thread_count is None: # by Milo - results = [None] * len(data) ## preallocate result list + if thread_count is None: + results = [None] * len(data) threads = [] for i in range(len(data)): t = threading.Thread(name='threading_and_return', target=apply_fn, args=(results, i, data[i], kwargs)) t.start() threads.append(t) - else: # by geometrikal + else: divs = np.linspace(0, len(data), thread_count + 1) divs = np.round(divs).astype(int) results = [None] * thread_count @@ -128,20 +122,18 @@ def apply_fn(results, i, data, kwargs): t.start() threads.append(t) - ## wait for all threads to complete for t in threads: t.join() if thread_count is None: try: return np.asarray(results) - except: # if dim don't match + except: return results else: return np.concatenate(results) -## Image def rotation(x, rg=20, is_random=False, row_index=0, col_index=1, channel_index=2, fill_mode='nearest', cval=0., order=1): """Rotate an image randomly or non-randomly. @@ -225,7 +217,7 @@ def rotation_multi(x, rg=20, is_random=False, row_index=0, col_index=1, channel_ # crop -def crop(x, wrg, hrg, is_random=False, row_index=0, col_index=1, channel_index=2): +def crop(x, wrg, hrg, is_random=False, row_index=0, col_index=1): """Randomly or centrally crop an image. Parameters @@ -238,13 +230,16 @@ def crop(x, wrg, hrg, is_random=False, row_index=0, col_index=1, channel_index=2 Size of height. is_random : boolean, If True, randomly crop, else central crop. Default is False. - row_index col_index and channel_index : int - Index of row, col and channel, default (0, 1, 2), for theano (1, 2, 0). + row_index: int + index of row. + col_index: int + index of column. Returns ------- numpy.array A processed image. + """ h, w = x.shape[row_index], x.shape[col_index] assert (h > hrg) and (w > wrg), "The size of cropping should smaller than the original image" @@ -267,7 +262,7 @@ def crop(x, wrg, hrg, is_random=False, row_index=0, col_index=1, channel_index=2 # central crop -def crop_multi(x, wrg, hrg, is_random=False, row_index=0, col_index=1, channel_index=2): +def crop_multi(x, wrg, hrg, is_random=False, row_index=0, col_index=1): """Randomly or centrally crop multiple images. Parameters @@ -281,6 +276,7 @@ def crop_multi(x, wrg, hrg, is_random=False, row_index=0, col_index=1, channel_i ------- numpy.array A list of processed images. + """ h, w = x[0].shape[row_index], x[0].shape[col_index] assert (h > hrg) and (w > wrg), "The size of cropping should smaller than the original image" @@ -321,6 +317,7 @@ def flip_axis(x, axis=1, is_random=False): ------- numpy.array A processed image. + """ if is_random: factor = np.random.uniform(-1, 1) @@ -352,6 +349,7 @@ def flip_axis_multi(x, axis, is_random=False): ------- numpy.array A list of processed images. + """ if is_random: factor = np.random.uniform(-1, 1) @@ -412,6 +410,7 @@ def shift(x, wrg=0.1, hrg=0.1, is_random=False, row_index=0, col_index=1, channe ------- numpy.array A processed image. + """ h, w = x.shape[row_index], x.shape[col_index] if is_random: @@ -441,6 +440,7 @@ def shift_multi(x, wrg=0.1, hrg=0.1, is_random=False, row_index=0, col_index=1, ------- numpy.array A list of processed images. + """ h, w = x[0].shape[row_index], x[0].shape[col_index] if is_random: @@ -518,6 +518,7 @@ def shear_multi(x, intensity=0.1, is_random=False, row_index=0, col_index=1, cha ------- numpy.array A list of processed images. + """ if is_random: shear = np.random.uniform(-intensity, intensity) @@ -593,6 +594,7 @@ def shear_multi2(x, shear=(0.1, 0.1), is_random=False, row_index=0, col_index=1, ------- numpy.array A list of processed images. + """ assert len(shear) == 2, "shear should be tuple of 2 floats, or you want to use tl.prepro.shear_multi rather than tl.prepro.shear_multi2 ?" if is_random: @@ -724,6 +726,7 @@ def swirl_multi(x, ------- numpy.array A list of processed images. + """ assert radius != 0, Exception("Invalid radius value") rotation = np.pi / 180 * rotation @@ -834,6 +837,7 @@ def elastic_transform_multi(x, alpha, sigma, mode="constant", cval=0, is_random= ------- numpy.array A list of processed images. + """ if is_random is False: random_state = np.random.RandomState(None) @@ -897,6 +901,7 @@ def zoom(x, zoom_range=(0.9, 1.1), is_random=False, row_index=0, col_index=1, ch ------- numpy.array A processed image. + """ if len(zoom_range) != 2: raise Exception('zoom_range should be a tuple or list of two floats. ' 'Received arg: ', zoom_range) @@ -932,6 +937,7 @@ def zoom_multi(x, zoom_range=(0.9, 1.1), is_random=False, row_index=0, col_index ------- numpy.array A list of processed images. + """ if len(zoom_range) != 2: raise Exception('zoom_range should be a tuple or list of two floats. ' 'Received arg: ', zoom_range) @@ -1011,6 +1017,7 @@ def brightness_multi(x, gamma=1, gain=1, is_random=False): ------- numpy.array A list of processed images. + """ if is_random: gamma = np.random.uniform(1 - gamma, 1 + gamma) @@ -1109,6 +1116,7 @@ def rgb_to_hsv(rgb): ------- numpy.array A processed image. + """ # Translated from source of colorsys.rgb_to_hsv # r,g,b should be a numpy arrays with values between 0 and 255 @@ -1517,6 +1525,7 @@ def channel_shift_multi(x, intensity, is_random=False, channel_index=2): ------- numpy.array A list of processed images. + """ if is_random: factor = np.random.uniform(-intensity, intensity) @@ -1549,6 +1558,7 @@ def drop(x, keep=0.5): ------- numpy.array A processed image. + """ if len(x.shape) == 3: if x.shape[-1] == 3: # color @@ -1743,7 +1753,7 @@ def array_to_img(x, dim_ordering=(0, 1, 2), scale=True): References ----------- - - `PIL Image.fromarray `__ + `PIL Image.fromarray `__ """ from PIL import Image @@ -1789,8 +1799,9 @@ def find_contours(x, level=0.8, fully_connected='low', positive_orientation='low -------- list of (n,2)-ndarrays Each contour is an ndarray of shape (n, 2), consisting of n (row, column) coordinates along the contour. + """ - return skimage.measure.find_contours(x, level, fully_connected='low', positive_orientation='low') + return skimage.measure.find_contours(x, level, fully_connected=fully_connected, positive_orientation=positive_orientation) def pt2map(list_points=[], size=(100, 100), val=1): @@ -1809,9 +1820,10 @@ def pt2map(list_points=[], size=(100, 100), val=1): ------- numpy.array An image. + """ i_m = np.zeros(size) - if list_points == []: + if len(list_points) == 0: return i_m for xx in list_points: for x in xx: @@ -1835,6 +1847,7 @@ def binary_dilation(x, radius=3): ------- numpy.array A processed binary image. + """ from skimage.morphology import disk, binary_dilation mask = disk(radius) @@ -1857,6 +1870,7 @@ def dilation(x, radius=3): ------- numpy.array A processed greyscale image. + """ from skimage.morphology import disk, dilation mask = disk(radius) @@ -1879,6 +1893,7 @@ def binary_erosion(x, radius=3): ------- numpy.array A processed binary image. + """ from skimage.morphology import disk, dilation, binary_erosion mask = disk(radius) @@ -1901,6 +1916,7 @@ def erosion(x, radius=3): ------- numpy.array A processed greyscale image. + """ from skimage.morphology import disk, dilation, erosion mask = disk(radius) @@ -1940,6 +1956,7 @@ def obj_box_coords_rescale(coords=[], shape=[100, 200]): ------- list of 4 numbers New coordinates. + """ imh, imw = shape[0], shape[1] imh = imh * 1.0 # * 1.0 for python2 : force division to be float point @@ -2119,6 +2136,7 @@ def obj_box_coord_upleft_to_centroid(coord): ------- list of 4 numbers New bounding box. + """ assert len(coord) == 4, "coordinate should be 4 values : [x, y, w, h]" x, y, w, h = coord @@ -2139,6 +2157,7 @@ def parse_darknet_ann_str_to_list(annotations): ------- list of list of 4 numbers List of bounding box. + """ annotations = annotations.split("\n") ann = [] @@ -2171,6 +2190,7 @@ def parse_darknet_ann_list_to_cls_box(annotations): list of list of 4 numbers List of bounding box. + """ class_list = [] bbox_list = [] @@ -2379,6 +2399,7 @@ def obj_box_crop(im, classes=[], coords=[], wrg=100, hrg=100, is_rescale=False, A list of classes list of list of 4 numbers A list of new bounding boxes. + """ h, w = im.shape[0], im.shape[1] assert (h > hrg) and (w > wrg), "The size of cropping should smaller than the original image" @@ -2531,6 +2552,7 @@ def obj_box_shift(im, A list of classes list of list of 4 numbers A list of new bounding boxes. + """ imh, imw = im.shape[row_index], im.shape[col_index] assert (hrg < 1.0) and (hrg > 0.) and (wrg < 1.0) and (wrg > 0.), "shift range should be (0, 1)" @@ -2663,6 +2685,7 @@ def obj_box_zoom(im, A list of classes list of list of 4 numbers A list of new bounding boxes. + """ if len(zoom_range) != 2: raise Exception('zoom_range should be a tuple or list of two floats. ' 'Received arg: ', zoom_range) @@ -3069,170 +3092,3 @@ def sequences_get_mask(sequences, pad_val=0): else: break # <-- exit the for loop, prepcess next sequence return mask - - -## Text -# see tensorlayer.nlp - -## Tensor Opt -# def distorted_images(images=None, height=24, width=24): -# """Distort images for generating more training data. -# -# Features -# --------- -# They are cropped to height * width pixels randomly. -# -# They are approximately whitened to make the model insensitive to dynamic range. -# -# Randomly flip the image from left to right. -# -# Randomly distort the image brightness. -# -# Randomly distort the image contrast. -# -# Whiten (Normalize) the images. -# -# Parameters -# ---------- -# images : 4D Tensor -# The tensor or placeholder of images -# height : int -# The height for random crop. -# width : int -# The width for random crop. -# -# Returns -# ------- -# result : tuple of Tensor -# (Tensor for distorted images, Tensor for while loop index) -# -# Examples -# -------- -# >>> X_train, y_train, X_test, y_test = tl.files.load_cifar10_dataset(shape=(-1, 32, 32, 3), plotable=False) -# >>> sess = tf.InteractiveSession() -# >>> batch_size = 128 -# >>> x = tf.placeholder(tf.float32, shape=[batch_size, 32, 32, 3]) -# >>> distorted_images_op = tl.preprocess.distorted_images(images=x, height=24, width=24) -# >>> sess.run(tf.initialize_all_variables()) -# >>> feed_dict={x: X_train[0:batch_size,:,:,:]} -# >>> distorted_images, idx = sess.run(distorted_images_op, feed_dict=feed_dict) -# >>> tl.vis.images2d(X_train[0:9,:,:,:], second=2, saveable=False, name='cifar10', dtype=np.uint8, fig_idx=20212) -# >>> tl.vis.images2d(distorted_images[1:10,:,:,:], second=10, saveable=False, name='distorted_images', dtype=None, fig_idx=23012) -# -# Notes -# ------ -# - The first image in 'distorted_images' should be removed. -# -# References -# ----------- -# - `tensorflow.models.image.cifar10.cifar10_input `__ -# """ -# logging.info("This function is deprecated, please use tf.map_fn instead, e.g:\n \ -# t_image = tf.map_fn(lambda img: tf.image.random_brightness(img, max_delta=32. / 255.), t_image)\n \ -# t_image = tf.map_fn(lambda img: tf.image.random_contrast(img, lower=0.5, upper=1.5), t_image)\n \ -# t_image = tf.map_fn(lambda img: tf.image.random_saturation(img, lower=0.5, upper=1.5), t_image)\n \ -# t_image = tf.map_fn(lambda img: tf.image.random_hue(img, max_delta=0.032), t_image)") -# exit() -# # logging.info(" [Warning] distorted_images will be deprecated due to speed, see TFRecord tutorial for more info...") -# try: -# batch_size = int(images._shape[0]) -# except: -# raise Exception('unknow batch_size of images') -# distorted_x = tf.Variable(tf.constant(0.1, shape=[1, height, width, 3])) -# i = tf.Variable(tf.constant(0)) -# -# c = lambda distorted_x, i: tf.less(i, batch_size) -# -# def body(distorted_x, i): -# # 1. Randomly crop a [height, width] section of the image. -# image = tf.random_crop(tf.gather(images, i), [height, width, 3]) -# # 2. Randomly flip the image horizontally. -# image = tf.image.random_flip_left_right(image) -# # 3. Randomly change brightness. -# image = tf.image.random_brightness(image, max_delta=63) -# # 4. Randomly change contrast. -# image = tf.image.random_contrast(image, lower=0.2, upper=1.8) -# # 5. Subtract off the mean and divide by the variance of the pixels. -# image = tf.image.per_image_whitening(image) -# # 6. Append the image to a batch. -# image = tf.expand_dims(image, 0) -# return tf.concat(0, [distorted_x, image]), tf.add(i, 1) -# -# result = tf.while_loop(cond=c, body=body, loop_vars=(distorted_x, i), parallel_iterations=16) -# return result -# -# -# def crop_central_whiten_images(images=None, height=24, width=24): -# """Crop the central of image, and normailize it for test data. -# -# They are cropped to central of height * width pixels. -# -# Whiten (Normalize) the images. -# -# Parameters -# ---------- -# images : 4D Tensor -# The tensor or placeholder of images -# height : int -# The height for central crop. -# width : int -# The width for central crop. -# -# Returns -# ------- -# result : tuple Tensor -# (Tensor for distorted images, Tensor for while loop index) -# -# Examples -# -------- -# >>> X_train, y_train, X_test, y_test = tl.files.load_cifar10_dataset(shape=(-1, 32, 32, 3), plotable=False) -# >>> sess = tf.InteractiveSession() -# >>> batch_size = 128 -# >>> x = tf.placeholder(tf.float32, shape=[batch_size, 32, 32, 3]) -# >>> central_images_op = tl.preprocess.crop_central_whiten_images(images=x, height=24, width=24) -# >>> sess.run(tf.initialize_all_variables()) -# >>> feed_dict={x: X_train[0:batch_size,:,:,:]} -# >>> central_images, idx = sess.run(central_images_op, feed_dict=feed_dict) -# >>> tl.vis.images2d(X_train[0:9,:,:,:], second=2, saveable=False, name='cifar10', dtype=np.uint8, fig_idx=20212) -# >>> tl.vis.images2d(central_images[1:10,:,:,:], second=10, saveable=False, name='central_images', dtype=None, fig_idx=23012) -# -# Notes -# ------ -# The first image in 'central_images' should be removed. -# -# Code References -# ---------------- -# - ``tensorflow.models.image.cifar10.cifar10_input`` -# """ -# logging.info("This function is deprecated, please use tf.map_fn instead, e.g:\n \ -# t_image = tf.map_fn(lambda img: tf.image.random_brightness(img, max_delta=32. / 255.), t_image)\n \ -# t_image = tf.map_fn(lambda img: tf.image.random_contrast(img, lower=0.5, upper=1.5), t_image)\n \ -# t_image = tf.map_fn(lambda img: tf.image.random_saturation(img, lower=0.5, upper=1.5), t_image)\n \ -# t_image = tf.map_fn(lambda img: tf.image.random_hue(img, max_delta=0.032), t_image)") -# exit() -# # logging.info(" [Warning] crop_central_whiten_images will be deprecated due to speed, see TFRecord tutorial for more info...") -# try: -# batch_size = int(images._shape[0]) -# except: -# raise Exception('unknow batch_size of images') -# central_x = tf.Variable(tf.constant(0.1, shape=[1, height, width, 3])) -# i = tf.Variable(tf.constant(0)) -# -# c = lambda central_x, i: tf.less(i, batch_size) -# -# def body(central_x, i): -# # 1. Crop the central [height, width] of the image. -# image = tf.image.resize_image_with_crop_or_pad(tf.gather(images, i), height, width) -# # 2. Subtract off the mean and divide by the variance of the pixels. -# image = tf.image.per_image_whitening(image) -# # 5. Append the image to a batch. -# image = tf.expand_dims(image, 0) -# return tf.concat(0, [central_x, image]), tf.add(i, 1) -# -# result = tf.while_loop(cond=c, body=body, loop_vars=(central_x, i), parallel_iterations=16) -# return result -# -# -# - -# diff --git a/tensorlayer/rein.py b/tensorlayer/rein.py index ad500a1f0..5021361e4 100644 --- a/tensorlayer/rein.py +++ b/tensorlayer/rein.py @@ -109,6 +109,7 @@ def log_weight(probs, weights, name='log_weight'): -------- Tensor The Tensor after appling the log weighted expression. + """ with tf.variable_scope(name): exp_v = tf.reduce_mean(tf.log(probs) * weights) diff --git a/tensorlayer/visualize.py b/tensorlayer/visualize.py index 8b3eabf97..12452a062 100644 --- a/tensorlayer/visualize.py +++ b/tensorlayer/visualize.py @@ -436,3 +436,66 @@ def plot_with_labels(low_dim_embs, labels, figsize=(18, 18), second=5, saveable= name=name, fig_idx=fig_idx) except ImportError: logging.info("Please install sklearn and matplotlib to visualize embeddings.") + + +def draw_weights(W=None, second=10, saveable=True, shape=[28, 28], name='mnist', fig_idx=2396512): + """Visualize every columns of the weight matrix to a group of Greyscale img. + + Parameters + ---------- + W : numpy.array + The weight matrix + second : int + The display second(s) for the image(s), if saveable is False. + saveable : boolean + Save or plot the figure. + shape : a list with 2 int + The shape of feature image, MNIST is [28, 80]. + name : a string + A name to save the image, if saveable is True. + fig_idx : int + matplotlib figure index. + + Examples + -------- + >>> tl.visualize.draw_weights(network.all_params[0].eval(), second=10, saveable=True, name='weight_of_1st_layer', fig_idx=2012) + + """ + import matplotlib.pyplot as plt + if saveable is False: + plt.ion() + fig = plt.figure(fig_idx) # show all feature images + size = W.shape[0] + n_units = W.shape[1] + + num_r = int(np.sqrt(n_units)) # 每行显示的个数 若25个hidden unit -> 每行显示5个 + num_c = int(np.ceil(n_units / num_r)) + count = int(1) + for row in range(1, num_r + 1): + for col in range(1, num_c + 1): + if count > n_units: + break + a = fig.add_subplot(num_r, num_c, count) + # ------------------------------------------------------------ + # plt.imshow(np.reshape(W[:,count-1],(28,28)), cmap='gray') + # ------------------------------------------------------------ + feature = W[:, count - 1] / np.sqrt((W[:, count - 1]**2).sum()) + # feature[feature<0.0001] = 0 # value threshold + # if count == 1 or count == 2: + # print(np.mean(feature)) + # if np.std(feature) < 0.03: # condition threshold + # feature = np.zeros_like(feature) + # if np.mean(feature) < -0.015: # condition threshold + # feature = np.zeros_like(feature) + plt.imshow(np.reshape(feature, (shape[0], shape[1])), cmap='gray', interpolation="nearest") #, vmin=np.min(feature), vmax=np.max(feature)) + # plt.title(name) + # ------------------------------------------------------------ + # plt.imshow(np.reshape(W[:,count-1] ,(np.sqrt(size),np.sqrt(size))), cmap='gray', interpolation="nearest") + plt.gca().xaxis.set_major_locator(plt.NullLocator()) # distable tick + plt.gca().yaxis.set_major_locator(plt.NullLocator()) + count = count + 1 + if saveable: + plt.savefig(name + '.pdf', format='pdf') + else: + plt.draw() + plt.pause(second) \ No newline at end of file