Get one of the previous layer by layer name (string)

[zsdonghao]

Hi all, I think we should support users to get one of the previous layer by using layer name, as follow:

x = tf.placeholder(tf.float32, [None, 300])
net = InputLayer(x)
net = DenseLayer(net, name='dense1')
net = DenseLayer(net, name='dense2')

net['dense1'].outputs  <==== this one

Reference

• tl.layers.get_layers_with_name

• pycaffe

net.tops['conv1'] = L.Convolution(bottom="image", kernel_size=3, stride=1, num_output=32, pad=1, weight_filler=dict(type='xavier'))
net.tops['relu1'] = L.ReLU(net.tops['conv1'], in_place=True)
net.tops['conv2'] = L.Convolution(net.tops['relu1'], kernel_size=3, stride=1, num_output=64, pad=1, weight_filler=dict(type='xavier'))
...

net.tops['conv1'] <==

[DEKHTIARJonathan]

Very easy to implement, I have a few students who might want to work on this (@BaptisteAmato interested ?)

Would you might not doing it now and add contribution wanted to the title of the issue?

[DEKHTIARJonathan]

@zsdonghao which API would you prefer using ?

import tensorflow as tf
import tensorlayer as tl

x = tf.placeholder(tf.float32, [None, 300])
net = InputLayer(x)
net = DenseLayer(net, name='dense1')
net = DenseLayer(net, name='dense2')

## API 1:
layer = net['dense1']

## API 2:
layer = net.tops['dense1']

[zsdonghao]

@DEKHTIARJonathan I prefer to use API 1. Thanks, I added help wanted label ~

[DEKHTIARJonathan]

Here is some code to help on the implementation, the "Layer" class needs to be made subscriptable.

Here is a simple example on how to do this:

class SuperHero(object):
    def __init__(self):
        # My code ===> I have the power to decide who is the strongest :D 
        self.heros_strength = {
            "SuperMan": 666,
            "Batman": 650,
            "Hulk": 6e10,
            "Prof. Xavier": 10
        }

    def __getitem__(self, item):
        try:
            return self.heros_strength[item]
        except KeyError:
            raise ValueError("The SuperHero `%s` is not cool enough to be known !" % item)


myHero = SuperHero()
print("Superman strength: %d" % myHero["SuperMan"])
print("Hulk strength: %d" % myHero["Hulk"])

print("SpiderMan strength: %d" % myHero["SpiderMan"])

@BaptisteAmato would you be interested in having a look at this ? It would a very good quickstart.
If you have any question, myself and @zsdonghao would be glad to help you if you decide to gve it a shot.

[BaptisteAmato]

I'll check this issue. I'm totally new to this repo so it'll help me dive into the code.

[BaptisteAmato]

It seems that in the reference tl.layers.get_layers_with_name, a Tensor is returned and not a Layer. Should "net['dense1']" return a Tensor? In that case, there could not be an output attribute "net['dense1'].outputs".

[DEKHTIARJonathan]

@zsdonghao I let you answer where you would like this contribution to be driven.

[zsdonghao]

@BaptisteAmato @DEKHTIARJonathan hi, for building a network, I think return a network could be better, otherwise, users have to feed the tensor into an InputLayer manually.

For example, the current layer support slicing as follow

>>> x = tf.placeholder("float32", [None, 100])
>>> n = tl.layers.InputLayer(x, name='in')
>>> n = tl.layers.DenseLayer(n, 80, name='d1')
>>> n = tl.layers.DenseLayer(n, 80, name='d2')
>>> print(n)
... Last layer is: DenseLayer (d2) [None, 80]

>>> n2 = n[:, :30]
>>> print(n2)
... Last layer is: Layer (d2) [None, 30]

The code is here:
https://github.com/tensorlayer/tensorlayer/blob/master/tensorlayer/layers/core.py#L477

[DEKHTIARJonathan]

How would you implement this?
Before returning the Tensor you feed it into an InputLayer?
Because I'm not sure if there is anyway to find the previously created Layer object

[BaptisteAmato]

@zsdonghao I totally agree. However current code seems kind of weird to me: when using integer indexes, a network is returned like this:

net_new = Layer(prev_layer=None, name=self.name)
net_new.inputs = self.inputs
net_new.outputs = self.outputs[key]
net_new.all_layers = list(self.all_layers[:-1])
net_new.all_layers.append(net_new.outputs)
net_new.all_params = list(self.all_params)
net_new.all_drop = dict(self.all_drop)
return net_new

It seems that it returns the right outputs but the other parameters are from the entire network...

[zsdonghao]

@BaptisteAmato yes.. that is a problem ..

What if we store all previous layer object here? https://github.com/tensorlayer/tensorlayer/blob/master/tensorlayer/layers/core.py#L408

[BaptisteAmato]

Yeah that's what I would do. @DEKHTIARJonathan what do you think?

[DEKHTIARJonathan]

Could easily be done within the base class Layer.

[DEKHTIARJonathan]

You have two ways.

First, you create a list of layers and you loop through it.

However, names in TensorFlow shall be unique, so I would say that dictionary is a much more efficient approach.

One down side doing this, @zsdonghao we deprecated the function that managed tl layers name. Shall we bring it back to life? Because we need to make sure that the graph has not been reset since the layer has been created.

If I have the time, I will have a look to Pytorch and look how they implemented this

[zsdonghao]

@DEKHTIARJonathan I think we can use TF name to check whether the layer is unique ? ~ Let me think~.

[DEKHTIARJonathan]

I think we can used a linked list in Layer Base class. Each layer has a direct connection to the previous one (Layer) or ones (list of Layers: ConcatLayer & MergeLayer).
Usually a network does not have thousands of thousands of Layers. We should be fine about the time & space complexity.

The advantage is that we don't need to maintain a list or dictionnary of layers ;) (This behaviour can rapidly be unstrusful). And we can add some additional check, everytime a layer is accessed, we can check if the layer still exists in the TF Graph, else we raise a RuntimeException.

Both features (linked_list of Layers & checking if the layer still exists in the TF Graph) can easily be implemented in the Base Layer Class without modifying the whole library.

[lgarithm]

I was going to add the linked list https://github.com/tensorlayer/tensorlayer/pull/469/files#diff-1a0995d02752855445936bb440663aceR401

[DEKHTIARJonathan]

What would you need to release to this feature @lgarithm?

[zsdonghao]

implementing here #755