Graph Scope

[lucabergamini]

Hi,
I'm a Pytorch beginner (previously working on th and tf) using your tensorboard-pytorch bridge to get some info during neural net training. It works like a charm for everything i've needed since now :). However i'm having some troubles with the graph visualization for some ConvNet and i've a few questions:

• Is it possible to define scope for module? (i.e. if I'have a nested module can i give it a name to obtain a compress visualization of everything inside it?) ;
• using the torch.cat() method i get some strange behaviours related to the order of the elements in the list of the first argument of cat() (see attached images, where the second one is correct for the code reported) .

Btw great work :)

def forward(self,i):
        # i stand as the input
        # conv_j is a module
        x = self.conv_0(i)
        x = self.conv_1(x)
        y = self.conv_r1(i)
        z = torch.cat((y,x),1)
        z = z.view(len(z),-1)
        z = self.fc1(z)
        z = F.relu(z)
        z = self.fc2(z)
        z = F.log_softmax(z)
        return z

[lanpa]

I tried the following code

import torch
import torch.nn as nn
from torch.autograd.variable import Variable
import torch.nn.functional as F
from collections import OrderedDict
from tensorboard import SummaryWriter
class M(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv_0 = nn.Conv2d(1,1,3)
        self.conv_1 = nn.Conv2d(1,1,3)
        self.conv_r1 = nn.Conv2d(1,1,5)
        self.fc1 = nn.Linear(2,1)
        self.fc2 = nn.Linear(1,1)
    def forward(self,i):
            # i stand as the input
            # conv_j is a module
            x = self.conv_0(i)
            x = self.conv_1(x)
            y = self.conv_r1(i)
            z = torch.cat((y,x),1)
            z = z.view(len(z),-1)
            z = self.fc1(z)
            z = F.relu(z)
            z = self.fc2(z)
            z = F.log_softmax(z)
            return z


writer = SummaryWriter('runbug')
m = M()
z = m(Variable(torch.Tensor(1,1,5,5), requires_grad=True))

writer.add_graph(m, z)
writer.close()

The result seems correct, can you provide a runnable code to reproduce the first graph?

As for defining scope for modules, it needs to retrieve the module name associated with certain function object. I will look into it in the future.

[lucabergamini]

Using your code everything works fine, but if you introduce ReLU and MaxPooling you get this:

import torch
import torch.nn as nn
from torch.autograd.variable import Variable
import torch.nn.functional as F
from collections import OrderedDict
from tensorboard import SummaryWriter
from datetime import datetime

class M(nn.Module):
    def __init__(self):
        super(M,self).__init__()
        self.conv_0 = nn.Conv2d(1,1,3)
        self.conv_1 = nn.Conv2d(1,1,3)
        self.conv_r1 = nn.Conv2d(1,1,5)
        self.fc1 = nn.Linear(2,1)
        self.fc2 = nn.Linear(1,1)
    def forward(self,i):
            # i stand as the input
            # conv_j is a module
            x = self.conv_0(i)
            x = F.relu(x)
            x = F.max_pool2d(x,1)
            x = self.conv_1(x)
            x = F.relu(x)
            x = F.max_pool2d(x,1)
            y = self.conv_r1(i)
            y = F.relu(y)
            y = F.max_pool2d(y,1)

            z = torch.cat((x,y),1)
            z = z.view(len(z),-1)
            z = self.fc1(z)
            z = F.relu(z)
            z = self.fc2(z)
            z = F.log_softmax(z)
            return z


writer = SummaryWriter('runs/'+datetime.now().strftime('%B%d  %H:%M:%S'))
m = M()
z = m(Variable(torch.Tensor(1,1,5,5), requires_grad=True))

writer.add_graph(m, z)
writer.close()

[lanpa]

OK, I just found a more compact network to reproduce strange output:

class M(nn.Module):
    def __init__(self):
        super(M,self).__init__()
        self.conv_x1 = nn.Conv2d(1,1,3)
        self.conv_x2 = nn.Conv2d(1,1,4)
        self.conv_y = nn.Conv2d(1,1,6)
    def forward(self,i):
        x = self.conv_x1(i)
        x = self.conv_x2(x)
        x = F.relu(x)
        y = self.conv_y(i)
        y = F.relu(y)
        z = torch.cat((x,y),1)
        return z

[lucabergamini]

As far as i can see your network seems fine to me.

However, changing to this:

class M(nn.Module):
    def __init__(self):
        super(M,self).__init__()
        self.conv_x1 = nn.Conv2d(1,1,3)
        self.conv_x2 = nn.Conv2d(1,1,4)
        self.conv_y = nn.Conv2d(1,1,6)
    def forward(self,i):
        x = self.conv_x1(i)
        x = F.relu(x)
        x = self.conv_x2(x)
        x = F.relu(x)
        y = self.conv_y(i)
        y = F.relu(y)
        y = F.max_pool2d(y,1)
        z = torch.cat((x,y),1)
        return z

Leads to strange behaviours.

P.S. I'm a bit puzzled about the input tensors, sometimes I can't get them rendered (as in this last figure).

[lanpa]

Did you set requires_grad=True for the input variable?
btw, the graph drawing is based on autograd's back-propagation graph, I am wondering whether autograd uses internal optimizations to speed up the computation and causes this behavior. Anyway, I will look into it later :)

[lucabergamini]

I completely forgot it, that makes a lot of sense :)

Yeah that could explains it all...unfortunately autograd is used only in Pytorch as far as i know, so we can't just compile the same network under TF to check if the graphs match. Thank you in advance anyway, I'll try to look at the code behind in my spare time :)

[miguelvr]

is it possible to keep the module operations inside the layer block (scope)? Right now it only seems to associate weights with the scope, but it would be nice to assign the module's forward operations to the scope as well

[lucabergamini]

You mean to have every layers inside a user-defined module inside a single block with the name od the module?

[miguelvr]

no, I meant the operations related to a given nn.Module being inside that module's scope.

If you add a graph for 2 linear layers you get 2 scopes one for each, containing only the weight and bias variables while the operations over theses variables are shown outside the scope.

It would be nice to have operations like the addmm for the linear to be positioned within the scope of it's module. The same applies to the examples above with the ConvNd and Threshold operations that appear outside the scope of the convolutional layers

[lucabergamini]

For the ```addmul`` case it should be easy to implement, because in fact the operation lies in the same module (the Linear one), while for the the threshold (AKA ReLU , Sigmoid, etc) I'm not quite sure since it has its own module (which you usually add in forward using F.relu() as example), so I think we need the equivalent of the scope from TF.

[miguelvr]

I think the activations should be outside if you define them outside, anyway! By doing that the graph would be much cleaner

[lanpa]

The name of variable comes from
https://github.com/lanpa/tensorboard-pytorch/blob/master/tensorboard/graph.py#L36
However there seem no similar trick to extract that information from autograd's graph. I think the reason is that it's a pretty low level operation, object's name is not important.
The good news is there is a ongoing work that makes trace possible. Once confirmed success, it should be easy to draw a pretty graph. My guess XD