How to build a GAN with O-CNN ?

[TrepangCat]

Hi, I am trying to build a GAN network with your O-CNN code. Here are some problems I want to ask:

1. During the forward process of convolution, does the input octree changes? or it is just for index.
2. How can I constract a decoder as the generator of GAN? Should I create a original octree as one of the inputs? And how?
3. Is there anything else I need to know for constracting a decoder?

[TrepangCat]

By the way, I am using pytorch.

[TrepangCat]

I build a generator. Is this right?

class G(nn.Module):
    def __init__(self, flags):
        super(G, self).__init__()

        self.depth = 6  # 64**3, voxels

        self.deconv_bn_relu_1 = ocnn.OctreeDeConvBnRelu(depth=1, channel_in=200, channel_out=512,
                                                        kernel_size=[4], stride=2)

        self.deconv_bn_relu_2 = ocnn.OctreeDeConvBnRelu(depth=2, channel_in=512, channel_out=256,
                                                        kernel_size=[4], stride=2)
        self.predict_2 = self._make_predict_module(channel_in=256)

        self.deconv_bn_relu_3 = ocnn.OctreeDeConvBnRelu(depth=3, channel_in=256, channel_out=128,
                                                        kernel_size=[4], stride=2)
        self.predict_3 = self._make_predict_module(channel_in=128)

        self.deconv_bn_relu_4 = ocnn.OctreeDeConvBnRelu(depth=4, channel_in=128, channel_out=64,
                                                        kernel_size=[4], stride=2)
        self.predict_4 = self._make_predict_module(channel_in=64)

        self.deconv_bn_relu_5 = ocnn.OctreeDeConvBnRelu(depth=5, channel_in=64, channel_out=32,
                                                        kernel_size=[4], stride=2)
        self.predict_5 = self._make_predict_module(channel_in=32)

        self.deconv_6 = ocnn.OctreeDeconv(depth=6, channel_in=32, channel_out=1,
                                          kernel_size=[4], stride=2)
        self.octree2voxel = ocnn.FullOctree2Voxel(6)
        self.sigmoid_6 = torch.nn.Sigmoid()

    def forward(self, input):
        # input is the latent code Z, with the size of (N, 200, 1, 1, 1)
        data = input

        # octree = ocnn.octree_new(batch_size=1, channel=4, node_dis=True, adaptive_layer=0)
        octree = ocnn.create_full_octree(depth=self.depth, channel=4, batch_size=1, node_dis=True)
        data = self.deconv_bn_relu_1(data, octree)

        data = self.deconv_bn_relu_2(data, octree)
        label = self.predict_2(data)
        octree = ocnn.octree_update(octree, label, depth=2, split=1)

        data = self.deconv_bn_relu_3(data, octree)
        label = self.predict_3(data)
        octree = ocnn.octree_update(octree, label, depth=3, split=1)

        data = self.deconv_bn_relu_4(data, octree)
        label = self.predict_4(data)
        octree = ocnn.octree_update(octree, label, depth=4, split=1)

        data = self.deconv_bn_relu_5(data, octree)
        label = self.predict_5(data)
        octree = ocnn.octree_update(octree, label, depth=5, split=1)

        data = self.deconv_6(data, octree)
        data = self.octree2voxel(data)
        data = self.sigmoid_6(data)

        return data


    @staticmethod
    def _make_predict_module(channel_in, channel_out=2, num_hidden=32):
        return torch.nn.Sequential(
            ocnn.OctreeConv1x1BnRelu(channel_in, num_hidden),
            ocnn.OctreeConv1x1(num_hidden, channel_out, use_bias=True))

And I have a few questions about octree code:

Q1: For ocnn.octree_new(), what are the meanings of parameter node_dis, adaptive_layer, and channel? (Is channel the dimension of normals?)
Q2: For ocnn.octree_grow(), what are the meanings of parameter full_octree?
Q3: When to use octree_grow()?
Q4: What are the differences between creating a full octree at the beginning of the forward process and using octree_grow() for each depth during the forward process?
Q5: Does the label in octree_update() decide which nodes in this depth have child nodes?
Q6: What is the function of ocnn.octree_align()? (I mean that if I use ocnn.octree_align(), what would happen?)

[wang-ps]

Hi, I am trying to build a GAN network with your O-CNN code. Here are some problems I want to ask:

1. During the forward process of convolution, does the input octree changes? or it is just for index.
2. How can I constract a decoder as the generator of GAN? Should I create a original octree as one of the inputs? And how?
3. Is there anything else I need to know for constracting a decoder?

1. In the encoder, the octree is fixed. In the decoder, the octree can be predicted dynamically.
2. It is non-trivial to build GAN with o-cnn. It is an open question.
3. You can refer to the code for shape completion here.

[wang-ps]

Q1: For ocnn.octree_new(), what are the meanings of parameter node_dis, adaptive_layer, and channel? (Is channel the dimension of normals?)
Q2: For ocnn.octree_grow(), what are the meanings of parameter full_octree?
Q3: When to use octree_grow()?
Q4: What are the differences between creating a full octree at the beginning of the forward process and using octree_grow() for each depth during the forward process?
Q5: Does the label in octree_update() decide which nodes in this depth have child nodes?
Q6: What is the function of ocnn.octree_align()? (I mean that if I use ocnn.octree_align(), what would happen?)

1. If node_dis = True, we interpret the fourth channel of octree features as the displacement. adaptive_layer indicates the octree layer where the adaptive octree starts. the channel is the channel number of octree features.
2. full_octree means currently the octree is a full octree
3. please refer to the code for shape completion for the usage of octree_grow
4. There is no big difference.
5. Yes.
6. The octree_align is the output guided skip connections proposed in our paper for shape completion here

[TrepangCat]

Thanks a lot. O-CNN is a wonderful work. I will keep going on constructing a GAN with the octree.

Regards,
TrepangCat

[timmonspatrick]

Hi @TrepangCat ,

I'm curious if you've made progress on your work with constructing a GAN with the octree, and if so, would you be willing to share your implementation?

[TrepangCat]

Sorry, I didn't finish it. There are some problems I can not solve.