Skip connect for SirenNeRF

[dichen-cd]

Hi @ventusff , thank you for your work!

I was wondering why do you remove the skip connection in SirenNeRF?
Does it harm the performance? Or does skip connection make it difficult to decide the initialization method for SirenLayer?

Thanks.

[ventusff]

Hi @DeanChan
Sorry for the late reply!

1. Mostly because other works based-on sinosoidal activations all also removes the skip connections.
   You can refer to:

• Figure.2 of Pi-GAN
• Figure.I in the supp of DIF-NET
• Network structure of official siren code:

  SingleBVPNet(
  (image_downsampling): ImageDownsampling()
  (net): FCBlock(
      (net): MetaSequential(
      (0): MetaSequential(
          (0): BatchLinear(in_features=2, out_features=256, bias=True)
          (1): Sine()
      )
      (1): MetaSequential(
          (0): BatchLinear(in_features=256, out_features=256, bias=True)
          (1): Sine()
      )
      (2): MetaSequential(
          (0): BatchLinear(in_features=256, out_features=256, bias=True)
          (1): Sine()
      )
      (3): MetaSequential(
          (0): BatchLinear(in_features=256, out_features=256, bias=True)
          (1): Sine()
      )
      (4): MetaSequential(
          (0): BatchLinear(in_features=256, out_features=1, bias=True)
      )
      )
  )
  )

2. From my point of view, the reason why ReLU-based networks need skip connections is that ReLU partially stops information propagating as the negative part of the function is zero. Hence, as the features pass through the networks deeper and deeper, larger and larger part of the information in the original input is lost. Thus introducing a skip connection to a relatively deep network may help alleviate the effect and brings fresh original input to the middle of the forward pass. This is also the case with ELU, SELU activations, which all have very small activation in the negative part.
   In terms of sinosoidal activations, however, the information of the whole original input is always kept in the features, since it's a periodic function, spanning across the whole domain, capable of holding all parts and maybe all frequencies of the input.

3. I don't think determining the initialization method is difficult. The most possible explanations are the above two.
   For example, you can just zero-out the init weights of the dims corresponding to the skip input, then the initialization is just the same as the no-skip-connection version of the network.

Be that as it may, you can still give it a try anyway :)

[dichen-cd]

@ventusff Many thanks for your detailed info!!!