What does the mlp and Mish function in modules.py do

[Dyongh613]

self.mlp = nn.Sequential(
LinearNorm(residual_channels, residual_channels * 4),
Mish(), # return x * torch.tanh(F.softplus(x))
LinearNorm(residual_channels * 4, residual_channels)
)

class Mish(nn.Module):
def forward(self, x):
return x * torch.tanh(F.softplus(x))

[keonlee9420]

Hi @qw1260497397 , it is for the embedding of diffusion time steps as described in the paper.

[keonlee9420]

Close due to inactivity.