Questions on using the CRF layer

[netw0rkf10w]

Hello,

I would like to ask some questions the CRF layers that you proposed.

Suppose I can get some CNN (e.g. FCN or DeepLab) using:

cnn = get_cnn()

This CNN can be trained with the usual cross-entropy loss without any issue.

Now to put a CRF on top of it, I define some model for combining a CNN and a CRF like this:

class CNNCRF(torch.nn.Module):
    def __init__(self, cnn, crf):
        super().__init__()
        self.cnn = cnn
        self.crf = crf

    def forward(self, x):
        unaries = self.cnn(x)
        edge_feat = [paccrf.create_YXRGB(x, 80.0, 13.0),
                 paccrf.create_position_feats(x.shape[2:], 3.0, bs=x.shape[0], device=x.device)]             
    return self.crf(unaries, edge_feat)

Then I create a CRF layer and add it to the model:

compat = '2d'
kernel_size = 11
blur=4
dilation = 1
crf_params = dict(num_steps=crf_iterations, perturbed_init=True, fixed_weighting=False, unary_weight=0.8,
pairwise_kernels=[
    dict(kernel_size=kernel_size, dilation=dilation, blur=blur, compat_type=compat, spatial_filter=False,
        pairwise_weight=2.0),
    dict(kernel_size=kernel_size, dilation=dilation, blur=blur, compat_type=compat, spatial_filter=False,
        pairwise_weight=0.6)])
crf = paccrf.PacCRF(num_classes, **crf_params)
model = CNNCRF(cnn, crf)  

I tried training this model by freezing the CNN part (the total number of trainable parameters of the model is then 885) using Adam optimizer with a learning rate of 1e-4, but it was very difficult to converge. Could you please tell me if what I did above is correct?

Thank you in advance for your response.

[netw0rkf10w]

In addition, I got the following error when training on multiple GPUs:

RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument find_unused_parameters=True to torch.nn.parallel.DistributedDataParallel; (2) making sure all forward function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's forward function. Please include the loss function and the structure of the return value of forward of your module when reporting this issue (e.g. list, dict, iterable). (prepare_for_backward at /pytorch/torch/csrc/distributed/c10d/reducer.cpp:514)

If I remove the CRF, everything works fine. So I must have done something wrong on that part...