Issue with evaluating model with beam search

[pavlos2094]

Hello,

Thank you for providing this well-written and useful repository. After having trained a model, I try to evaluate the saved checkpoint model using beam search with a command similar to the one from the README:

python core/train.py --config configs/baseline.yaml --mode eval --restore experiments/finals-baseline/checkpoint.pt --expname eval-baseline --beam-size 10

However, I am getting an issue which produces a stack trace like this:

Traceback (most recent call last):  
    File 'KOBE/core/train.py', line 371, in <module>  
        score = eval_model(model, data, params, config, device, writer)  
    File 'KOBE/core/train.py', line 250, in eval_model
        samples, alignment = model.beam_sample(
    File 'KOBE/core/models/tensor2tensor.py', line 467, in beam_sample
        b.advance(output[j, :], attn[j, :]) # batch index
    File 'KOBE/core/models/beam.py', line 101, in advance
        self.attn.append(attnOut.index_select(0, prevK))
RuntimeError: "index_select_out_cuda_impl" not implemented for 'Float'
Process finished with exit code 1

It seems to me that it actually makes sense to happen, since we are trying to index a tensor (attnOut) with a tensor of floats (prevK). Here is the code chunk from beam.py for reference:

prevK = bestScoresId / numWords
self.prevKs.append(prevK)
self.nextYs.append((bestScoresId - prevK * numWords))
self.attn.append(attnOut.index_select(0, prevK))

Am I doing something wrong here? Thanks.

[qibinc]

Hi @pavlos2094 ,

Thanks for your interest in this work. This problem is caused by how division is treated in different versions of PyTorch (line prevK = bestScoresId / numWords). Please either use the torch version in README or modify this line to prevK = bestScoresId // numWords

[pavlos2094]

@qibinc thank you for your immediate response!

[ssexuejinwei]

@qibinc i follow your suggestion with "modify the line prevK = bestScoresId // numWords", but it seems still something wrong. The progress bar does not change for maintaining 0% and can not get the evaluating result
0%| | 0/82 [00:00<?, ?it/s]

[qibinc]

Hi @ssexuejinwei ,

We have rewritten this codebase with latest versions of DL frameworks and significantly improved the installation, code quality, reproducibility and visualization.

We have also replaced beam search with nucleus sampling, a method proven to be more suitable in text generation than beam search to avoid repetition. You can find how to use it at https://github.com/THUDM/KOBE#evaluating-kobe.

We also added BERTScore for evaluation, a metric better than BLEU in aligning with human judgement.

Here is a screenshot of the KOBE-v2's training progress, in case your are still interested:

Best,