Fails to evaluate wait-k model on SimulEval if sequence is shorter than k

[fury00812]

(Repost of the issue: facebookresearch/fairseq#3445)

I've got an error while evaluating the wait-k model following the instruction of README.md.

2021-04-03 10:36:28 | INFO     | simuleval.scorer | Evaluating on text
2021-04-03 10:36:28 | INFO     | simuleval.scorer | Source: /userdir/iwslt17.test0.en
2021-04-03 10:36:28 | INFO     | simuleval.scorer | Target: /userdir/iwslt17.test0.ja
2021-04-03 10:36:28 | INFO     | simuleval.scorer | Number of sentences: 1549
2021-04-03 10:36:28 | INFO     | simuleval.server | Evaluation Server Started (process id 132622). Listening to port 12321
2021-04-03 10:36:31 | WARNING  | simuleval.scorer | Resetting scorer
2021-04-03 10:36:31 | INFO     | simuleval.cli    | Output dir: /userdir/iwslt17.test0
2021-04-03 10:36:31 | INFO     | simuleval.cli    | Start data writer (process id 132639)
2021-04-03 10:36:31 | INFO     | simuleval.cli    | Evaluating SimulTransTextAgentJA (process id 132556) on instances from 0 to 1548
2021-04-03 10:36:31 | INFO     | fairseq.tasks.translation | [en] dictionary: 16004 types
2021-04-03 10:36:31 | INFO     | fairseq.tasks.translation | [ja] dictionary: 16004 types
Traceback (most recent call last):
  File "/userdir/.venv/bin/simuleval", line 11, in <module>
    load_entry_point('simuleval', 'console_scripts', 'simuleval')()
  File "/userdir/SimulEval/simuleval/cli.py", line 165, in main
    _main(args.client_only)
  File "/userdir/SimulEval/simuleval/cli.py", line 192, in _main
    evaluate(args, client, server_process)
  File "/userdir/SimulEval/simuleval/cli.py", line 145, in evaluate
    decode(args, client, result_queue, indices)
  File "/userdir/SimulEval/simuleval/cli.py", line 108, in decode
    action = agent.policy(states)
  File "/userdir/fairseq-v0.10.2/examples/simultaneous_translation/eval/agents/simul_t2t_enja.py", line 196, in policy
    x, outputs = self.model.decoder.forward(
  File "/userdir/fairseq-v0.10.2/fairseq/models/transformer.py", line 817, in forward
    x, extra = self.extract_features(
  File "/userdir/fairseq-v0.10.2/examples/simultaneous_translation/models/transformer_monotonic_attention.py", line 219, in extract_features
    x, attn, _ = layer(
  File "/userdir/.venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/userdir/fairseq-v0.10.2/examples/simultaneous_translation/modules/monotonic_transformer_layer.py", line 160, in forward
    x, attn = self.encoder_attn(
  File "/userdir/.venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/userdir/fairseq-v0.10.2/examples/simultaneous_translation/modules/monotonic_multihead_attention.py", line 667, in forward
    alpha = self.expected_alignment_infer(
  File "/userdir/fairseq-v0.10.2/examples/simultaneous_translation/modules/monotonic_multihead_attention.py", line 528, in expected_alignment_infer
    assert tgt_len == 1
AssertionError

This problem occurs when the sequence is shorter than k.

Environment

• fairseq Version: 0.10.2 (commit 14807a361202ba34dbbd3a533899db57a0ebda19)
• SimulEval Version: latest (commit 1753363071f989ea3b79fdf5a21b96089a002f36)

[fury00812]

Hi, we applied ad-hoc changes to address the problem.

1. comment out monotonic_multihead_attention.py#L528,
2. insert the following code between L693 and L694:

if p_choose.numel() != bsz*self.num_heads*tgt_len*src_len:
    p_choose = p_choose[:, 1]

this worked.