Using Bleu for batch input

[Ishan-Kumar2]

The example provided in docs for Bleu is for Single input. In that case the output from the engine should be in this format

def evaluate_step():
  ...
  predictions = "Predicted Sentence 1".split()
  references = ["Reference Sentence 1".split(), "Reference Sentence 1.2".split()]
  return (predictions, references)

When calculating Bleu score for a Batch what is the format for output from the engine?
It should be something like

#For a batch of size 2
predictions = ["Predicted Sentence 1".split(), "Predicted Sentence 2".split()]
references = [["Reference Sentence 1.1".split(), "Reference Sentence 1.2".split()], ["Reference Sentence 2.1".split(), "Reference Sentence 2.2".split()]]

Doing this gives an error.

TypeError: unhashable type: 'list'

The typing for update requires predictions to be a Sequence[Any] and References to be Sequence[Sequence[Any]]]. Does this mean the batch input is not possible?

[vfdev-5]

Cc @sdesrozis @gucifer

[sdesrozis]

@Ishan-Kumar2 Thanks for pointing this out.

You are right, we didn't expose a batchwise api. Note that the internal function to compute the bleu score could be use in that sense. However, it will compute a bleu score per batch (then average the batch scores) which would be different to the actual sentence bleu score.

ignite/ignite/metrics/nlp/bleu.py

Line 190 in cc2c1aa

self._sum_of_bleu += self._corpus_bleu(references=[y], candidates=[y_pred])

Would you like to help on this ? It would be a great improvement to fix the batch usage.

EDIT : diving further on this, we compute the average of the sentence bleu score as the corpus bleu score. It is the macro average bleu score. We could improve the implementation computing the corpus bleu score, or the micro average bleu score, meaning an internal logic refactoring (basically an accumulation of ngram counter, then the brevity penalty computing after sync).

For details, see the following comment from nltk (inspiration of our implementation)

https://github.com/nltk/nltk/blob/193cf488a3aa31b0e99219f1a8c4eb7567903069/nltk/translate/bleu_score.py#L113

[Ishan-Kumar2]

@sdesrozis thanks for the update, I am interested in contributing to this.
I agree we should implement the micro avg version also. Then the corpus_bleu can be renamed to sentence_bleu and the new corpus_bleu should be used for corpus/batch level calculations.

[sdesrozis]

Firstly, the _corpus_bleu should be split in 2 parts : (1) the accumulation of ngrams counter (maybe the lenghts of cand/hyp too) and (2) the computation of bp, smoothing and geometric mean. The sentence bleu score (ie macro avg) is (1) + (2) for each sentence and (3) average. The corpus bleu score (ie micro avg) is (1) for each sentence then (2).

For batch version, the macro avg means apply the sentence score to each sentence of the batch. We must add a loop over the batch, it is quite straighforward. The micro avg is natively fine with batch version (apply (1) on batch then (2)).

For DDP, the work is fine for the actual macro avg. Concerning the micro avg version, a synchronization is needed to sum the different counters and lenghts.

HTH

[Ishan-Kumar2]

Hey @sdesrozis
Since the _corpus_bleu already has a loop to iterate over a batch, what if instead of splitting it into 2 seperate functions, we call it once for micro-avg case and multiple times(for each hypothesis in batch) for macro avg.
Like this

def update(self, output: Tuple[Sequence[Sequence[Any]], Sequence[Sequence[Sequence[Any]]]], macro_averaging=False) -> None:
        y_pred, y = output
        
        if macro_averaging:
            #Calculate bleu score for each hypothesis indiviually and then average
            for refs, hyp in zip(y, y_pred):
                self._sum_of_bleu += self._corpus_bleu(references=[refs], candidates=[hyp])
                self._num_sentences += 1

        else:
            #Calculate bleu for entire corpus
            self._sum_of_bleu += self._corpus_bleu(references=y, candidates=y_pred)
            self._num_sentences += 1

[sdesrozis]

What you mentioned is correct only if the input is the whole corpus.

The corpus is split in batch, so we have to firstly accumulate the ngram counter, then finally compute the bp and mean when the corpus has been covered. It implies multiple update calls (accumulation) and one compute (bp + mean). In other words, the micro average you suggested is not correct.

And if you consider the distributed computing, it means each processor has a part of the corpus. Again, we accumulate, (sync) then bp + mean.

[Ishan-Kumar2]

Oh okay got it. My understanding of how ignite metrics work was slightly incorrect, cleared now!
Will raise a PR soon.