fix spbleu ram issue; loosen check for TestConfidenceFile

[mshannon-sil]

This PR fixes two bugs.

The first is #940, where the spbleu metric was reinitializing the Flores200 tokenizer on every iteration in the loop and consequently eating up all the RAM to the point of failure. I have moved intialization outside the loop to prevent that. The memory issue has gone away, and it has also had the side effect of drastically increasing the speed of the test step when spbleu is included.

The second is #947, where multilingual drafting was not compatible with saving confidence scores. This was only due to the fact that the code to check for a valid filename for a TestConfidenceFile was too restrictive and did not account for filenames where the lang id is inserted toward the beginning. I have loosened the check just to look for the presence of "trg-predictions" in the trg draft file name and confirmed multilingual drafting experiments now complete successfully.

---

This change is 

[TaperChipmunk32]

silnlp/nmt/test.py line 267 at r1 (raw file):

    scorers = scorers.intersection(SUPPORTED_SENTENCE_SCORERS)
    other_scores = {k: v for k, v in other_scores.items() if k.lower() in scorers}
    other_verse_scores: Dict[str, float] = {}

Why was this removed? It seems to still be needed later in this function.

[mshannon-sil]

silnlp/nmt/test.py line 267 at r1 (raw file):

Previously, TaperChipmunk32 (Matthew Beech) wrote…

Why was this removed? It seems to still be needed later in this function.

That's because for some reason there was a duplicate definition. I removed this definition, and kept the one inside the loop some lines later in line 298.

[TaperChipmunk32]

@TaperChipmunk32 made 1 comment.
Reviewable status: 0 of 2 files reviewed, 1 unresolved discussion (waiting on mshannon-sil).

[TaperChipmunk32]

silnlp/nmt/test.py line 300 at r1 (raw file):

            other_verse_scores: Dict[str, float] = {}
            if "chrf3" in scorers:
                chrf3_verse_score = sacrebleu.sentence_chrf(

Why was this changed?

[mshannon-sil]

silnlp/nmt/test.py line 300 at r1 (raw file):

Previously, TaperChipmunk32 (Matthew Beech) wrote…

Why was this changed?

When I was making the fixes, I noticed that a couple scorers were still using corpus level metrics, even though these are sentence level scores. So I adjusted those scorers to use their sentence-level rather than corpus-level scorers.

[TaperChipmunk32]

@TaperChipmunk32 made 1 comment and resolved 2 discussions.
Reviewable status: 0 of 2 files reviewed, all discussions resolved (waiting on mshannon-sil).