Removes vocabs and propogates fixes for breaks

[jerinphilip]

This PR removes sourceVocab() and targetVocab() exposed by Service as requested by Mozilla for interface towards a cleaner API.

sourceVocab() is unused and harmless to remove. Removing targetVocab() breaks existing marian-decoder-new and consequently regression-tests. Further changes in the PR propogates the vocab change to provide a Response based output printing in the regression-test-app. We have lost capabilities to completely benchmark with marian-decoder for time being as the new OutputCollector equivalent is a dumbed down version which only couts lines.

Keeping the changes minimal for reducing review-load (edit: oops, sorry, accidentally removed a bit of marian::Histories in the process). Marking as ready for review.

Regression tests (2/4) are failing for bytearray loads for bergamot-translator-app and service, which is already known and will fix with submodule update.

Regression-tests for service-cli which now tests the preliminary alignment and annotations are succeeding if PR merges. bergamot-translator-app stays intact, so no harm done.

Checklist

• Prettify diff to not run over unnecessary lines.
• Regression-tests - basic: [1/4]
• Regression-tests - speed: [1/1]

[jerinphilip]

There is a potential bug in main. If a sentence is attempted to be inserted corresponding to an empty std::vector<string_view>, we have an invalid sentence in the flat container for string_views, where I additionally mark sentence boundaries. This breaks on WNGT20 sources.shuf in line 1695, where an invalid character sentence is provided and the source vocab eats it. The resulting target is empty with no string_view and causes issues with word-counts and also sentence boundary.

For now, I have pushed the eos corresponding string_view in to get WNGT20 to complete, which seems the simplest. Looks harmless for the time-being. I have strengthened the unit-tests in the debugging process. (These are the few additional commits).

This is handling invalid user-input - not something strictly wrong with the container (Annotation).

[kpu]

Focused PRs please. This appears to be both fixing annotations and removing vocabs. Separate them.

[jerinphilip]

@kpu #85. I can only test the changes here, which is why they're in source. can you review that PR first?

(I discovered bug here, cherry-picked commits separating them to a different PR at #85; When they're in source, this diff will simplify).

[kpu]

By ordering these you've make it much harder to get in. You can simplify the diff here now.

[jerinphilip]

By ordering these you've make it much harder to get in. You can simplify the diff here now.

I have decoupled changes to simplify diff, this source breaks marian-decoder-new and the speed tests bound to it.

[abhi-agg]

@jerinphilip Does it make sense to merge main to this PR? It seems to be out of date with main. Although, I don't know whether it will impact the review or not 👍

[kpu]

@abhi-agg I hit the magic merge button, please review

[jerinphilip]

Does it make sense to merge main to this PR?

Well, there are no more bugs in main than what exists already with this in, so... Just breaks marian-decoder-new on invalid inputs thanks to sentencepiece that's all. Fix is in #85 which after hours of struggle I believe cannot improve much beyond.

[jerinphilip]

I'll take advantage of this situation and work the other way. I pull this source into annotation-bugfix branch through main.