Frequency threshold?

[emjotde]

Hi,
Would it make sense to add a minimal frequency threshold instead of a maximal number of symbols? That way we could make sure that the symbols are being seens at least N times during NMT training.

[rsennrich]

Hi Marcin,
I added a frequency threshold - I'd guess it will be more useful to use both the frequency threshold and max vocabulary size in combination, rather than the frequency threshold alone.

Note that the frequency threshold does not prevent seeing rare symbols at training time. A symbol can be rare at training time because most of its occurrences have been merged into larger symbols, or because the merge operations are learned on a different corpus (for instance on the concatenation of the source and target text).