You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In learn_bpe.py, the function prune_stats has code as follow:
for item,freq in list(stats.items()):
if freq < threshold:
del stats[item]
if freq < 0:
big_stats[item] += freq
else:
big_stats[item] = freq
I want to ask why the freq can bellow zero? This conditional judgment is for what?
The text was updated successfully, but these errors were encountered:
big_stats, which is the full collection of symbol pairs
stats, which is a pruned version of big_stats that initially only contains the most frequent symbol pairs, but is regularly synced with big_stats. Most operations (updating symbol pair frequencies; finding the most frequent one) are done on stats.
if a symbol pair is not among the the most frequent pairs, it may be pruned from stats (and the default frequency is 0), but the frequency of a symbol pair will actually decrease as neighbouring symbols are merged (if you have "a b c", and "b c" are merged, then the frequency of "a b" decreases). That's how frequencies in stats can become negative, and why the code you're quoting makes sure to update big_stats correspondingly.
In learn_bpe.py, the function prune_stats has code as follow:
for item,freq in list(stats.items()):
if freq < threshold:
del stats[item]
if freq < 0:
big_stats[item] += freq
else:
big_stats[item] = freq
I want to ask why the freq can bellow zero? This conditional judgment is for what?
The text was updated successfully, but these errors were encountered: