Memoize beam hypos on the way

[uralik]

So actually previous way of getting current hypos was terrible (I did it so yeah, terrible), with ngram block 7 speed was like this on twitter:

33s elapsed: {'exs': 32, '%done': '0.31%', 'time_left': '10717s', 'accuracy': 0, 'f1': 0.08652, 'bleu': 5.141e-06, 'token_acc': 0.3022, 'loss': 4.142, 'ppl': 62.91}                                                                                                                                                        
60s elapsed: {'exs': 64, '%done': '0.62%', 'time_left': '9728s', 'accuracy': 0, 'f1': 0.07378, 'bleu': 5.256e-06, 'token_acc': 0.3022, 'loss': 4.177, 'ppl': 65.2}                                                                                                                                                          
109s elapsed: {'exs': 96, '%done': '0.92%', 'time_left': '11792s', 'accuracy': 0, 'f1': 0.08872, 'bleu': 0.004175, 'token_acc': 0.3049, 'loss': 4.102, 'ppl': 60.47}                                                                                                                                                        
159s elapsed: {'exs': 128, '%done': '1.23%', 'time_left': '12795s', 'accuracy': 0, 'f1': 0.08498, 'bleu': 0.003132, 'token_acc': 0.3106, 'loss': 4.014, 'ppl': 55.35}

with new hyps cache it is like this:

2s elapsed: {'exs': 32, '%done': '0.31%', 'time_left': '704s', 'accuracy': 0, 'f1': 0.08652, 'bleu': 5.141e-06, 'token_acc': 0.3022, 'loss': 4.142, 'ppl': 62.91}                                                                                                                                                          
6s elapsed: {'exs': 96, '%done': '0.92%', 'time_left': '733s', 'accuracy': 0, 'f1': 0.08872, 'bleu': 0.004175, 'token_acc': 0.3049, 'loss': 4.102, 'ppl': 60.47}                                                                                                                                                           
9s elapsed: {'exs': 128, '%done': '1.23%', 'time_left': '767s', 'accuracy': 0, 'f1': 0.08498, 'bleu': 0.003132, 'token_acc': 0.3106, 'loss': 4.014, 'ppl': 55.35}                                                                                                                                                          
11s elapsed: {'exs': 160, '%done': '1.54%', 'time_left': '758s', 'accuracy': 0, 'f1': 0.07828, 'bleu': 0.002505, 'token_acc': 0.2996, 'loss': 4.15, 'ppl': 63.41}

my bad