Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Empty prediction on CNN/DM with beam > 1 #457
The problem may come from the dataset which I processed again. I'm basically running:
with beam_size = 1: redundant, but not empty, e.g.
with beam_size > 1: empty, each beam produce and eos (token_id = 3)
but also some that contains redundancy:
This is not a trivial problem, I really don't know how this happens.
Hm, I have run into a related problem in the past where one in every ~20 predictions was empty, even with beam size 1. Looking at the other top predictions, everything seems normal. I have not been able to replicate this error consistently yet, but a simple fix is to set the probability of EOS to -1e7 or so for the very first step. Let me know if you make progress figuring this bug out!
I just checked Abisee's work, she is indeed using an min_length option, and discarding beams that are too short.
Adding an option would make the implementation in line with her work then.