Completing the CNN/Dailymail summarization pipeline

[urvashik]

This commit adds the following features:

• Raw test data generation for the train, dev and test splits for the cnn/dailymail data. This allows for evaluating the performance of the model in a way that is comparable to other models

• A script to compute rouge scores using the official pyrouge distribution.

  • Ensure that you have pyrouge setup on your machine correctly (Will add a pyrouge test script in the future)
  • The script outputs warnings for empty model predictions. This is because the model did not generate a summary for that example. These were not suppressed since its a useful data point to have.

• A bash script that first runs the moses tokenizer on predictions and targets and then computes the rouge score using the aforementioned pyrouge based script.

• Just want to explicitly point out the changes made to utils/decoding.py in removing the extra space while generating decodes/targets files, since it was never merged before. Not sure if that might break other things.

[googlebot]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed, please reply here (e.g. I signed it!) and we'll verify. Thanks.

---

• If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
• If your company signed a CLA, they designated a Point of Contact who decides which employees are authorized to participate. You may need to contact the Point of Contact for your company and ask to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the project maintainer to go/cla#troubleshoot.
• In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again.

[urvashik]

I signed it

[lukaszkaiser]

Enormous thanks Urvashi!