How to easily extracting attentions?

[zarzen]

Hello Tian Shi,
Thanks for sharing the project!

Is it easy to visualize attention like this?

BTW, I pasted following content into the demo website. The summary/highlights is exact copy of the source text. Is this because the text is too short? And why can't I input one sentence for abstractive summary?

Evernote helps you focus on what matters most and have access to your information when you need it. Input typed notes or scan handwritten notes. Add to-do’s, photos, images, web pages, or audio ... and it’s all instantly searchable. Create notebooks, organizers, planners. Organize notes any way you want and share with anyone. And Evernote syncs your notes and notebooks across your devices so your information is always with you, everywhere you go.

"Use Evernote as the place you put everything … Don’t ask yourself which device it’s on—it’s in Evernote" – The New York Times

“When it comes to taking all manner of notes and getting work done, Evernote is an indispensable tool.” – PC Mag

[tshi04]

It is easy to output the attention weights using LeafNATS. If you are just using the pretrained models, the headline2_summary2_app can output the generated summaries/headlines along with the attention weights.

I also observed that problem. Actually, it is an issue for Pointer-Generator network (90% of a summary is copied from source document). You can refer the paper
https://arxiv.org/abs/1808.10792
Beforehand, I only considered the Rouge scores. But after I built this webapp, I find that models trained with Newsroom dataset copy exactly the first three sentences in most cases. I think this problem will be resolved in the future. We might need other evaluations in addition to ROUGE scores.

[zarzen]

Thanks for quick reply!
The specific implementation I am looking for is the one for sentence level abstractive text summary. While I see the headline2_summary2_app is a document-level text summary if I am correct. Did you implement the mentioned in the paper "A Neural Attention Model for Abstractive Summarization." or the first part of the paper "Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond" ?

Thanks!

[tshi04]

You can try out different network structures with following options:

parser.add_argument('--rnn_network', default='lstm', help='gru | lstm')
parser.add_argument('--attn_method', default='luong_concat', help='luong_dot | luong_concat | luong_general')
parser.add_argument('--repetition', default='vanilla', help='vanilla | temporal | asee (coverage). Repetition Handling')
parser.add_argument('--pointer_net', type=str2bool, default=True, help='Use pointer network?')
parser.add_argument('--oov_explicit', type=str2bool, default=True, help='explicit OOV?')
parser.add_argument('--attn_decoder', type=str2bool, default=True, help='attention decoder?')
parser.add_argument('--share_emb_weight', type=str2bool, default=True, help='share_emb_weight')

I have just implemented the application module in pointer-generator-network. You can output attention weights as well just like headline2_summary2_app.