Code for the paper Bidirectional Context-Aware Hierarchical Attention Network for Document Understanding
The Hierarchical Attention Network (HAN) has made great strides, but it suffers a major limitation: at level 1, each sentence is encoded in complete isolation. In this work, we propose and compare several modifications of HAN in which the sentence encoder is able to make context-aware attentional decisions (CAHAN). Furthermore, we propose a bidirectional document encoder that processes the document forwards and backwards, using the preceding and following sentences as context. Experiments on three large-scale sentiment and topic classification datasets show that the bidirectional version of CAHAN outperforms HAN everywhere, with only a modest increase in computation time. While results are promising, we expect the superiority of CAHAN to be even more evident on tasks requiring a deeper understanding of the input documents, such as abstractive summarization.
HAN (left) vs CAHAN (right) on an example extracted from the Yelp dataset.
HAN (left) vs CAHAN (right) on a motivational example.
V1
: Weights and records of the experiments.
weights
- Initial weightsbaseline
- Trained weights and results on the baselineagg=sum_bidir=True_discount=1_cutgradient=False
- Trained weights and results for the experiment with summed attention, bidirectional contextual attention and discount factor = 1.
code
: All the scripts needed to run the experiments. To run the experiments you can run themain_*
scripts.
This repository was developped using python 3.6
and Cuda 9.0
.
Requirements are contained in the requirements.txt
file.
If you use this code or build up on the idea proposed in the paper, please cite it as:
@article{remy2019bidirectional,
title={Bidirectional Context-Aware Hierarchical Attention Network for Document Understanding},
author={Remy, Jean-Baptiste and Tixier, Antoine Jean-Pierre and Vazirgiannis, Michalis},
journal={arXiv preprint arXiv:1908.06006},
year={2019}
}
If you liked this work you can follow the authors:
- Jean-Baptiste Remy
- Antoine Jean-Pierre Tixier
- Michalis Vazirgiannis