Skip to content
code for our NAACL 2019 paper: "BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis"
Branch: master
Clone or download
Type Name Latest commit message Commit time
Failed to load latest commit information.
eval fix a typo Apr 13, 2019
script add testing bert model when bert-base is not present May 2, 2019
src remove csv Apr 27, 2019
LICENSE Create LICENSE Apr 3, 2019 additional running environment Apr 27, 2019
result.ipynb add result.ipynb code to run all tasks. Apr 4, 2019

BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis

code for our NAACL 2019 paper "BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis".

***feedbacks are welcomed for missing instructions ***

Problem to Solve

We focus on 3 review-based tasks: review reading comprehension (RRC), aspect extraction (AE) and aspect sentiment classification (ASC).

RRC: given a question ("how is the retina display ?") and a review ("The retina display is great.") find an answer span ("great") from that review;

AE: given a review sentence ("The retina display is great."), find aspects("retina display");

ASC: given an aspect ("retina display") and a review sentence ("The retina display is great."), detect the polarity of that aspect (positive).



The code is tested on Ubuntu 16.04 with Python 3.6.8(Anaconda), PyTorch 1.0.1 and pytorch-pretrained-bert 0.4. We suggest make an anaconda environment for all packages and uncomment environment setup in script/ script/ script/


The post-training code additionally use apex 0.1 to speed up training on FP16, which is compiled with PyTorch 1.0.1(py3.6_cuda10.0.130_cudnn7.4.2_2) and CUDA 10.0.130 on RTX 2080 Ti. It is possible to avoid use GPUs that do not support apex (e.g., 1080 Ti), but need to adjust the max sequence length and number of gradient accumulation but (although the result can be better).

Fine-tuning code is tested without using apex 0.1 to ensure stability.


Our evaluation wrapper code is written in ipython notebook eval/eval.ipynb. But you are free to call the evaluation code of each task separately. AE eval/ additionally needs Java JRE/JDK to be installed.

Fine-tuning setup

step1: make 2 folders for post-training and fine-tuning.

mkdir -p pt_model ; mkdir -p run

step2: place post-trained BERTs into pt_model/. Our post-trained Laptop weights can be download here and restaurant here. You are free to download other BERT weights into this folder(e.g., bert-base). Make sure to add an entry into src/

step3: make 3 folders for 3 tasks:

place fine-tuning data to each respective folder: rrc/, ae/, asc/. A pre-processed data in json format can be found here.

step4: fire a fine-tuning from a BERT weight, e.g.

cd script
bash rrc laptop_pt laptop pt_rrc 10 0

Here rrc is the task to run, laptop_pt is the post-trained weights for laptop, laptop is the domain, pt_rrc is the fine-tuned folder in run/, 10 means run 10 times and 0 means use gpu-0.


bash rrc rest_pt rest pt_rrc 10 0
bash ae laptop_pt laptop pt_ae 10 0
bash ae rest_pt rest pt_ae 10 0
bash asc laptop_pt laptop pt_asc 10 0
bash asc rest_pt rest pt_asc 10 0

step5: evaluation

RRC: download SQuAD 1.1 evaluation script (e.g. ) to eval/.

AE: place official evaluation .jar files as eval/A.jar and eval/eval.jar. place testing xml files as (the step 4 of this has a similar setup)


ASC: built-in as part of eval/eval.ipynb

open result.ipynb and run as you wish

Post-training setup

Assume you are on the root folder of this repository.

step1: domain post-training data:

Download pre-processed data from here and place it under the root folder.

Alternatively you can create your own data. For example for laptop, mkdir -p domain_corpus ; mkdir -p domain_corpus/laptop ; mkdir -p domain_corpus/raw. Place newline separated reviews laptop.txt as domain_corpus/raw/laptop.txt The script in step3 will detect if your .npz is unavailable under domain_corpus/laptop then it will start to preprocess domain_corpus/raw/laptop.txt first.

step2: MRC post-training data:

Place and rename SQuAD 1.1 (for consistency with other review-based tasks) as squad/train.json.

step3: preprocessing and post-training

bash laptop 5 70000 0
bash rest 1 140000 0


If you find this work useful, please cite as following.

    title = "BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis",
    author = "Xu, Hu and Liu, Bing and Shu, Lei and Yu, Philip S.",
    booktitle = "Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics",
    month = "jun",
    year = "2019",
You can’t perform that action at this time.