#Implementation of the paper "Attend to the beginning: A study on using bidirectional attention for extractivesummarization" To appear in (FLAIRS33). https://arxiv.org/pdf/2002.03405.pdf

It is a modification of the original SummaRuNNer Paper (https://arxiv.org/pdf/1611.04230.pdf) for discussion Thread summarization.

##1- Modifications ###a- Post to comment Co-attention: Co-attention mechanism is added between initial post sentence representations and each comment sentence representations. The co-attention produces a set of post aware comment representations. ###b- Pretraining: Model can be pretrained using different dataset, and then fine-tuned with different dataset. The finetuning can be done with the pretrained vocab or the embedding layer can be reinitialized with different vocab. ###c- Gradual layer unfreezing: Gradual layer unfreezing is integrating in this code influenced by https://arxiv.org/pdf/1801.06146.pdf.

Layers are freezed at the begining of the training process, then layers are gradually unfreezed starting from the top most layer down to the embedding layer. ###d- Backtranslation (under development):

---

##2- How to use:

###a- Data preprocessing: To either train, tune or test a model first you need to preprocess your data. You can run the preprocess.py file to achieve that. To run the preprocess.py, you can implement your data reading function in data_loader.py.

The reading function should read the data split into train, val, test parts and convert to the thread object. please refer to data_loader.py for examples.

In preprocess.py adjust these parameters to your liking :

params['DATA_Path'] = './cnn_data/finished_files/' The path to your data

params['data_set_name'] = 'cnn'' dataset name, appended to saved checkpoint and output.

params['use_BERT'] = True/False use Bert embedding or not, using Bert embedding takes longer for preprocessing.

params['BERT_Model_Path'] = '../pytorch-pretrained-BERT/bert_models/uncased_L-12_H-768_A-12/' The path to the bert model

params['BERT_embedding_size'] = 768 The size of bert embeddings

params['BERT_layers'] = [-1] or [-1, -2] or [-1, -2, -3], etc.. The indcies of bert layers to be used, where a word representation is the concatination of these layers.

params['vocab_size'] = 70000 The maximum size of vocab to use.

params['use_back_translation'] = False use back-translation or not

params['back_translation_file'] = None back-translated file path

params['Global_max_sequence_length'] = 25 maximum number of tokens to keep in a sequence, longer sequences will be truncated and shorter ones will be padded

params['Global_max_num_sentences'] = 20 maximum number of sentences in a comment, longer comments will be truncated.

params['use_external_vocab'] = False use external vocab to encode the data ? if False the data will be encodeded using the vocab extracted during the preprocessing.

params['external_vocab_file'] = './checkpoint/forum_vocab.pickle' external vocab file path.

params['encoding_batch_size'] = 64 The batch size used in encoding data, only used when using Bert for encoding.

params['data_split_size'] = 15000 The size of data chunk to be processed at a time, if data is larger than params['data_split_size'] samples, then preprocessed data will be split into multiple pieces.

params['device'] = torch.device('cuda' if torch.cuda.is_available() else 'cpu') use cpu or cuda

###b- Train a model To train a model using the preprocessed data you can run the main.py. Make sure to adjust the parameters to your preferences.

params['DATA_Path'] =

params['data_set_name'] = 'cnn'

params['use_coattention'] = False use co-attention or not.

params['use_BERT'] = False use BERT embeddings or not.

params['BERT_embedding_size'] = 768

params['BERT_layers'] = [-1]

params['embedding_size'] = 64 Embedding size, if BERT not used.

params['hidden_size'] = 128 Hidden size for sentence and document RNNs

params['batch_size'] = 8 The batch size for training.

params['lr'] = 0.001 Learning rate.

params['vocab_size'] = 70000 max vocab size.

params['Global_max_sequence_length'] = 25, params['Global_max_num_sentences'] = 20 Use the same as the ones used in preprocessing

params['num_epochs'] = 50 Num of training epoch

params['start_epoch'] = 0 The starting epoch index, should be helpful to avoid overwriting saved checkpoints.

params['write_summarizes'] = True

params['output_dir'] = './output/'

params['save_model'] = True

params['save_model_path'] = './checkpoint/models/'

params['load_model'] = True

params['reinit_embeddings'] = True

params['load_model_path'] = './checkpoint/bilstm_model_cnn_19.pkl'

params['device'] = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

params['task'] = 'Train' ### Train, Test

params['write_post_fix'] = '0'

params['tune_postfix'] = '_tune_guf'

params['gradual_unfreezing'] = True

###c- use a pretrained model for Tuning To tune, run main.py after setting the following parameters.

params['load_model'] = True Set for True to load a checkpoint.

params['reinit_embeddings'] = True True if the embedding layer needs to be reinitialized, or False to use the same embedding layer used while training.

params['load_model_path'] = './checkpoint/bilstm_model_cnn_19.pkl' The checkpoint to load for tuning

params['task'] = 'Train' ### Train, Test set True to continue training

params['tune_postfix'] = '_tune_guf' a postfix to add for checkpoint saving and output files.

params['gradual_unfreezing'] = True use gradual unfreezing while tuning.

###d- Use a checkpoint for Testing