This is the repository of my bachelor thesis project.
Text segmentation is one of the methods to generate summaries for long inputs. However, previous segmentation methods ignore the concrete information of the text. Moreover, there is confusion in the training samples due to the cut-off of language models. Our work researched on how to optimize text segmentation methods to generate summaries of higher-quality.
We designed an unsupervised topic segmentation algorithm of text using pre-trained GPT2, which makes divide decision based on computing the loss for the next sentence.
We did experiments by reproducing the SUMM^N framework (Now it is opensourced) and improving it by applying our topic segmentation method.
We used PyQt to implement a summarization system.
It has following functions:
- Detect any txt file in user's device
- Preview the selected txt file and generate summary for the selected text. With pretrained summarization model.
- Calculate the rouge score if reference is provided.
Set up the environment
pip install -r requirements.txt
Enter the system folder
cd system
Unzip the model files here
python call_ui.py
Then you will see the following interfaces
Loading interface
User interface