Advance Natural Language Processing

Tasks for Advance Natural Language Processing Course at ITMO University.

Index

1. Data cleaning and duplicaets detection
2. Quora duplicates detection
3. Attention mechanism and Bert
4. Fine tuning and dividing documents on 10 topic categories
5. Topic Modeling

Data cleaning and duplicaets detection

This file contains :

• Data Cleaning:

1. Remove non-english words
2. Remove html-tags (try to do it with regular expression, or play with beautifulsoap library)
3. Apply lemmatization / stemming
4. Remove stop-words

• Duplicates detection using LSH

Quora duplicates detection

The task in this file isbuild an LSTM-based siamese homework and search for the duplicates in quora question pairs dataset.

Attention mechanism and Bert

The task in this file is to create attention mechanism with the numpy tool. And use of pre-trained models for text processing.

Fine tuning

The task in this file is to divide documents on 10 topic categories using Huggingface Datasets library.

Topic Modeling

In this file I've applied topic modeling with NMF (using sklearn.decomposition.NMF) and topic modeling with LDA (using gensim implementation) in addition to applying the following two quality fuctions: coherence, and normalized PMI.