Skip to content

Latest commit

 

History

History
114 lines (87 loc) · 4.12 KB

README.md

File metadata and controls

114 lines (87 loc) · 4.12 KB
Raw

NLP Project

Using Weak Supervision to perform Named Entity Recognition (NER) and Offensive Language Identification (OLI) on r/Singapore, EDMW and OLID comments.

Quick start:

pip install requirements.txt in NER/OLID folder.

NER

Workflow to obtain fine-tuned NER model using weak supervision/gold labels:

To scrape comments for NER, create a config.py file in the NER folder with Reddit API info. (Refer to NER/example_config.py)
Adjust parameters from NER/NER_v7/main_config.py

  • NER/NER_v7/make_spacy_weak_supervision.py (pipeline to scrape and preprocess comments, apply and resolve aggregated labelling functions, serialize .spacy binary file for fine-tuning)
  • NER/NER_cli_train.ipynb (jupyter notebook to fine-tune model in command line using serialized file)
  • NER/NER_v7/evaluate.py (test performance on NER task from best model saved to disk after fine-tuning)

Performance

Tested on 500 r/Singapore comments:

Fine-tuned model Precision Recall Micro F1
Weak Supervision 80.1 81.2 80.6
Base spaCy transformer 84.5 78.9 81.6
Gold labels 85.3 83.7 84.5

OLI

Workflow to obtain fine-tuned OLID model using weak supervision/gold labels:

  • OLID/ws_v0/download_transformer_pipeline.py (download hugging face transformers required for weak supervision pipeline)
  • OLID/ws_v0/make_spacy.py (pipeline to preprocess comments, apply and resolve aggregated labelling functions, serialize .spacy binary file for fine-tuning)
  • NER/NER_cli_train.ipynb (jupyter notebook to fine-tune model in command line using serialized file)
  • NER/NER_v7/*_analysis.py (test OLID performance on chosen dataset from best model saved to disk after fine-tuning)

Performance

Tested on OLID dataset:

Task A
Fine-tuned model F1 OFF (240) F1 NOT (620) Macro F1 (840)
Weak Supervision 72.5 90.0 81.2
Weak Supervision (Small) 69.6 87.9 78.7
Gold labels 71.9 89.1 80.5
Gold labels (Small) 65.2 86.1 75.6
CNN 70.0 90.0 80.0
Task B
Fine-tuned model F1 TIN (213) F1 UNT (27) Macro F1 (240)
Weak Supervision 91.4 28.6 60.0
Weak Supervision (Small) 91.4 28.6 60.0
Gold labels 80.0 30.2 55.1
Gold labels (Small) 94.0 24.2 59.1
CNN 92.0 42.0 67.0
Task C
Fine-tuned model F1 IND (100) F1 GRP (78) F1 OTH (35) Macro F1 (213)
Weak Supervision 77.3 61.9 25.9 55.0
Weak Supervision (Small) 74.7 56.2 17.3 49.4
Gold labels 72.6 46.7 27.7 49.0
Gold labels (Small) 71.4 43.4 17.8 44.2
CNN 75.0 67.0 0.0 47.0

Tested on 1000 EDMW comments:

Task A
Fine-tuned model F1 OFF (116) F1 NOT (84) Macro F1 (200)
Weak Supervision labels 87.3 81.3 84.3
Gold labels 87.7 84.4 86.1
Task B
Fine-tuned model F1 TIN (91) F1 UNT (25) Macro F1 (116)
Weak Supervision labels 86.8 46.5 66.6
Gold labels 88.5 60.0 74.3
Task C
Fine-tuned model F1 IND (36) F1 GRP (44) F1 OTH (11) Macro F1 (91)
Weak Supervision labels 62.7 39.3 16.7 39.6
Gold labels 68.4 79.1 0.0 49.2

Tested on 1000 r/Singapore comments:

Task A
Fine-tuned model F1 OFF (62) F1 NOT (138) Macro F1 (200)
Weak Supervision 65.5 87.2 76.3
Gold labels 63.7 86.4 75.1
Task B
Fine-tuned model F1 TIN (52) F1 UNT (10) Macro F1 (62)
Weak Supervision labels 91.1 66.7 78.9
Gold labels 86.0 48.0 67.0
Task C
Fine-tuned model F1 IND (18) F1 GRP (19) F1 OTH (15) Macro F1 (52)
Weak Supervision labels 64.0 46.7 25.0 45.2
Gold labels 73.7 48.5 42.4 54.9

Web Application

Workflow:

  • Open command prompt in /flask and key in the command npm run dev
  • Run flask_axios_api.py
  • Open 127.0.0.1:1234 or localhost:1234 in a web browser to access the web application