Skip to content
Model for predicting categories of entities by its mentions
Jupyter Notebook Python Vue JavaScript Shell HTML
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
category_prediction
data
docker
logs
notebooks
.env
.flaskenv
.gitignore
allen_conf.json
allen_conf_test.json
allen_conf_test_prep.json
allen_conf_vocab.json
deploy_prod.sh
download_data.sh
gunicorn_config.py
readme.md
requirements.txt
run_server.py
serve.sh
wsgi.py

readme.md

Category prediction model

This repo contains AllenNLP model for prediction of Named Entity categories by its mentions.

Data

Fake data

You can generate some fake data using this Notebook

Real data (Work in progress)

Filtered OneShotWikilinks dataset with manually selected categories.

Data preparation steps

Prepare splitted data with:

!split -n l/10 --verbose ../data/fake_data_train.tsv ../data/fake_data_train.tsv_

Install

pip install -r requirements.txt

Run

Train

rm -rf ./data/vocabulary ; allennlp make-vocab -s ./data/ allen_conf_vocab.json --include-package category_prediction

allennlp train -f -s data/stats allen_conf.json --include-package category_prediction
allennlp train -f -s data/stats allen_conf.json --include-package category_prediction -o '{"trainer": {"cuda_device": 0}}'

Continue training with different params

rm -rf data/stats2/  # Clear new serialization dir
allennlp fine-tune -s data/stats2/ -c allen_conf.json -m ./data/stats/model.tar.gz --include-package category_prediction -o '{"trainer": {"cuda_device": 0}, "iterator": {"base_iterator": {"batch_size": 64}}}'

Validate

allennlp evaluate ./data/stats/model.tar.gz ./data/fake_data_test.tsv --include-package category_prediction

Server

Debug

MODEL=./data/trained_models/6th_augmented/model.tar.gz python run_server.py

Prod

gunicorn -c gunicorn_config.py wsgi:application

Docker

Build

cd docker
docker build --tag mention .

Run with passing pyenv into container

docker run --rm --restart unless-stopped -v $HOME:$HOME -p 8000:8000 \
        -v $HOME/.pyenv:/root/.pyenv \ 
        -e ENV_PATH=$HOME/virtualenv/path \
        -e APP_PATH=$HOME/project/root/path mention

GCE related notes

Fix 100% GPU utilization

sudo nvidia-smi -pm 1
You can’t perform that action at this time.