Skip to content

tokestermw/spacy_kenlm

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

spacy_kenlm: KenLM extension for spaCy 2.0

This package adds kenLM support as a spaCy 2.0 extension.

Usage

Train a kenLM language model first (or use the test model from test.arpa).

Add the spaCyKenLM to the spaCy pipeline to return scores.

import spacy
from spacy_kenlm import spaCyKenLM

nlp = spacy.load('en_core_web_sm')

spacy_kenlm = spaCyKenLM()  # default model from test.arpa

nlp.add_pipe(spacy_kenlm)

doc = nlp('How are you?')

# doc score
doc._.kenlm_score

# span score
doc[:2]._.kenlm_score

# token score
doc[2]._.kenlm_score

Installation

Install from the pip package.

pip install spacy_kenlm