New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GH-1768: turn on tokenizer by default in Sentence object #1806

Merged

alanakbik merged 1 commit into master from GH-1768-tokenizer-default

Aug 13, 2020

Collaborator

alanakbik commented Aug 13, 2020

The Sentence object now executes tokenization by default, e.g.:

# Tokenizes by default
sentence = Sentence("I love Berlin.")
print(sentence)

# i.e. this is equivalent to
sentence = Sentence("I love Berlin.", use_tokenizer=False)
print(sentence)

# i.e. if you don't want to use tokenization, set it to False
sentence = Sentence("I love Berlin.", use_tokenizer=False)
print(sentence)

Closes #1768


          GH-1768: turn on tokenizer by default in Sentence object

c02aef4

alanakbik merged commit 040d386 into master

alanakbik deleted the GH-1768-tokenizer-default branch

August 13, 2020 13:58

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment