Skip to content

Commit

Permalink
fix: #27
Browse files Browse the repository at this point in the history
  • Loading branch information
severinsimmler committed Nov 23, 2018
1 parent d285f9e commit d97ee7e
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion src/cophi/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ def construct_ngrams(tokens, n=2, sep=" "):
return (sep.join(ngram)
for ngram in zip(*(itertools.islice(i, token, None)
for token, i in enumerate(itertools.tee(tokens,
2)))))
n)))))


def find_tokens(document, token_pattern=r"\p{L}+\p{P}?\p{L}+", maximum=None):
Expand Down

0 comments on commit d97ee7e

Please sign in to comment.