Skip to content

Keep original unigrams in tokens_compound() #2399

Description

@koheiw

It is possible to keep unigrams in tokens_ngrams() by setting n = 1 but it is not in tokens_compound(). Keeping unigrams makes it easy to apply pattern matching in the downstream. How about adding an option like keep_unigram?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions