JapaneseTokenizer.pipe added

[KoichiYasuoka]

Description

For spacymoji with Japanese().

Types of change

To use JapaneseTokenizer.pipe as same as Tokenizer.pipe

Checklist

• I have submitted the spaCy Contributor Agreement.
• I ran the tests, and all new and existing tests passed.
• My changes don't require a change to the documentation, or if they do, I've added all required information.

[adrianeboyd]

Hi, thanks for this PR! Looking at the details, I think it would make more sense to add it to DummyTokenizer so it fixes this issue for several languages at once (Chinese, Japanese, Korean, Thai), but then you also have to handle an unimplemented __call__ method in DummyTokenizer. I think you could add:

class DummyTokenizer(object):
    def __call__(self, text):
        raise NotImplementedError

    def pipe(self, texts, **kwargs):
        for text in texts:
            yield self(text)

    ...

If you wanted you could warn about n_threads as in Tokenizer.pipe, but I don't think this is a big deal. None of the tokenizers do any batching, and nlp.pipe() just calls nlp.make_doc(), so you don't notice the missing tokenizer.pipe() in most typical use cases.

[adrianeboyd]

Thanks, this looks good!

[KoichiYasuoka]

Thank you for your suggestion, @adrianeboyd , and Thai() seems OK now with spacymoji. But, among the tests, explosion.spaCy (Test Python35Windows) failed. Umm...

[adrianeboyd]

Don't worry about the python 3.5 test, that's unrelated (and should be fixed by #6522).