JapaneseTokenizer.pipe added (#6515)

* JapaneseTokenizer.pipe added For [spacymoji](https://spacy.io/universe/project/spacymoji) with `Japanese()`. * DummyTokenizer.pipe added instead
explosion · Dec 8, 2020 · 0afb54a · 0afb54a
1 parent df4891b
commit 0afb54a
Showing 1 changed file with 7 additions and 0 deletions.
diff --git a/spacy/util.py b/spacy/util.py
@@ -838,6 +838,13 @@ def update(self, other):
 
 
 class DummyTokenizer(object):
+    def __call__(self, text):
+        raise NotImplementedError
+
+    def pipe(self, texts, **kwargs):
+        for text in texts:
+            yield self(text)
+
     # add dummy methods for to_bytes, from_bytes, to_disk and from_disk to
     # allow serialization (see #1557)
     def to_bytes(self, **kwargs):