Skip to content
Permalink
Browse files

fix(preprocess): fix offset in sentence splitter

  • Loading branch information...
hanxiao committed Sep 2, 2019
1 parent d21a6a1 commit d94329b369e3181077efada65212814dd5e1a5d1
Showing with 1 addition and 1 deletion.
  1. +1 −1 gnes/preprocessor/text/split.py
@@ -53,6 +53,6 @@ def apply(self, doc: 'gnes_pb2.Document') -> None:
c = doc.chunks.add()
c.doc_id = doc.doc_id
c.text = f[:self.max_sent_len]
c.offset_1d = ci
c.offset = ci
c.weight = len(c.text) / len(doc.raw_text)
c.offset_nd.extend([s, e])

0 comments on commit d94329b

Please sign in to comment.
You can’t perform that action at this time.