Skip to content
This repository has been archived by the owner. It is now read-only.
Permalink
Browse files
fix(preprocess): fix offset in sentence splitter
  • Loading branch information
hanhxiao committed Sep 2, 2019
1 parent d21a6a1 commit d94329b369e3181077efada65212814dd5e1a5d1
Showing with 1 addition and 1 deletion.
  1. +1 −1 gnes/preprocessor/text/split.py
@@ -53,6 +53,6 @@ def apply(self, doc: 'gnes_pb2.Document') -> None:
c = doc.chunks.add()
c.doc_id = doc.doc_id
c.text = f[:self.max_sent_len]
c.offset_1d = ci
c.offset = ci
c.weight = len(c.text) / len(doc.raw_text)
c.offset_nd.extend([s, e])

0 comments on commit d94329b

Please sign in to comment.