增加基于语义的切分方式 #248
Merged
增加基于语义的切分方式 #248
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
添加了达摩院开发的语义文档切分模型damo/nlp_bert_document-segmentation_chinese-base，目前测试下来发现对小说、百科语料（和模型训练数据就是维基百科有关系）比较友好。如果有需要的话可以根据自己场景的数据训练定制化的语义切分模型，可以参考论文：https://arxiv.org/abs/2107.09278
|
非常感谢提出的贡献，目前已将这种方式定义为AliTextSplitter类，后续将提供选项让用户选择分句方式。 |
|
该类在内网环境无法加载阿里云的语义切割模型资源，有什么解决办法吗？ |
|
可以下载到本地再加载， |
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
添加了达摩院开发的语义文档切分模型damo/nlp_bert_document-segmentation_chinese-base，目前测试下来发现对小说、百科语料（和模型训练数据就是维基百科有关系）比较友好。如果有需要的话可以根据自己场景的数据训练定制化的语义切分模型，可以参考论文：https://arxiv.org/abs/2107.09278