Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

自定义词典是如何起作用的？ #50

Closed
dylgithub opened this issue Feb 5, 2019 · 2 comments
Closed

自定义词典是如何起作用的？ #50

dylgithub opened this issue Feb 5, 2019 · 2 comments

Comments

@dylgithub
Copy link

您好，请问如何确保加入自定义词典中的词必然会被划分成词的？也就是说具体是如何对自定义词典处理的？
@rockyzhengwu
Copy link
Owner

先得到模型分词结果，然后加入词典做机械分词，理论上并不能保证词典中的词会被分词，实际中给词典的词特别大的权重就可以实现词典中的词一定会被切分

@dylgithub
Copy link
Author

您好，由于个人能力有限所以也没看源码，请问您说的加入词典做机械分词的意思是基于模型的分词结果，把词典中的词在分词结果中再强制结合成词吗？jieba中是把词典的权重取log值作为边的权重，因为jieba是基于词典的分词工具，而foolnltk是基于神经网络的分词工具，这里的权重具体怎么使用的？望告知，谢谢！

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rockyzhengwu @dylgithub