Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Found a bug in file dict_util.go #173

Open
ntcat opened this issue Jul 28, 2023 · 1 comment
Open

Found a bug in file dict_util.go #173

ntcat opened this issue Jul 28, 2023 · 1 comment

Comments

@ntcat
Copy link

ntcat commented Jul 28, 2023

in this func:
func (seg *Segmenter) Reader(reader *bufio.Reader, files ...string) error

those code lines:
if fsErr != nil {
if fsErr == io.EOF {
...
}
must put after :
...
seg.Dict.AddToken(token)
Otherwise,last line of dictory file will be missed.exclude the last line is empty.

@ntcat
Copy link
Author

ntcat commented Jul 28, 2023

and, change this line:

		if freq == 0.0  {
			continue
		}

to:

		if freq == 0.0 && fsErr != io.EOF {
			continue
		}

otherwise, will met dead loop, when dictory file is empty.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant