Go efficient text segmentation; support english, chinese, japanese and other. Go 语言高性能分词
Switch branches/tags
Nothing to show
Clone or download

README.md

gse

Go efficient text segmentation; support english, chinese, japanese and other.

CircleCI Status codecov Build Status Go Report Card GoDoc GitHub release Join the chat at https://gitter.im/go-ego/ego

简体中文

Dictionary with double array trie (Double-Array Trie) to achieve, Sender algorithm is the shortest path based on word frequency plus dynamic programming.

Support common and search engine two participle mode, support user dictionary, POS tagging, run JSON RPC service.

Support HMM cut text use Viterbi algorithm

Text Segmentation speed single thread 9MB/s,goroutines concurrent 42MB/s (8 nuclear Macbook Pro).

Install / update

go get -u github.com/go-ego/gse

Build-tools

go get -u github.com/go-ego/re 

re gse

To create a new gse application

$ re gse my-gse

re run

To run the application we just created, you can navigate to the application folder and execute:

$ cd my-gse && re run

Use

package main

import (
	"fmt"

	"github.com/go-ego/gse"
)

func main() {
	var seg gse.Segmenter
	// Loading the default dictionary
	seg.LoadDict()
	// Load the dictionary
	// seg.LoadDict("your gopath"+"/src/github.com/go-ego/gse/data/dict/dictionary.txt")

	// Text Segmentation
	text := []byte("你好世界, Hello world.")
	fmt.Println(seg.String(text, true)) 

	segments := seg.Segment(text)
  
	// Handle word segmentation results
	// Support for normal mode and search mode two participle,
	// see the comments in the code ToString function.
	// The search mode is mainly used to provide search engines 
	// with as many keywords as possible
	fmt.Println(gse.ToString(segments, true)) 
}

Look at an custom dictionary example

package main

import (
	"fmt"

	"github.com/go-ego/gse"
)

func main() {
	var seg gse.Segmenter
	seg.LoadDict("zh,testdata/test_dict.txt,testdata/test_dict1.txt")

	text1 := []byte("你好世界, Hello world")
	fmt.Println(seg.String(text1, true)) 

	segments := seg.Segment(text1)
	fmt.Println(gse.ToString(segments))
}

Look at an Chinese example

Look at an Japanese example

Authors

License

Gse is primarily distributed under the terms of both the MIT license and the Apache License (Version 2.0), base on sego.