Approximate nearest neighbor search server and dynamic index written in Golang.
Switch branches/tags
Nothing to show
Clone or download
Permalink
Failed to load latest commit information.
cmd Added health check API. Jul 11, 2017
rpmbuild gannoy-server is removed from command currently. Jun 25, 2017
.gitignore gannoy-server is removed from command currently. Jun 25, 2017
.travis.yml Fixed CI. remove go 1.7. why... May 2, 2017
Dockerfile.rpmbuild gannoy-server is removed from command currently. Jun 25, 2017
LICENSE Added a LICENSE. Jun 28, 2017
README.md Fixed API reference in README. Jun 28, 2017
const.go Implemented heap sort method for partial sort. Apr 27, 2017
converter.go converter now builds tree in bulk. Aug 25, 2017
distance.go Fixed a bug that happen panic when do makeTree. May 3, 2017
distance_test.go Added GannoyIndex AddItem and RemoveItem tests. May 4, 2017
docker-compose.yml Added building gannoy rpm system. Jun 20, 2017
file.go Reduce allocation memory when File.Create and Update. Jun 19, 2017
file_test.go Fixed failed test. Jun 14, 2017
free.go Added maps tests. May 2, 2017
free_test.go Added maps tests. May 2, 2017
gannoy.go converter now builds tree in bulk. Aug 25, 2017
gannoy_test.go Added GannoyIndex GetNnsByKey tests. May 4, 2017
lock.go normalize kernel version to semantic versionning Jul 24, 2017
lock_test.go normalize kernel version to semantic versionning Jul 24, 2017
maps.go Added isExist method to Maps. Jun 28, 2017
maps_test.go Added isExist method to Maps. Jun 28, 2017
meta.go Fixed a bug that happen panic when tree count is large. May 9, 2017
meta_test.go Added meta tests. May 2, 2017
node.go getNodeByKey receiver be called as pointer for avoiding map race cond… May 8, 2017
node_test.go Added node tests. May 2, 2017
queue.go Fixed wrong priority. Apr 27, 2017
random.go Rename to gannoy Apr 22, 2017
sort.go Use partial sort. Apr 27, 2017
sort_test.go Added sort tests. May 2, 2017
storage.go Added error handling for finding node. May 1, 2017
util.go Added distance tests. May 2, 2017
version.go Start versioning. Jun 25, 2017

README.md

Gannoy Build Status

Approximate nearest neighbor search server and dynamic index written in Golang. Gannoy is inspired by spotify/annoy and provides a dynamic database and API server.

Quick start

# Create database
$ gannoy create -d 100 DATABASE_NAME
# Start server
$ gannoy-db

Regiter features using gannoy API.

$ curl 'http://localhost:1323/databases/DATABASE_NAME/features/KEY' \
       -H "Content-type: application/json" \
       -X PUT \
       -d '{"features": [1.0, 0.5, 0.2, ...]}'

Search similar items.

$ curl 'http://localhost:1323/search?database=DATABASE_NAME&key=KEY'
[10, 23, 2, 20, 300, 45, 11, 8, 39, 88]

See also gannoy create --help or gannoy-db --help.

Install

$ go get github.com/monochromegane/gannoy/...

Recommendation environment is Linux (kernel >= 3.15.0).

Gannoy uses fcntl system call and F_OFD_SETLKW command to lock the necessary minimum range from multiple goroutines for speeding up.

API

GET /search

Search approximate nearest neighbor items.

query parameters

key value
database Search for similar items from this database name.
key Search for similar items from this key's feature.
limit Maxium number of result.

Response

  • Response 200 (application/json)
    • return list of item keys.
  • Response 404 (no content)
    • return no content if you specify not found database or key.

POST /databases/:database/features

Register features using a specified key.

URI parameters

key value
database Create item in this database name.

JSON parameters

key value
key Create item using this key.
features List of feature value.

Note: KEY must be integer.

Response

  • Response 200 (no content)
    • return no content.
  • Response 422 (no content)
    • return no content if you specify not found database or unprocessable parameter.

PUT /databases/:database/features/:key

Register or update features using a specified key.

URI parameters

key value
database Create or update item in this database name.
key Create or update item using this key.

Note: KEY must be integer.

JSON parameters

key value
features List of feature value.

Response

  • Response 200 (no content)
    • return no content.
  • Response 422 (no content)
    • return no content if you specify not found database or unprocessable parameter.

DELETE /databases/:database/features/:key

Register or update features using a specified key.

URI parameters

key value
database Delete item from this database name.
key Delete item using this key.

Response

  • Response 200 (no content)
    • return no content.
  • Response 422 (no content)
    • return no content if you specify not found database or unprocessable parameter.

Run with Server::Starter

Gannoy can run with Server::Starter for supporting graceful restart.

$ start_server --port 8080 --pid-file app.pid -- gannoy-db -s # gannoy-db listen Server::Starter port if you pass s option.

Configuration

Gannoy can load option from configuration file.

If you prepare a configuration file named gannoy.toml like the following:

data-dir = "/var/lib/gannoy"
log-dir  = "/var/log/gannoy"
lock-dir = "/var/run/gannoy"
server-starter = true

You can specify the name with c option.

$ gannoy-db -c gannoy.toml

Note: A priority of flag is command-line flag > configration file > flag default value. See also monochromegane/conflag.

Building rpm

Note: Requirements are Docker and docker-compose.

$ docker-compose build gannoy-rpmbuild
$ docker-compose run gannoy-rpmbuild

Result (gannoy-x.x.x-x.x86_64.rpm) is put in rpmbuild/RPMS/x86_64 directory on host.

You can install the rpm and running gannoy-db process on CentOS.

$ sudo rpm -ivh gannoy-x.x.x-x.x86_64.rpm
$ sudo systemctl start gannoy-db

Data migration from annoy

You can migrate spotify/annoy database file.

$ gannoy-converter -d 100 ANNOY_FILE DATABASE_NAME

License

MIT

Author

monochromegane