Skip to content

Commit

Permalink
update README
Browse files Browse the repository at this point in the history
  • Loading branch information
@zhenghaoz
zhenghaoz committed Mar 5, 2021
1 parent 3d5614c commit 857901f
Showing 1 changed file with 23 additions and 142 deletions.
165 changes: 23 additions & 142 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,155 +2,43 @@ Language: English | [中文](https://github.com/zhenghaoz/gorse/blob/master/READ

# gorse: Go Recommender System Engine

<img width=160 src="https://img.sine-x.com/gorse.png"/>
<img width=160 src="https://gorse.io/zh/docs/img/gorse.png"/>

| Build | Coverage | Report | GoDoc | RTD | Demo |
|---|---|---|---|---|---|
| ![build](https://github.com/zhenghaoz/gorse/workflows/build/badge.svg) | [![codecov](https://codecov.io/gh/zhenghaoz/gorse/branch/master/graph/badge.svg)](https://codecov.io/gh/zhenghaoz/gorse) | [![Go Report Card](https://goreportcard.com/badge/github.com/zhenghaoz/gorse)](https://goreportcard.com/report/github.com/zhenghaoz/gorse) | [![GoDoc](https://godoc.org/github.com/zhenghaoz/gorse?status.svg)](https://godoc.org/github.com/zhenghaoz/gorse) | [![Documentation Status](https://readthedocs.org/projects/gorse/badge/?version=latest)](https://gorse.readthedocs.io/en/latest/?badge=latest) | [![Website](https://img.shields.io/website-up-down-green-red/https/steamlens.gorse.io.svg)](https://steamlens.gorse.io) |
| Build | Coverage | Report | GoDoc | Document |
|---|---|---|---|---|
| ![build](https://github.com/zhenghaoz/gorse/workflows/build/badge.svg) | [![codecov](https://codecov.io/gh/zhenghaoz/gorse/branch/master/graph/badge.svg)](https://codecov.io/gh/zhenghaoz/gorse) | [![Go Report Card](https://goreportcard.com/badge/github.com/zhenghaoz/gorse)](https://goreportcard.com/report/github.com/zhenghaoz/gorse) | [![GoDoc](https://godoc.org/github.com/zhenghaoz/gorse?status.svg)](https://godoc.org/github.com/zhenghaoz/gorse) | [![Build Docs](https://github.com/gorse-io/gorse-io.github.io/actions/workflows/build_docs.yml/badge.svg)](https://gorse.io/zh/docs/) |

`gorse` is an offline recommender system backend based on collaborative filtering written in Go.
Gorse is an open source recommendation system written in Go. Gorse aims to be an universal open source recommender system that can be easily introduced into a wide variety of online services. By importing items, users and interaction data into Gorse, the system will automatically train models to generate recommendations for each user. Project features are as follows.

This project is aim to provide a high performance, easy-to-use, programming language irrelevant recommender micro-service based on collaborative filtering. We could build a simple recommender system on it, or set up a more sophisticated recommender system using candidates generated by it. It features:
- Supports multi-way matching (latest, popular, collaborative filtering based on matrix factorization) and FM-based personalized ranking.
- Single node training, distributed prediction, and ability to achieve horizontal scaling in the recommendation phase.
- Provide RESTful APIs for data CRUD and recommendation requests.
- Provide CLI tools for data import and export, model tuning, and cluster status checking.

- Implements 4 ranking based recommenders.
- Supports data loading, data splitting, model training, model evaluation and model selection.
- Provides the data import/export tool, model evaluation tool and RESTful recomender server.
- Accelerates computations by SIMD instructions and multi-threading.
## User Guide

For more information:
- [Quick Start](https://gorse.io/en/docs/chapter_2.html)
- [Configuration](https://gorse.io/en/docs/ch02-01-config.html)
- [Commands](https://gorse.io/en/docs/ch02-02-command.html)
- [RESTful APIs](https://gorse.io/en/docs/ch02-03-api.html)

- Visit [GoDoc](https://godoc.org/github.com/zhenghaoz/gorse) for detailed documentation of codes.
- Visit [ReadTheDocs](https://gorse.readthedocs.io/) for tutorials, examples and usages.
- Visit [SteamLens](https://github.com/zhenghaoz/SteamLens) for a Steam games recommender system based on *gorse*.
## [Recommendation Principles](https://gorse.io/en/docs/ch01-01-principle.html)

## Install
The process of Gorse recommending items consists of two phases, **recall** and **sort**. The recall phase finds a collection of candidate items from the whole set of items for subsequent sorting. Due to the large number of items, the recommendation system is unable to perform the computational workload of sorting all items, so the recall phase mainly uses some simple strategies or models to collect the candidate items. At present, the system has implemented three types of recall methods, namely "recent popular items", "latest items" and "collaborative filtering". The sorting stage sorts the recalled items by removing duplicate items and historical items, and the sorting stage combines the items and user characteristics to make recommendations with more accuracy.

- Download from [release](https://github.com/zhenghaoz/gorse/releases).
- Build from source:
<img width=210 src="https://gorse.io/en/docs/img/dataflow.png"/>

Install Golang and run `go get`:
## [System Architecture](https://gorse.io/en/docs/ch01-02-architect.html)

```bash
$ go get github.com/zhenghaoz/gorse/...
```
Gorse is a single node training and distributed prediction recommender system. Gorse stores data in MySQL or MongoDB, with intermediate data cached in Redis. The cluster consists of a master node, multiple worker nodes, and server nodes. The master node is responsible for ranking model training, collaborative filtering model training, non-personalized item matching, configuration management, and membership management. The server node is responsible for exposing the RESTful APIs and online real-time recommendations. Worker nodes are responsible for personalized matching for each user - currently only collaborative filtering is supported. In addition, administrator can perform model tuning, data import and export, and system status checking via the CLI.

It will download all packages and build the `gorse` command line into your `$GOBIN` path.

If your CPU supports AVX2 and FMA3 instructions, use the `avx2` build tag to enable AVX2 and FMA3 instructions.

```bash
$ go get -tags='avx2' github.com/zhenghaoz/gorse/...
```

## Usage

```
gorse is an offline recommender system backend based on collaborative filtering written in Go.
Usage:
gorse [flags]
gorse [command]
Available Commands:
export-feedback Export feedback to CSV
export-items Export items to CSV
help Help about any command
import-feedback Import feedback from CSV
import-items Import items from CSV
serve Start a recommender sever
test Test a model by cross validation
version Check the version
Flags:
-h, --help help for gorse
Use "gorse [command] --help" for more information about a command.
```

### Evaluate a Recommendation Model

*gorse* provides the tool to evaluate models. We can run `gorse test -h` or check [online documents](https://gorse.readthedocs.io/en/latest/usage/cross_validation.html) to learn its usage. For example:

```bash
$ gorse test bpr --splitter k-fold --load-csv u.data --csv-sep $'\t' --eval-precision --eval-recall --eval-ndcg --eval-map --eval-mrr --n-negative 0
...
+--------------+----------+----------+----------+----------+----------+----------------------+
| | FOLD 1 | FOLD 2 | FOLD 3 | FOLD 4 | FOLD 5 | MEAN |
+--------------+----------+----------+----------+----------+----------+----------------------+
| Precision@10 | 0.321041 | 0.327128 | 0.321951 | 0.318664 | 0.317197 | 0.321196(±0.005931) |
| Recall@10 | 0.212509 | 0.213825 | 0.213336 | 0.206255 | 0.210764 | 0.211338(±0.005083) |
| NDCG@10 | 0.380665 | 0.385125 | 0.380003 | 0.369115 | 0.375538 | 0.378089(±0.008974) |
| MAP@10 | 0.122098 | 0.123345 | 0.119723 | 0.116305 | 0.119468 | 0.120188(±0.003883) |
| MRR@10 | 0.605354 | 0.601110 | 0.600359 | 0.577333 | 0.599930 | 0.596817(±0.019484) |
+--------------+----------+----------+----------+----------+----------+----------------------+
```

`u.data` is the CSV file of ratings in [MovieLens 100K](https://grouplens.org/datasets/movielens/100k/) dataset and `u.item` is the CSV file of items in [MovieLens 100K](https://grouplens.org/datasets/movielens/100k/) dataset. All CLI tools are listed in the [CLI-Tools](https://github.com/zhenghaoz/gorse/wiki/CLI-Tools) section of Wiki.

### Setup a Recommender Server

It's easy to setup a recomendation service with `gorse`.

- **Step 1**: Import feedback and items.

```bash
$ gorse import-feedback ~/.gorse/gorse.db u.data --sep $'\t' --timestamp 2
$ gorse import-items ~/.gorse/gorse.db u.item --sep '|'
```

It imports feedback and items from CSV files into the database file `~/.gorse/gorse.db`. The low level storage engine is implemented by BoltDB.

- **Step 2**: Start a server.

```bash
$ gorse serve -c config.toml
```

It loads configurations from [config.toml](https://github.com/zhenghaoz/gorse/blob/master/example/file_config/config.toml) and start a recommendation server. It may take a while to generate all recommendations. Detailed information about configuration is in the [Configuration](https://github.com/zhenghaoz/gorse/wiki/Configuration) section of Wiki. Before set hyper-parameters for the model, it is useful to test the performance of chosen hyper-parameters by the [model evaluation tool](https://github.com/zhenghaoz/gorse/wiki/CLI-Tools#cross-validation-tool).

- **Step 3**: Get recommendations.

```bash
$ curl 127.0.0.1:8080/recommends/1?number=5
```

It requests 5 recommended items for the 1-th user. The response might be:

```
[
{
"ItemId": "919",
"Popularity": 96,
"Timestamp": "1995-01-01T00:00:00Z",
"Score": 1
},
{
"ItemId": "474",
"Popularity": 194,
"Timestamp": "1963-01-01T00:00:00Z",
"Score": 0.9486470268850127
},
...
]
```

`"ItemId"` is the ID of the item and `"Score"` is the score generated by the recommendation model used to rank. See [RESTful APIs](https://github.com/zhenghaoz/gorse/wiki/RESTful-APIs) in Wiki for more information about RESTful APIs.


## Recommenders

- Cross-validation of ranking models on MovieLens 100K [[Source](https://github.com/zhenghaoz/gorse/blob/master/example/benchmark_ranking/main.go)].

| Model | Precision@10 | Recall@10 | MAP@10 | NDCG@10 | MRR@10 | Time |
| ------- | ------------------------------------- | ------------------------------- | ------------------------- | --------------------------- | ------------------------- | ------- |
| ItemPop | 0.19081 | 0.11584 | 0.05364 | 0.21785 | 0.40991 | 0:00:03 |
| KNN | 0.28584 | 0.19328 | 0.11358 | 0.34746 | 0.57766 | 0:00:41 |
| BPR | 0.32083 | 0.20906 | 0.11848 | 0.37643 | 0.59818 | 0:00:13 |
| ALS | 0.34727 | 0.23665 | 0.14550 | 0.41614 | 0.65439 | 0:00:14 |
<img width=480 src="https://gorse.io/en/docs/img/arch.png"/>

## Contributors

[![](https://sourcerer.io/fame/zhenghaoz/zhenghaoz/gorse/images/0)](https://sourcerer.io/fame/zhenghaoz/zhenghaoz/gorse/links/0)[![](https://sourcerer.io/fame/zhenghaoz/zhenghaoz/gorse/images/1)](https://sourcerer.io/fame/zhenghaoz/zhenghaoz/gorse/links/1)[![](https://sourcerer.io/fame/zhenghaoz/zhenghaoz/gorse/images/2)](https://sourcerer.io/fame/zhenghaoz/zhenghaoz/gorse/links/2)[![](https://sourcerer.io/fame/zhenghaoz/zhenghaoz/gorse/images/3)](https://sourcerer.io/fame/zhenghaoz/zhenghaoz/gorse/links/3)[![](https://sourcerer.io/fame/zhenghaoz/zhenghaoz/gorse/images/4)](https://sourcerer.io/fame/zhenghaoz/zhenghaoz/gorse/links/4)[![](https://sourcerer.io/fame/zhenghaoz/zhenghaoz/gorse/images/5)](https://sourcerer.io/fame/zhenghaoz/zhenghaoz/gorse/links/5)[![](https://sourcerer.io/fame/zhenghaoz/zhenghaoz/gorse/images/6)](https://sourcerer.io/fame/zhenghaoz/zhenghaoz/gorse/links/6)[![](https://sourcerer.io/fame/zhenghaoz/zhenghaoz/gorse/images/7)](https://sourcerer.io/fame/zhenghaoz/zhenghaoz/gorse/links/7)
<a href="https://github.com/zhenghaoz/gorse/graphs/contributors">
<img src="https://contrib.rocks/image?repo=zhenghaoz/gorse" />
</a>

Any kind of contribution is expected: report a bug, give a advice or even create a pull request.

Expand All @@ -161,10 +49,3 @@ Any kind of contribution is expected: report a bug, give a advice or even create
- [Guibing Guo's librec](https://github.com/guoguibing/librec)
- [Nicolas Hug's Surprise](https://github.com/NicolasHug/Surprise)
- [Golang Samples's gopher-vector](https://github.com/golang-samples/gopher-vector)

## Limitations

`gorse` has limitations and might not be applicable to some scenarios:

- **No Scalability**: `gorse` is a recommendation service on a single host, so it's unable to handle large data.
- **No Features**: `gorse` exploits interactions between items and users while features of items and users are ignored.

0 comments on commit 857901f

Please sign in to comment.