Skip to content
Plugin to integrate Learning to Rank (aka machine learning for better relevance) with Elasticsearch
Java Python
Branch: master
Clone or download

Latest commit

softwaredoug Fixes #286 - Add feature normalization as part of model definition (#292
)

* Incomplete pass at parsing out feature norms...

* Rearrange parsing to factory, begin adding tests

* First xcontent parsing test

* Add min/max normalizer

* Reorganize to add toXContent

* Fix remaining tests; change to float...

* push functionality into StoredFeatureNormalizerSet

* Remove FeatureNormalizer for existing Normalizer interface

* Add equals and hashCode

* Pass ftr norms explicitly to RankerQuery

* Add random feature norm to tests

* More random testing

* Add FtrNorms to StoredLtrQueryBuilderTest

* Finish getting sltr tests to pass

* Push normalization to a specialized ranker

* Add failing feature normalization test

* Hint which features to normalize

* Reorg / more randomness to ftr norm rank tests

* Remove dumb whitespace change

* Java docs for FeatureNormalizerSet

* Correct minmax norm based on research

* Remove whitespace change

* Remove misleading comment

* Javadocs

* Checkstyle issue

* Remove meaningless whitespace change

* Add failing model storage test

* Fix serialization issue of ftr norm defs

* Serialization test for StoredLtrModel

* Low hanging code review comments from @nomoa

* Enforce min / max of 0

* Assume empty feature norms if no bytes avail

* checkstyle

* Deserialize stream w/o ftr norms

* Fix integ test

* checkstyle errors

* Wrap Normalized features instead of new ranker

* Wrap Normalized features instead of new ranker

* checkstyle

* throw if already set

* throw if already set

* Actuall return NormalizedFeature

* Remove StorableElement from FtrNormDefn

* checkstyle

* Remove unneeded type

* Use Elasticsearch versions for binary stream

* Revamp parser to pass featureName to constructor

* Upgrade to 7.6.2; use this as cutoff for ftr norms

* clean up normalized feature/query

* use instanceof

* Checkstyle

* scorers can be null if nothing to be scored

* Checkstyle

* Failing test normalizing of missing vals

* Change to feature normalizing ranker

* Change to lambda

* Reorganize ltr query tests to examine normalization

* Set missing feature values to 0

* Better variable naming

* Review comments

* Change to ftr norm size

* checkstyle

* Use masters gradle wrappers

* this time w/ gradle.bat

* Add feature normalization info to docs

* Set ftr norm binary decode/encode to on/after 7.7
Latest commit 61df683 May 20, 2020

Files

Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
demo Re-enable the download of tmdb which was commented Jul 17, 2019
docs Fixes #286 - Add feature normalization as part of model definition (#292 May 20, 2020
gradle/wrapper Upgrade to Elasticsearch 7.7 (#306) May 18, 2020
licenses Refactored expression integration Aug 3, 2017
sample_models Produce sample model scripts for each supported model Mar 9, 2017
scripts Rename scripts to include xgboost Jan 21, 2020
src Fixes #286 - Add feature normalization as part of model definition (#292 May 20, 2020
.gitignore X-Pack support (#126) Feb 8, 2018
.travis.yml Upgrade to Elasticsearch 7.7 (#306) May 18, 2020
CONTRIBUTING.md Update CONTRIBUTING.md May 14, 2020
KNOWN_ISSUES.md Add a Known Issues section May 16, 2018
LICENSE.txt Adds Lucene learning to rank query; begins test Dec 25, 2016
NOTICE.txt Add required notice Sep 30, 2017
README.md Update README.md Apr 28, 2020
build.gradle Upgrade to Elasticsearch 7.7 (#306) May 18, 2020
gradlew Elastic 7.6 port (#279) Feb 27, 2020
gradlew.bat Upgrade to Elasticsearch 7.7 (#306) May 18, 2020
settings.gradle Upgrade to elasticsearch 7.3.0 Aug 23, 2019

README.md

Build Status

The Elasticsearch Learning to Rank plugin uses machine learning to improve search relevance ranking. It's powering search at places like Wikimedia Foundation and Snagajob!

What this plugin does...

This plugin:

  • Allows you to store features (Elasticsearch query templates) in Elasticsearch
  • Logs features scores (relevance scores) to create a training set for offline model development
  • Stores linear, xgboost, or ranklib ranking models in Elasticsearch that use features you've stored
  • Ranks search results using a stored model

Where's the docs?

We recommend taking time to read the docs. There's quite a bit of detailed information about learning to rank basics and how this plugin can ease learning to rank development.

You can also participate in regular 4 hour trainings on Elasticsearch Learning to Rank, which support the free work done on this plugin.

I want to jump in!

If you want to just jump in, go straight to the demo. The demo uses Ranklib, a relatively straightforward Java Learning to Rank library, to train models. Follow the directions in the demo README, edit code, and have fun!

Installing

See the full list of prebuilt versions and select the version that matches your Elasticsearch version. If you don't see a version available, see the link below for building or file a request via issues.

To install, you'd run a command like this but replacing with the appropriate prebuilt version zip:

./bin/elasticsearch-plugin install http://es-learn-to-rank.labs.o19s.com/ltr-1.0.0-es6.1.2.zip

(It's expected you'll confirm some security exceptions, you can pass -b to elasticsearch-plugin to automatically install)

If you already are running Elasticsearch, don't forget to restart!

Known issues

As any other piece of software, this plugin is not exempt from issues. Please read the known issues to learn about the current issues that we are aware of. This file might include workarounds to mitigate them when possible.

Build and Deploy Locally

Notes if you want to dig into the code or build for a version there's no build for, please feel free to run the build and installation process yourself:

./gradlew clean check
./bin/elasticsearch-plugin install file:///path/to/elasticsearch-learning-to-rank/build/distributions/ltr-<LTR-VER>-es<ES-VER>.zip

How to Contribute

For more information on helping us out (we need your help!), developing with the plugin, creating docs, etc please read CONTRIBUTING.md.

Who built this?

Other Acknowledgments & Stuff To Read

You can’t perform that action at this time.