Skip to content
Search relevance evaluation toolkit
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
quaerite-analysis
quaerite-cli merge es branch May 8, 2019
quaerite-connectors merge es branch May 8, 2019
quaerite-core add es examples; make sure they work Apr 29, 2019
quaerite-examples merge es branch May 8, 2019
quaerite-parent back to work on 1.0.0-SNAPSHOT... Mar 22, 2019
quaerite-solr-tools merge es branch May 8, 2019
.gitignore initial commit Mar 1, 2019
LICENSE added MITRE copyright to main license file; added description for the… Mar 1, 2019
README.md update readme to reflect elastic branch merge May 8, 2019
pom.xml add elevate tools. This closes issue #38. May 2, 2019

README.md

Welcome to Quaerite

Background and Goals

This project includes tools to help evaluate relevance ranking. This code has been tested with Solr 4.x, 7.x and 8.x, and ES 6.x and 7.x.

This project is not intended to compete with existing relevance evaluation tools, such as Splainer, Quepid, Rated Ranking Evaluator, or Luigi's Box. Rather, this project was developed for use cases not currently covered by open source software packages. The author encourages collaboration among these projects.

NOTE: This project is under construction and is quite dynamic.
There will be breaking changes before the first major release.

While the name of this project may change in the future, we selected quaerite -- Latin imperative "seek", root of English "query" -- to allude not only to the challenges of creating queries, but also to the challenges of tuning search engines. One may spend a not insignificant amount of time tuning countless parameters. In the end, we hope that invenietis with slightly less effort than without this project. For the pronunciation, see this link.

Similarities and Differences between the Genetic Algorithm (GA) in Quaerite and Learning to Rank

In the research literature, the application of a GA or Genetic Programming (GP) is one method for learning to rank (see, e.g. Andrew Trotman on GP).

However, for integrators and developers who work in the Lucene ecosystem, "Learning to Rank" (LTR) connotes a specific methodology/module initially added to Apache Solr by Bloomberg and then offered as a plugin for ElasticSearch by Doug Turnbull and colleagues at OpenSource Connections, Wikimedia Foundation and Snagajob Engineering. In the following, I use LTR to refer to this Lucene-ecosystem-specific module and methodology.

In no way do I see this implementation of GA as a competitor to LTR; rather, it is another tool that might help complement LTR and/or other tuning methodologies.

Similarities

  • All of the basic requirements for quality search must first be met -- analyzer chains must be well designed for the data, the underlying data in the index should be accurate, well organized and well curated
  • There must be sufficient, high quality, accurate and representative ground truth judgments for training and testing
  • Machine learning can only do so much -- further tuning and/or adding new methods of enrichment may be required

Differences

  • In practice, LTR is designed to perform more costly calculations as a re-ranking step...that is, after the search engine has returned the best n documents, LTR is typically applied to carry out more costly calculations on this smaller subset of documents to re-rank the results based on the models built offline. The goal of this implementation of GA (and the other tools in Quaerite) is to help tune the parameters used in the initial search system's ranking, NOT as part of a secondary reranking.
  • Bloomberg, OpenSource Connections, Wikimedia and Snagajob have spent quite a bit of time and effort developing and integrating these modules to make them easy to use. This toolkit has been developed with far fewer resources for use initially by one relevance engineer...there are areas for improvement.

Current Status

As of this writing, Quaerite allows for experimentation with the following parameters: bq, qf, pf, pf2, pf3, solr url (so that you can run experiments against different cores and/or different versions of Solr), customHandler (so that you can compare different customized handlers), tie.

Before the BETA release, we'll add: q.op, mm, bf and a more parameterizable version of bq.

Getting Started

See the quaerite-examples module and its README.

Releases

Road Map

High priorities

Planned Releases

  • 1.0.0-beta1 May, 2019

Related Open Source Projects

License (see also LICENSE.txt)

Copyright (c) 2019, The MITRE Corporation. All rights reserved.

Approved for Public Release; Distribution Unlimited. Case Number 18-3138.

Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

You can’t perform that action at this time.