New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Search enhancement: pinned queries #44345

Merged

markharwood merged 2 commits into elastic:master from markharwood:fix/44074

Aug 16, 2019

Contributor

markharwood commented Jul 15, 2019

New query type allows selected documents to be promoted above “any organic” search results.

This is the first feature in a new module search-business-rules which will house licensed (non OSS) logic for rewriting queries according to business rules.
The new PinnedQueryBuilder class offers a new pinned query in the DSL that takes an array of promoted IDs and an “organic” query and ensures the documents with the promoted IDs rank higher than those matched purely by the organic query.

This first commit contains the core query and tests - docs to follow

Closes #44074

markharwood added >enhancement :Search Relevance/Ranking v8.0.0 v7.4.0 labels

markharwood requested a review from jpountz

July 15, 2019 13:19

markharwood self-assigned this

Collaborator

elasticmachine commented Jul 15, 2019

Pinging @elastic/es-search

jpountz reviewed

View reviewed changes

...ck/plugin/search-business-rules/src/main/java/org/apache/lucene/search/CappedScoreQuery.java Outdated Show resolved Hide resolved

...ck/plugin/search-business-rules/src/main/java/org/apache/lucene/search/CappedScoreQuery.java Outdated Show resolved Hide resolved

...ck/plugin/search-business-rules/src/main/java/org/apache/lucene/search/CappedScoreQuery.java Show resolved Hide resolved

...ck/plugin/search-business-rules/src/main/java/org/apache/lucene/search/CappedScoreQuery.java Outdated Show resolved Hide resolved

...k/plugin/search-business-rules/src/main/java/org/apache/lucene/search/CappedScoreWeight.java Show resolved Hide resolved

...ness-rules/src/main/java/org/elasticsearch/xpack/searchbusinessrules/PinnedQueryBuilder.java Show resolved Hide resolved

...ness-rules/src/main/java/org/elasticsearch/xpack/searchbusinessrules/PinnedQueryBuilder.java Outdated Show resolved Hide resolved

...ness-rules/src/main/java/org/elasticsearch/xpack/searchbusinessrules/PinnedQueryBuilder.java Outdated Show resolved Hide resolved

...ness-rules/src/main/java/org/elasticsearch/xpack/searchbusinessrules/PinnedQueryBuilder.java Outdated Show resolved Hide resolved

...ness-rules/src/main/java/org/elasticsearch/xpack/searchbusinessrules/PinnedQueryBuilder.java Outdated Show resolved Hide resolved

jtibshirani reviewed

View reviewed changes

...ck/plugin/search-business-rules/src/main/java/org/apache/lucene/search/CappedScoreQuery.java Show resolved Hide resolved

jpountz added the release highlight label

jpountz reviewed

View reviewed changes

x-pack/plugin/search-business-rules/build.gradle

    
                  classname 'org.elasticsearch.xpack.searchbusinessrules.SearchBusinessRules'

                  extendedPlugins = ['x-pack-core']

              }

              archivesBaseName = 'x-pack-searchbusinessrules'

Contributor

jpountz Jul 17, 2019

why do we have this?

Contributor Author

markharwood Jul 17, 2019

It's based on a cut-and-paste of vectors' build.gradle with modifications where it seemed appropriate.

...ck/plugin/search-business-rules/src/main/java/org/apache/lucene/search/CappedScoreQuery.java Outdated Show resolved Hide resolved

...ck/plugin/search-business-rules/src/main/java/org/apache/lucene/search/CappedScoreQuery.java Show resolved Hide resolved

...ck/plugin/search-business-rules/src/main/java/org/apache/lucene/search/CappedScoreQuery.java Outdated

    
                  @Override

                  public Weight createWeight(IndexSearcher searcher, ScoreMode scoreMode, float boost) throws IOException {

                      final Weight innerWeight = searcher.createWeight(query, ScoreMode.COMPLETE, 1f);

Contributor

jpountz Jul 17, 2019

should we pass scoreMode rather than hardcode ScoreMode.COMPLETE, and boost rather than 1f?

Contributor Author

markharwood Jul 17, 2019

apologies, this was carried over from ConstantScoreQuery too.

...ck/plugin/search-business-rules/src/main/java/org/apache/lucene/search/CappedScoreQuery.java Outdated Show resolved Hide resolved

x-pack/plugin/search-business-rules/src/main/java/org/apache/lucene/search/CappedScorer.java Outdated Show resolved Hide resolved

...ness-rules/src/main/java/org/elasticsearch/xpack/searchbusinessrules/PinnedQueryBuilder.java Outdated Show resolved Hide resolved

...ness-rules/src/main/java/org/elasticsearch/xpack/searchbusinessrules/PinnedQueryBuilder.java Outdated Show resolved Hide resolved

...ness-rules/src/main/java/org/elasticsearch/xpack/searchbusinessrules/PinnedQueryBuilder.java Outdated Show resolved Hide resolved

...ness-rules/src/main/java/org/elasticsearch/xpack/searchbusinessrules/PinnedQueryBuilder.java Outdated Show resolved Hide resolved

jpountz reviewed

View reviewed changes

...ness-rules/src/main/java/org/elasticsearch/xpack/searchbusinessrules/PinnedQueryBuilder.java

    
                          organicAndPinned.add(pinnedQueries.build());

                          // Cap the scores of the organic query

                          organicAndPinned.add(new CappedScoreQuery(organicQuery.toQuery(context), MAX_ORGANIC_SCORE));

                          return new DisjunctionMaxQuery(organicAndPinned, 0);

Contributor

jpountz Jul 17, 2019

FYI seeing this made me notice that disjunction max queries are a bit inefficient at retrieving top hits at the moment, even though there are some easy wins, especially when the tie-break multiplier is 0, so I opened https://issues.apache.org/jira/browse/LUCENE-8922

markharwood mentioned this pull request

English-minimal analyzer has bad plural stemming #42892

Open

markharwood force-pushed the fix/44074 branch 6 times, most recently from b4276d6 to b7afe05 Compare

July 23, 2019 10:47

Contributor Author

markharwood commented Jul 23, 2019

@hub-cap This basic-licensed Query will presumably need an Apache-licensed client-side equivalent for use in HLRC. Much code would be duplicated between client and server-side classes so my assumption is server-side QueryBuilder with the execution logic should extend a client-side QueryBuilder that holds the search criteria. The server would override this method in the client-side class:

public class PinnedQueryBuilder extends AbstractQueryBuilder<PinnedQueryBuilder> {
...
@Override
protected Query doToQuery(QueryShardContext context) throws IOException {
    throw new UnsupportedOperationException("Client side-only class for use in HLRC");
}

Does this seem a reasonable way forward?

Member

javanna commented Jul 24, 2019

heya @markharwood it is correct that client and server code should diverge, especially in this case. I am not sure about server code extending client code though. We don't want server to depend on HLRC? I think the way forward may be to have two separate classes.

Contributor Author

markharwood commented Jul 24, 2019 •

edited

Loading

We don't want server to depend on HLRC?

My concern with duplicating code is the potential to diverge e.g. it then relies on 2 classes agreeing on search criteria choices like:

the name of the query
the validation of params (eg if we reject an empty set for IDs or not)

I guess adding test code helps ensure they don't diverge but it may require more HLRC tests to guarantee that.
I suppose the fundamental challenge in organising code here is that the Basic-license code can depend on Apache licensed classes but not vice-versa. Maybe some of the constants and validation logic that would be useful to share needs to be put somewhere common to both HLRC and server-side?

I am not sure about server code extending client code though

To be clear in this case it's a QueryBuilder class - and they currently all exist in package org.elasticsearch.index.query in server. I'm guessing the long-term plan is to move them out of there to break HLRC dependencies on server code

Member

javanna commented Jul 25, 2019

I'm guessing the long-term plan is to move them out of there to break HLRC dependencies on server code

heya @markharwood indeed, you should chat with @hub-cap to get more details on this though. Not sure if we plan on having some common library shared between server and client. Probably not in the short term.

Contributor

jpountz commented Jul 25, 2019

Agreed with @javanna I think it's unlikely we'll get a shared library in the short term. There are already a number of request objects that we duplicated between server and client, I'm leaning towards doing the same here, at least in the initial version. Furthermore I don't think we have to have the validation logic on the client side, we could only have it on the server side to address the concern that validation logic diverges across the client and the server?

Contributor Author

markharwood commented Jul 25, 2019

OK - I'll put a copy of the existing PinnedQueryBuilder into server's org.elasticsearch.index.query package (minus the "toQuery" implementation) for use by HLRC.
Thanks for the comments @jpountz and @javanna

Contributor

jpountz commented Jul 25, 2019

Is there a reason why it couldn't go directly into the client module?

Contributor Author

markharwood commented Jul 25, 2019

Is there a reason why it couldn't go directly into the client module?

I was putting it with all the other existing QueryBuilders in server but I can break with that tradition if you like

Contributor Author

markharwood commented Aug 5, 2019

@hub-cap would appreciate your ruling on this packaging issue - new Basic-licensed query (a first) and where to put the Apache-licensed QueryBuilder for use in HLRC

$@polyfractal$

Contributor

polyfractal commented Aug 5, 2019

Registering interest in this discussion :) @imotov, @nknize and I are running into the same issue on other in-flight PRs (query or agg in Basic+ that needs to be exposed in HLRC). So there will at least be a handful of functionality that will go down whatever route is decided.

markharwood force-pushed the fix/44074 branch from e58f170 to de2dae3 Compare

August 8, 2019 14:47

markharwood force-pushed the fix/44074 branch from de2dae3 to 79269fd Compare

August 9, 2019 07:43

markharwood requested a review from hub-cap

August 12, 2019 08:18

Contributor Author

markharwood commented Aug 12, 2019

@hub-cap added you as a reviewer to check out the HLRC approach to packaging this basic-licensed query.

jimczi requested changes

View reviewed changes

Contributor

jimczi left a comment

I left one comment, I think we should duplicate the request/response like we did for other APIs in the client module and not in server.

server/src/main/java/org/elasticsearch/index/query/PinnedQueryBuilder.java Outdated Show resolved Hide resolved

markharwood force-pushed the fix/44074 branch from 79269fd to 7da0188 Compare

August 13, 2019 14:44

Contributor Author

markharwood commented Aug 13, 2019 •

edited

Loading

Following a conversation with Jim we've decided to pull out HLRC support from this PR to allow us to get it into master and will then open a separate discussion to figure out the right approach for HLRC

markharwood force-pushed the fix/44074 branch 2 times, most recently from de74c31 to 73c479d Compare

August 13, 2019 15:11

Contributor Author

markharwood commented Aug 15, 2019

Good to go, @jimczi ?
I pulled the HLRC stuff and got a green build here.

jimczi approved these changes

View reviewed changes

Contributor

jimczi left a comment

LGTM

x-pack/plugin/search-business-rules/.gitignore Outdated

    
            @@ -0,0 +1 @@
          
              /bin/

Contributor

jimczi Aug 16, 2019

Is this needed ?

server/src/main/java/org/elasticsearch/index/query/QueryBuilders.java Outdated

    
            @@ -722,5 +722,5 @@ public static GeoShapeQueryBuilder geoDisjointQuery(String name, String indexedS
          
                   */

                  public static ExistsQueryBuilder existsQuery(String name) {

                      return new ExistsQueryBuilder(name);

                  }

                  }

Contributor

jimczi Aug 16, 2019

Can you revert this ?

markharwood force-pushed the fix/44074 branch from 73c479d to 547c7c5 Compare

August 16, 2019 08:58

markharwood added 2 commits

August 16, 2019 12:36


          Search enhancement: - new query type allows selected documents to be …

5556e63

…promoted above “any organic” search results.

This is the first feature in a new module `search-business-rules` which will house licensed (non OSS) logic for rewriting queries according to business rules.
The PinnedQueryBuilder class offers a new `pinned` query in the DSL that takes an array of promoted IDs and an “organic” query and ensures the documents with the promoted IDs rank higher than the organic matches.

Closes elastic#44074


          Remove redundant .gitignore file

752e1be

markharwood force-pushed the fix/44074 branch from 547c7c5 to 752e1be Compare

August 16, 2019 11:37

markharwood merged commit f0dbc68 into elastic:master

markharwood added backport pending and removed backport pending labels

markharwood added a commit to markharwood/elasticsearch that referenced this pull request


          Search enhancement: pinned queries (elastic#44345)

d9cdf7b

Search enhancement: - new query type allows selected documents to be promoted above any "organic” search results.
This is the first feature in a new module `search-business-rules` which will house licensed (non OSS) logic for rewriting queries according to business rules.
The PinnedQueryBuilder class offers a new `pinned` query in the DSL that takes an array of promoted IDs and an “organic” query and ensures the documents with the promoted IDs rank higher than the organic matches.

Closes elastic#44074

markharwood added a commit that referenced this pull request


          Search enhancement: pinned queries (#44345) (#45657)

7d5ab17

* Search enhancement: pinned queries (#44345)

Search enhancement: - new query type allows selected documents to be promoted above any "organic” search results.
This is the first feature in a new module `search-business-rules` which will house licensed (non OSS) logic for rewriting queries according to business rules.
The PinnedQueryBuilder class offers a new `pinned` query in the DSL that takes an array of promoted IDs and an “organic” query and ensures the documents with the promoted IDs rank higher than the organic matches.

Closes #44074

markharwood mentioned this pull request

Added HLRC support for PinnedQueryBuilder #45779

Merged

codebrain mentioned this pull request

7.4 meta ticket elastic/elasticsearch-net#4133

Closed

56 tasks

Mpdreamz added a commit to elastic/elasticsearch-net that referenced this pull request


          Search enhancement: pinned queries

1194b38

Addresses elastic/elasticsearch#44345

Mpdreamz mentioned this pull request

Search enhancement: pinned queries elastic/elasticsearch-net#4143

Merged

Mpdreamz added a commit to elastic/elasticsearch-net that referenced this pull request


          Search enhancement: pinned queries (#4143)

2c125f3

* Search enhancement: pinned queries

Addresses elastic/elasticsearch#44345

* update docs in test

* remove params overloads on Ids that already convert to Ids

* add note to IdsQuery

Mpdreamz added a commit to elastic/elasticsearch-net that referenced this pull request


          Search enhancement: pinned queries (#4143)

36d9210

* Search enhancement: pinned queries

Addresses elastic/elasticsearch#44345

* update docs in test

* remove params overloads on Ids that already convert to Ids

* add note to IdsQuery

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment