top_children and function_score / script_score don't play nice together

[synhershko]

Consider the following scenario, executed with 1.3.2:

DELETE /test

PUT /test
{
  "settings": {"number_of_replicas": 0,"number_of_shards": 1}, 
  "mappings": {
    "tagged-page":{
        "_parent": {
           "type": "pages"
        },
        "properties": {
            "name": {
              "type": "string","index": "not_analyzed"
            },
            "score": {
              "type": "float"
            }
        }
    },
    "pages": {
      "properties": {
        "title":{"type": "string", "index": "no" }
      }
    }
  }
}

POST /test/pages/1
{
    "title":"page1"
}

POST /test/tagged-page?parent=1
{
    "name":"foo", "score":1.0
}

POST /test/tagged-page?parent=1
{
    "name":"bar", "score":5.0
}

POST /test/pages/2
{
    "title":"page2"
}

POST /test/tagged-page?parent=2
{"name":"cafe", "score":5.0 }

POST /test/tagged-page?parent=2
{"name":"bar", "score":1.0}

POST /test/pages/3
{
    "title":"page3"
}

POST /test/tagged-page?parent=3
{"name":"foo", "score":100.0 }

POST /test/tagged-page?parent=3
{"name":"bar", "score":3.0}

POST /test/pages/4
{
    "title":"page4"
}

POST /test/tagged-page?parent=4
{"name":"foo", "score":5.0 }

POST /test/tagged-page?parent=4
{"name":"bar", "score":20.0}

POST /test/pages/5
{
    "title":"page5"
}

POST /test/tagged-page?parent=5
{"name":"foo", "score":3.0 }

POST /test/pages/6
{
    "title":"page6"
}

POST /test/tagged-page?parent=6
{"name":"cafe", "score":2.0}

POST /test/_refresh

POST /test/pages/_search
{
  "query": {
    "top_children" : {
        "type": "tagged-page",
        "query": {
            "function_score": {
                "query": {
                    "term": {
                        "name": "foo"
                    }
                },
                "script_score": {
                    "lang" : "groovy",
                    "script": "doc[\"score\"].value"
                }
            }
        },
        "score" : "max",
        "factor" : 5,
        "incremental_factor" : 2
    }
  }
}

I get the following error:

{
   "error": "SearchPhaseExecutionException[Failed to execute phase [query_fetch], all shards failed; shardFailures {[BDBL4U4jTbCVsGvmCYzGpg][test][0]: QueryPhaseExecutionException[[test][0]: query[filtered(score_child[tagged-page/pages](filtered(function score (name:foo,function=script[doc[\"score\"].value], params [null]))->cache(_type:tagged-page)))->cache(_type:pages)],from[0],size[10]: Query Failed [Failed to execute main query]]; nested: RuntimeException[org.elasticsearch.script.groovy.GroovyScriptExecutionException: ElasticsearchIllegalArgumentException[No field found for [score] in mapping with types [pages]]]; nested: GroovyScriptExecutionException[ElasticsearchIllegalArgumentException[No field found for [score] in mapping with types [pages]]]; }]",
   "status": 500
}

Note the No field found for [score] in mapping with types [pages]. Meaning, function_score was trying to execute the script_score within the context of the query to pages, instead of against the internal query to tagged-page as I would expect.

[martijnvg]

@synhershko Thanks for opening this. This is a bug also the has_child query suffers from the same problem. If the document type has already been set in the url then fields of other types can't be resolved via a script in any (sub) query.

[synhershko]

@martijnvg any ETA on a fix for this?

[sandstrom]

I've got a possibly similar issue, where the function is improperly run against some results/documents, causing an 'unknown field' error.

# This query is run against two indices, 'users' and 'speakers'.
# But I think the situation would be the same if it was a single index, 
# with two different types in it.
# 
# I get the following error:
# `QueryParsingException[[users] Unknown field [mentioned_at]];`
#
# It seems like it's trying to run the query against all results, 
# instead of being scoped to the query.
# All speakers has the `mentioned_at` field, and no user has it.

query_body = {
  :query => {
    :bool => {
      :should => [
        {
          :function_score => {
            :query => {
              :multi_match => {
                :query => "MY QUERY HERE",
                :type => :phrase_prefix,
                :fields => ['name']
              }
            },
            :filter => { :type => { :value => 'speaker' } },
            :functions => [
              {
                :gauss => {
                  'mentioned_at' => {
                    :scale => '180d', # '6M' syntax couldn't be used, probably due to ES-bug
                    :offset => '5d',
                    :decay => 0.5,
                  }
                }
              }
            ]
          }
        },
        {
          :filtered => {
            :query => {
              :multi_match => {
                :query => "MY QUERY HERE",
                :type => :phrase_prefix,
                :fields => ['name']
              }
            },
            :filter => { :type => { :value => 'user' } },
          }
        }
      ]
    }
  }
}

[martijnvg]

@synhershko Sorry, I missed your comment... This issue should be resolved by #4081, internally this issue will change the fact that there are multiple mappings per type, so there will be one mapping per index.

Right now because in the url your scope has been reduced to the parent type and no other query can therefor in scripting access fields outside of this scope. This restriction should be removed when #4081 is added.

The best work around right now is to not define type in the url and do this in the query dsl instead.

[martijnvg]

@sandstrom This is because your query runs on an index that simply doesn't have that field and the function_score quest is strict about that.

I think the best way to get around this issue is to wrap the function_score query in the indices query and restrict on what indices the sub query is actually ran on:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-indices-query.html#query-dsl-indices-query

[sandstrom]

@martijnvg Doesn't filter run before the query?

My guess was that function_score can take either a query or a filter, but not both. There is a | in the documentation, which I'm guessing signifies a bitwise OR.

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.html#_using_function_score

[martijnvg]

@sandstrom This depends on what place the filter is defined. If the filter is defined in the query dsl via a filtered query it depends, but if it is defined in post_filter then the filter is always checked last.

Yes, the function_score query can either accept a filter or a query. If both are defined then the last one parsed will actually be used.

[sandstrom]

@martijnvg Thanks for confirming my guess, much appreciated! ⛵

I've opened a separate issue to track that: #8638

[clintongormley]

As of 2.0 this now works correctly, eg:

POST /test/pages/_search
{
  "query": {
    "has_child": {
      "score_mode": "max",
      "type": "tagged-page",
      "query": {
        "function_score": {
          "boost_mode": "replace",
          "query": {
            "term": {
              "name": "foo"
            }
          },
          "script_score": {
            "lang": "groovy",
            "script": "doc[\"score\"].value"
          }
        }
      }
    }
  }
}