function_score/script_score query not returning scored results

[bernardosgr]

Hi guys, I've stumbled upon this issue and it seems to be a problem of the client... It might be a bit counter-intuitive but I'm fairly at ease with stating this because the issue is not present when I run the same query through a different client.

I'm currently trying to query ES through Python, using elasticsearch-py but with no success... I'm testing the request on the "elasticsearch-head" plugin and it works fine (results come scored). Also using the rawes client reveals no issues with the query.

However, when I do it in elasticsearch-py something seems to be wrong and although I get the same results, they are not scored properly...

here is the code:

custom_query={
  "query": {
    "function_score": {
      "query": {
        "match_all": {}
      },
      "functions": [
        {
          "script_score": {
            "lang": "groovy", 
            "script": "_score+20",
            "params":{"myMap":{}}
          }
        }
      ]
    }
  }
}

This is my custom query, very simple, just to test the scoring mechanism.

src = es.search(index=['test'],scroll='60s',search_type='scan',size=10,body=custom_query)

now I test it out:

while(1):
    src = es.scroll(scroll_id=sid, scroll='60s')
    sid = src['_scroll_id']
    kws.extend(src['hits']['hits'])

All this gives is a set of results that have been improperly scored... I've even enabled "_explanation" to understand if the script is being utilized...
This meaning:

• "_score" is always 0
• "_explanation" shows something like:

    {
        'description': 'function score',
        product of:'',
            'value': 21.0,
            'details': [
                {
                    'description': 'ConstantScore(*:*)',
                    product of:'',
                        'value': 1.0,
                        'details': [
                            {'description': 'boost', 'value': 1.0}, {'description': 'queryNorm', 'value': 1.0}
                        ]
                },
                {
                    'description': 'Math.min of',
                    'value': 21.0,
                    'details': [
                        {
                            'description': 'script score function',
                            computed with script:"return _score+20;" and parameters: \n{myMap={}}', 'value': 21.0
                        },
                        {
                            'description': 'maxBoost',
                            'value': 3.4028235e+38
                        }
                    ]
                },
                {
                    'description': 'queryBoost',
                    'value': 1.0
                }
            ]
    }

So, at least from the explanation, scoring seems appropriate but "_score" is not getting changed.
I really don't want to switch to rawes because of the helpers allowing me to scroll through the set of results with fair ease. Could this be an issue with "scroll"? Should I use "scan" instead and set "preserve_order" to True? I've half-tried it with no positive outcome...

Thanks a lot in advance,
Bernardo

[honzakral]

I am afraid this is not an issue with the python client - it works fine with normal search (no search_type=scan) and the behavior you show is the same with raw HTTP, try curl:

curl -XGET 'http://localhost:9200/_search?pretty&scroll=5m&search_type=scan' -d '{
  "query": {
    "function_score": {
      "functions": [
        {
          "script_score": {
            "lang": "groovy",
            "params": {
              "myMap": {}
            },
            "script": "_score+20"
          }
        }
      ],
      "query": {
        "match_all": {}
      }
    }
  }
}'

And then

curl -XGET 'http://localhost:9200/_search/scroll?pretty&scroll=5m' -d 'SCROLL_ID'

it behaves the same. Looks like in search type scan the function score doesn't apply correctly.

Since this is not documented with elasticsearch I would consider it a bug there - either it should work or it should be properly documented that it doesn't. I will consult the elasticsearch developers and open a ticket there if it's indeed the case.

Thanks for the thorough report!

[honzakral]

Actually I forgot that scan search_type doesn't do scoring at all - so function_score won't work there at all. I will create an issue in elasticsearch repo to make the docs clearer on that.

[bernardosgr]

Yep, that is in fact the problem. I have no trouble doing custom scoring with just a regular scroll-id and passing it on to ES through the rawes client. Just a shame it's such a low-level client...

Thanks for your help and for reporting the issue. I was starting to feel weird when I saw nobody having my problem XD

[honzakral]

You can do the same - just ask for scroll and pass it to scroll through this client.

Alternatively (I just learned) you can also add a track_scores parameter when doing the search and it will add the scores (still not sort by it though). This is exactly what you are looking for. It is new so not yet supported by elasticsearch-py to use it pass it in params:

es.search(body=..., search_type='scan', scroll='5m', params={'track_scores': 'true'})