Function score query fails with field_value_factor on not (yet) existing field

[tschubotz]

Hi,

My ES function score query contains several different functions based on a set of different fields. I'm having problems with field_value_factor on not existing fields.
I'm not sure if that's expected behavior or a bug in ES.
On the elasticsearch Google group, nobody had an idea, so any help is appreciated.

A minimal example on how to reproduce the problem:

Example document:

curl -XPOST "http://localhost:9200/blog/page/1?pretty" -d '
{
  "author": "tobias",
  "content": "Great",
  "views": 1
}'

A search request, that boosts based on the function score looks like that:

curl -XPOST "http://localhost:9200/blog/_search" -d '
{
  "query": {
    "function_score": {
      "functions": [
        {
          "field_value_factor": {
            "field": "views",
            "modifier": "log2p"
          }
        }
      ]
    }
  }
}'

Everything is fine until this point.
When I change the "field" param to "likes", the requests fails:

curl -XPOST "http://localhost:9200/blog/_search" -d '
{
  "query": {
    "function_score": {
      "functions": [
        {
          "field_value_factor": {
            "field": "likes",
            "modifier": "log2p"
          }
        }
      ]
    }
  }
}'

-> ElasticsearchException[Unable to find a field mapper for field [likes]

"likes" doesn't exist yet in the "page" document type, but it might exist in the future. Or there could be another type "page2", where the "likes" field exists, and I need to query both "page" and "page2" in 1 request.

What I tried, is to put an exist filter beforehand, however, the error remains the same.

curl -XPOST "http://localhost:9200/blog/_search" -d '
{
    "query": {
        "function_score": {
            "functions": [
                {
                    "filter": {
                        "exists": {
                            "field": "likes"
                        }
                    },
                    "field_value_factor": {
                        "field": "likes"
                    }
                }
            ]
        }
    }
}'

-> ElasticsearchException[Unable to find a field mapper for field [likes]

Is this a bug or expected bahavior?
I would expect ES to just skip the function, in case the "exists" filter evaluates to False or the field doesn't exist.

The error originates from FieldValueFactorFunctionParser.java (line 89), in FieldValueFactorFunctionParser.parse.
Is there really a need to check for the existence of all fields during parsing?

With other function types, there is no problem, so apparently this parsing is not performed. E.g. the following works without an error:

curl -XPOST "http://localhost:9200/blog/_search" -d '
{
  "query": {
    "function_score": {
      "functions": [
        {
            "filter": {
                "range": {"likes": {"from": 1}}
            }, 
            "weight": 1.5
        }
      ]
    }
  }
}'

Any ideas?

Please let me know, if you need further information.

Best regards,
Tobias

[rjernst]

The "working" example you have isn't really the same; it is doing a filter, not a function score, in which case the field not existing in the mappings means "match no docs". But for field_value_factor (and other functions that try to take the value of a field), if the field does not exist in mappings at all, then the only option is to produce a fake value for each doc (we can do this, but it is trappy and arbitrary). IMO this does need some more standard behavior throughout ES apis. For example, we should have the ability to be lenient (warn or ignore) for missing fields, but in those cases we need to have:

1. Well defined behavior
2. A standard way to choose what the behavior should be (on Enforce _parent field resolution to be strict #9521 it was discussed for only queries, but function scores are only sort of queries, maybe it should be the same though).
3. Default to error on missing field (otherwise this is trappy for new users who e.g. mispell a field name in a query)

[clintongormley]

In this particular case, we could probably handle this by adding a missing parameter, which would be used if a document doesn't have a value, or if the type doesn't have the field mapped.

[clintongormley]

#10845 has already added a missing value to field_value_factor but it doesn't work for an index which is missing the field in its mapping.

Perhaps the way to go here is to extend support for missing mappings with the suggestion here: #9521 (comment)

[clintongormley]

After discussing this in fix it friday, we think that the new missing parameter should be extended to apply to types where the field itself is not mapped.

[andrestc]

I will take a shot at this one.