Field stats: added index_constraint option

[martijnvg]

Field stats index constraints allows to omit all field stats for indices that don't match with the index constraint. An index constraint can exclude indices' field stats based on a field's min_value and max_value statistics.

For example index constraints can be useful to find out the min and max value of a particular property of your data in a time based scenario. The following request only returns field stats for the answer_count property for indices holding questions created in the year 2014:

curl -XPOST 'http://localhost:9200/_field_stats?level=indices' -d '{
   "fields" : ["answer_count"]
   "index_constraints" : {
      "creation_date" : {
         "min_value" : {
            "gte" : "2014-01-01T00:00:00.000Z",
         },
         "max_value" : {
            "lt" : "2015-01-01T00:00:00.000Z"
         }
      }
   }
}'

PR for #11187

[jpountz]

I'm personally a bit confused by the API. For instance the documentation says The following request only returns field stats for indices that have_timestampdate values between the defined range. but if I read the code correctly, an index that only has two values: one that is less than the minimum value and another one that is greater than the max value, then it would match although it has no values in the range?

Instead, maybe the API should apply to the min/max values instead of the field themselves to be clearer about what it does? Eg.

curl -XPOST 'http://localhost:9200/_field_stats?level=indices' -d '{
   "fields" : {
      "_timestamp.max" : {
         "gte" : "2014-01-01T00:00:00.000Z"
      },
      "_timestamp.min" : {
         "lt" : "2015-01-01T00:00:00.000Z"
      }
   }
}'

[jpountz]

My above comment does not necessarily mean I think it's a bug, but I wanted to raise the corner case so that our API gurus like @clintongormley could help figure out if the API is right.

[martijnvg]

@jpountz @clintongormley The field stats filtering doesn't really behave like a filter on docs in an index. It is rather more of an overlapping mechanism / range matching.

Lets say an index contains logs from 02-01-2015 to 03-01-2015. A range from 01-01-2015 to 04-01-2015 should include the field stats for that index in the field stats response.

So maybe we should name and document it differently?

[jpountz]

+1 I would like it better if the API made it clearer this performs range overlapping - the current one feels more like it will return indices that have a value in the specified range, which is not what it does.

[clintongormley]

I like @jpountz 's suggestion a lot - makes things much clearer

[martijnvg]

I like to make a small change to the format @jpountz is suggesting:

curl -XPOST 'http://localhost:9200/_field_stats?level=indices' -d '{
   "fields" : {
      "_timestamp" : {
          "max" : {
              "gte" : "2014-01-01T00:00:00.000Z"
          },
          "min" : {
             "lt" : "2015-01-01T00:00:00.000Z"
          }
      }
   }
}'

By having on top level _timestamp element, it is clearer that we compute field stats for one field only.

[clintongormley]

See #11187 (comment)

[martijnvg]

I've updated this PR based on the comments in #11187. Also the title and description have been updated to match index constraints instead of field stats filter.

[rjernst]

does this mean we will silently ignore other tokens?

[martijnvg]

yes... instead lets fail if there is another token.

[martijnvg]

@rjernst Thanks for looking at this, I've updated the PR.

[martijnvg]

If there is no more feedback then I like to merge this PR in the next 48 hours.

[clintongormley]

@jpountz @rjernst @bleskes any more feedback?

[jpountz]

Should it be "unexpected" rather than "unknown"?

[martijnvg]

yes, it should! I'll change it

[jpountz]

I left some comments, otherwise LGTM. The one I'm most concerned about is what to do if a request has both a body and a fields parameter?

[martijnvg]

On 30 June 2015 at 14:36, Adrien Grand notifications@github.com wrote:

I left some comments, otherwise LGTM. The one I'm most concerned about is
what to do if a request has both a body and a fields parameter?

—
Reply to this email directly or view it on GitHub
#11259 (comment)
.

good point, we should just fail the request.