script_fields does not return _source and seems to ignore the fields part

[machwatt]

Summary of problem or feature request

I try to include a script_fields part to the $params for a client->search(). The script is working, i tried the same request with Kibana/Sense. In Sense i get a response like this:

 "hits": {
    "total": 990,
    "max_score": 1.4142135,
    "hits": [
      { some entries here }

In elasticsearch-php i get the same response but with an empty array for hits[hits].

Code snippet of problem

$params['body']['script_fields'] = $scriptFields;
$result = $this->client->search($params);

System details

• Operating System: OSX 10.11.5
• PHP Version: 7.0.8
• ES-PHP client version: 2.0.0
• Elasticsearch version 2.3.3

[polyfractal]

Could you paste your full query from Sense, and the full PHP code that constructs the request? There's probably just a typo somewhere which is making the query behave differently.

[machwatt]

Sure, i'll try to sum it up.

Thanks for your help 👍

The plain json request

GET units/unit/_search
{
  "query": {
    "function_score": {
      "query": {
        "bool" : {
          "must" : [
            {
              "geo_bounding_box" : {
                "geoLocation" : {
                  "top_left" : {
                    "lat" : 54.702854057633,
                    "lon" : 7.8999204101563
                  },
                  "bottom_right" : {
                    "lat" : 52.546765426354,
                    "lon" : 12.426287597656
                  }
                }
              }
            },
            {
              "range": {
                "maximumNumberOfPersons": {
                  "gte" : 2
                }
              }
            }
          ]
        }
      },
      "functions": [
        {
          "field_value_factor": {
            "field": "displayQuality",
            "factor": 1.2
          }
        }
      ]
    }
  },
  "from": 0,
  "size": 12,
  "script_fields": {
    "additionalinformation" : {
      "script": {
        "file" : "compute-unit-information"
      }
    }
  }
}

And the code building up the query. Divided in parts through some service classes.

$query = [
    'function_score' => [
        'query' => [
            'bool' => [

            ]
        ],
        'functions' => [
            ['field_value_factor' => [
                'field' => 'displayQuality',
                'factor' => $this->elasticSearchSettings['scoring']['displayQualityFactor']
            ]]
        ]
    ]
];

I'll inject this later into this $params array:

$params = [
    'index' => $this->getIndex(),
    'type' => $this->getType(),
    'body' => [
        'from' => $page * $size,
        'size' => $size,
        'query' => $query
    ]
];

And under some circumstances i'll add this to the body part:

$params['body']['script_fields'] = $scriptFields;
$params['body']['fields'] = [
    '_source'
]; 

And later i'll run:

$result = $this->client->search($params);

I'll attach a screenshot of the $params array that is given to the client as parameter.

[polyfractal]

Hmmm.... I don't see anything incorrect at the moment. I'll play around with a smaller version here and see if I can recreate it.

So in PHP, you are getting the response back which has the same number of hits and score... but no actual hits in the ['hits']['hits'] array? What happens if you selectively omit the script_fields, fields or _source parameters (e.g. use only one of the three)?

Side note: you don't need to include fields: _source. The default behavior is to return the _source field, and if you just want source filtering the _source param is a better way to do it. fields is mainly for "stored" fields, which aren't used very often.

[machwatt]

First thanks for the side note. I removed the fields: _source part. I included this before i had to exclude some parts via _source.

But its just like before:

This $params

producing the following response:

But in Sense the ['hits']['hits'] is not empty. And shows the script fields.
I tried even with removing the query, aggs and _source part but got an empty ['hits']['hits'] back.

[polyfractal]

That's super odd, I'm stumped. It must be a syntax problem; the client doesn't validate or touch the body parameter at all... it just encodes and sends straight to Elasticsearch. So there must be a subtle syntax issue somewhere... but I'm not seeing it.

Can you insert a print_r(json_encode($params['body'])) before the call? Maybe there is something funky about the json encoding?

Also, would it be possible to gist up the JSON response that you get via Sense or curl (assuming there isn't any sensitive information in the search results)? I'd like to see if perhaps the response is not valid JSON. That could make the json decoder barf but potentially not throw an exception.

Older versions of Elasticsearch allowed you to index malformed JSON for example, which sometimes causes problems with decoders when the source is returned.

Not sure, will keep thinking...

[machwatt]

It seems you were right. The print_r creates a SyntaxError: Unexpected end of JSON input I'll investigate a little in that and provide you further feedback.

Update
It seems like its an issue in `$params['body']['source']. I'm investigating further

Update 2
I dont know really...
I just assigned did json_encode($params['body']); without print_r(). This is the resulting encoded json:
https://gist.github.com/machwatt/bc1b30fefe81057329745e4886a03616

I was not able yet to track down the issue while using print_r. I'll do that tomorrow as i need to get out of office now :/

Thanks for your investigation! Is there a way to get you some 🍺?

[polyfractal]

Oooh, interesting. Eagerly await your analysis :)

Also curious why PHP didn't throw that exception... wonder if we're suppressing something on accident. Will take a look in a bit when i get a chance.

[machwatt]

The json_decode runs well as it seems. I'll write you back tomorrow!

[machwatt]

I just dont get it. I debugged into the Connection class which performs the request and this is the body of it: https://gist.github.com/machwatt/b476315c2bc54865a428410366e17a99

The ['hits'] in the resulting $promise->wrappedPromise is empty then. The resulting error above was written by a FE javascript lib... I'll just check if i can go any further into this.

[machwatt]

Think i found it. There is an error in the script which is not logged in the ES log, i'll try figuring out why this is not the case. I missed a timestamp to ms transformation in the script, what i did in the Sense JSON...
Then the ['hits'] array is empty because he cant add the script_fields but there is no error provided, that something went wrong. Sad story.

I hope you didn't do much investigation. The promise of a 🍺 is meant seriously :)

You might close this issue then 👍

[polyfractal]

Ah, cool. Glad you (sorta) figured it out :) Curious that no error from ES was sent back. Did you notice any 'failed shards' in the "shards" response element?

Closing, feel free to reopen if you run into something else that looks to be a bug in the client :)