Scripting: ScriptDocValues.getValues() returns an reused list

[dw]

Using ElasticSearch 1.4.0 from your official .deb, given an index with:

"plays": {
    "mappings": {
        "play": {
            "_source": {
                "enabled": false
            },
            "_timestamp": {
                "default": null,
                "enabled": true
            },
            "dynamic": "strict",
            "properties": {
                "artist": {
                    "index": "not_analyzed",
                    "type": "string"
                },

And:

"settings": {
    "index": {
        "creation_date": "1416491787644",
        "merge": {
            "scheduler": {
                "max_thread_count": "1"
            }
        },
        "number_of_replicas": "1",
        "number_of_shards": "1",
        "refresh_interval": "-1",
        "uuid": "XDRbDOoLSR-cMFjQAm9TjQ",
        "version": {
            "created": "1040099"
        }
    }
}

A search like:

{
    "script_values": {
        "artist": {
            "script": "_doc['artist'].values"
        }
    }
}

Will return a result set whose hits contain an artist array whose elements correspond to the elements for the last search result, i.e. all previous hits artist arrays assume the same contents as that of the last hit. It looks like an object is being reused somehow, although casting a glance at ScriptDocValues.java I can't see how. Is it possible listLoaded is not being reset somehow?

A trivial workaround is:

{
    "script_values": {
        "artist": {
            "script": "_doc['artist'].values.take(100)"
        }
    }
}

[dw]

Aah, clearly the "problem" is that list is reset and reused for every doc in the query, which looks like a performance thing. I'm happy if this was marked as a doc bug rather than a functional bug, the current behaviour seems fair.

[jpountz]

@dw I tried to reproduce your issue without success. Here is what I ran:

DELETE plays 

PUT plays
{
  "mappings": {
    "play": {
      "dynamic": "strict",
      "properties": {
        "artist": {
          "type": "string",
          "index": "not_analyzed"
        }
      }
    }
  },
  "settings": {
    "number_of_shards": 1
  }
}

PUT plays/play/1
{
  "artist": [ "me", "you" ]
}

PUT plays/play/2
{
  "artist": [ "you", "him" ]
}

GET plays/_search
{
  "script_fields": {
    "artist": {
      "script": "doc['artist'].values"
    }
  }
}

which returned

{
   "took": 4,
   "timed_out": false,
   "_shards": {
      "total": 1,
      "successful": 1,
      "failed": 0
   },
   "hits": {
      "total": 2,
      "max_score": 1,
      "hits": [
         {
            "_index": "plays",
            "_type": "play",
            "_id": "1",
            "_score": 1,
            "fields": {
               "artist": [
                  [
                     "me",
                     "you"
                  ]
               ]
            }
         },
         {
            "_index": "plays",
            "_type": "play",
            "_id": "2",
            "_score": 1,
            "fields": {
               "artist": [
                  [
                     "him",
                     "you"
                  ]
               ]
            }
         }
      ]
   }
}

Can you maybe guide me a bit so that I can reproduce the issue? I used a fresh install of elasticsearch 1.4.0.

[dw]

@jpountz huh strange. How many shards does your index have? If >1 and both docs went to a different shard, and the script_value is executed in the context of a shard, that might explain why you don't see it

[dw]

(I've only been using ElasticSearch a week, so still guessing about things quite heavily)

[dw]

Ah, doh, sorry, didn't notice you'd set the shards to 1. Hmm, not sure

[jpountz]

OK I found the reason: this bug only occurs on a per-segment basis, so you need to run an optimize to reproduce the bug. Here is a recreation for the bug:

DELETE plays 

PUT plays
{
  "mappings": {
    "play": {
      "dynamic": "strict",
      "properties": {
        "artist": {
          "type": "string",
          "index": "not_analyzed"
        }
      }
    }
  },
  "settings": {
    "number_of_shards": 1
  }
}

PUT plays/play/1
{
  "artist": [ "me", "you" ]
}

PUT plays/play/2
{
  "artist": [ "you", "him" ]
}

POST plays/_optimize?max_num_segments=1

GET plays/_search
{
  "script_fields": {
    "artist": {
      "script": "doc['artist'].values"
    }
  }
}

which returns

{
   "took": 7,
   "timed_out": false,
   "_shards": {
      "total": 1,
      "successful": 1,
      "failed": 0
   },
   "hits": {
      "total": 2,
      "max_score": 1,
      "hits": [
         {
            "_index": "plays",
            "_type": "play",
            "_id": "2",
            "_score": 1,
            "fields": {
               "artist": [
                  [
                     "me",
                     "you"
                  ]
               ]
            }
         },
         {
            "_index": "plays",
            "_type": "play",
            "_id": "1",
            "_score": 1,
            "fields": {
               "artist": [
                  [
                     "me",
                     "you"
                  ]
               ]
            }
         }
      ]
   }
}