Transform script and completion suggest payload - malformed JSON output

[jo-m]

I am using the completion suggester. The payload is being created using a transform script. This somehow breaks JSON output. A detailed shell script for reproducing the bug is included. This was tested with version 1.4.0.

#!/bin/bash

ELASTIC_URL='http://elastic:9200'
INDEX_NAME='test'
OBJ_ID=some_user_id

# delete index
curl -XDELETE "$ELASTIC_URL/$INDEX_NAME" &> /dev/null

# create index with mapping
# defined suggest field and includes transform script
# for populating the payload
curl -XPOST "$ELASTIC_URL/$INDEX_NAME" -d @- &> /dev/null <<EOF
{
    "mappings" : {
        "app_user" : {
            "_source" : {
                "includes" : [
                    "_id",
                    "_rev",
                    "type",
                    "profile.callname",
                    "profile.fullname",
                    "email"
                ]
            },
            "properties" : {
                "suggest" : { "type" : "completion",
                              "payloads" : true
                }
            },
            "transform" : [
                {"script": "ctx._source.suggest = ['input':[ctx._source.email, ctx._source.profile.fullname, ctx._source.profile.callname]]"},
                {"script": "ctx._source.suggest.payload = ['display': ctx._source.profile.fullname, 'display_detail': ctx._source.email]"}
            ]
        }
    }
}
EOF

# put the object
curl -XPUT "$ELASTIC_URL/$INDEX_NAME/app_user/$OBJ_ID" -d @- &> /dev/null <<EOF
{
    "_id": "$OBJ_ID",
    "type": "app_user",
    "version": 1,
    "profile": {
        "callname": "User Name",
        "fullname": "Dr. User Name",
        "profile_text": "Lorem ipsum dolor sit amet blah blah"
    },
    "created": "2014-08-14T20:07:28Z",
    "last_login": "2014-08-14T20:07:28Z",
    "login_counter": 0,
    "email": "user@example.org"
}
EOF

# wait for index to be updated
sleep 4

# interestingly, here the payload is displayed correctly
echo 'TRANSFORMED OBJECT:'
curl -XGET "$ELASTIC_URL/$INDEX_NAME/app_user/$OBJ_ID?pretty&_source_transform" 2> /dev/null | jsonlint

# but not when we retrieve the document by search.
echo 'RETRIEVED BY SEARCH:'
curl -XGET "$ELASTIC_URL/$INDEX_NAME/_suggest" -d '{
    "all-suggest": {
        "text": "User Name",
        "completion": {
            "field": "suggest"
        }
    }
}'

This produces output like this:

TRANSFORMED OBJECT:
{
  "_index": "test",
  "_type": "app_user",
  "_id": "some_user_id",
  "_version": 1,
  "found": true,
  "_source": {
    "_id": "some_user_id",
    "email": "user@example.org",
    "suggest": {
      "input": [
        "user@example.org",
        "Dr. User Name",
        "User Name"
      ],
      "payload": {
        "display": "Dr. User Name",
        "display_detail": "user@example.org"
      }
    },
    "type": "app_user",
    "profile": {
      "callname": "User Name",
      "fullname": "Dr. User Name"
    }
  }
}
RETRIEVED BY SEARCH:
{"_shards":{"total":5,"successful":5,"failed":0},"all-suggest":[{"text":"User Name","offset":0,"length":9,"options":[{"text":"User Name","score":1.0,"payload"::)
�displayLDr. User Name�display_detailOuser@example.org�}]}]}

[nik9000]

Thanks for the reproduction steps. I'll have a look.

[nik9000]

Ok, figured it out. I'll have a pull request to fix it in a bit.

[jo-m]

very cool ;)

[nik9000]

@jo-m OK - fix in pull request sent. To actually get the fix applied you'd have to upgrade to a version of Elasticsearch with the fix merged (assuming it is merged) and reindex the documents. You might also want to look at #8975 which, if we go ahead and fix that too, would make reindexing the documents no longer required.

[jo-m]

thank you!

[clintongormley]

thanks @nik9000!