Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transform script and completion suggest payload - malformed JSON output #8959

Closed
jo-m opened this issue Dec 15, 2014 · 6 comments
Closed

Transform script and completion suggest payload - malformed JSON output #8959

jo-m opened this issue Dec 15, 2014 · 6 comments

Comments

@jo-m
Copy link

jo-m commented Dec 15, 2014

I am using the completion suggester. The payload is being created using a transform script. This somehow breaks JSON output. A detailed shell script for reproducing the bug is included. This was tested with version 1.4.0.

#!/bin/bash

ELASTIC_URL='http://elastic:9200'
INDEX_NAME='test'
OBJ_ID=some_user_id

# delete index
curl -XDELETE "$ELASTIC_URL/$INDEX_NAME" &> /dev/null

# create index with mapping
# defined suggest field and includes transform script
# for populating the payload
curl -XPOST "$ELASTIC_URL/$INDEX_NAME" -d @- &> /dev/null <<EOF
{
    "mappings" : {
        "app_user" : {
            "_source" : {
                "includes" : [
                    "_id",
                    "_rev",
                    "type",
                    "profile.callname",
                    "profile.fullname",
                    "email"
                ]
            },
            "properties" : {
                "suggest" : { "type" : "completion",
                              "payloads" : true
                }
            },
            "transform" : [
                {"script": "ctx._source.suggest = ['input':[ctx._source.email, ctx._source.profile.fullname, ctx._source.profile.callname]]"},
                {"script": "ctx._source.suggest.payload = ['display': ctx._source.profile.fullname, 'display_detail': ctx._source.email]"}
            ]
        }
    }
}
EOF

# put the object
curl -XPUT "$ELASTIC_URL/$INDEX_NAME/app_user/$OBJ_ID" -d @- &> /dev/null <<EOF
{
    "_id": "$OBJ_ID",
    "type": "app_user",
    "version": 1,
    "profile": {
        "callname": "User Name",
        "fullname": "Dr. User Name",
        "profile_text": "Lorem ipsum dolor sit amet blah blah"
    },
    "created": "2014-08-14T20:07:28Z",
    "last_login": "2014-08-14T20:07:28Z",
    "login_counter": 0,
    "email": "user@example.org"
}
EOF

# wait for index to be updated
sleep 4

# interestingly, here the payload is displayed correctly
echo 'TRANSFORMED OBJECT:'
curl -XGET "$ELASTIC_URL/$INDEX_NAME/app_user/$OBJ_ID?pretty&_source_transform" 2> /dev/null | jsonlint

# but not when we retrieve the document by search.
echo 'RETRIEVED BY SEARCH:'
curl -XGET "$ELASTIC_URL/$INDEX_NAME/_suggest" -d '{
    "all-suggest": {
        "text": "User Name",
        "completion": {
            "field": "suggest"
        }
    }
}'

This produces output like this:

TRANSFORMED OBJECT:
{
  "_index": "test",
  "_type": "app_user",
  "_id": "some_user_id",
  "_version": 1,
  "found": true,
  "_source": {
    "_id": "some_user_id",
    "email": "user@example.org",
    "suggest": {
      "input": [
        "user@example.org",
        "Dr. User Name",
        "User Name"
      ],
      "payload": {
        "display": "Dr. User Name",
        "display_detail": "user@example.org"
      }
    },
    "type": "app_user",
    "profile": {
      "callname": "User Name",
      "fullname": "Dr. User Name"
    }
  }
}
RETRIEVED BY SEARCH:
{"_shards":{"total":5,"successful":5,"failed":0},"all-suggest":[{"text":"User Name","offset":0,"length":9,"options":[{"text":"User Name","score":1.0,"payload"::)
�displayLDr. User Name�display_detailOuser@example.org�}]}]}
@nik9000
Copy link
Contributor

nik9000 commented Dec 15, 2014

Thanks for the reproduction steps. I'll have a look.

@nik9000
Copy link
Contributor

nik9000 commented Dec 15, 2014

Ok, figured it out. I'll have a pull request to fix it in a bit.

@jo-m
Copy link
Author

jo-m commented Dec 15, 2014

very cool ;)

nik9000 added a commit to nik9000/elasticsearch that referenced this issue Dec 16, 2014
When I originally wrote the transform feature I didn't think that the
XContentType of the reencoded source mattered.  It actually matters because
payloads for the completion suggester are stored and returned exactly
as encoded by this XContentType.

This revision changes the transform feature from always reencoding with smile
to always reencoding with the provided XContentType to support the completion
suggester.

Closes elastic#8959
@nik9000
Copy link
Contributor

nik9000 commented Dec 16, 2014

@jo-m OK - fix in pull request sent. To actually get the fix applied you'd have to upgrade to a version of Elasticsearch with the fix merged (assuming it is merged) and reindex the documents. You might also want to look at #8975 which, if we go ahead and fix that too, would make reindexing the documents no longer required.

@jo-m
Copy link
Author

jo-m commented Dec 16, 2014

thank you!

@clintongormley
Copy link

clintongormley commented Dec 17, 2014

thanks @nik9000!

jpountz pushed a commit that referenced this issue Dec 22, 2014
When I originally wrote the transform feature I didn't think that the
XContentType of the reencoded source mattered.  It actually matters because
payloads for the completion suggester are stored and returned exactly
as encoded by this XContentType.

This revision changes the transform feature from always reencoding with smile
to always reencoding with the provided XContentType to support the completion
suggester.

Closes #8959
jpountz pushed a commit that referenced this issue Dec 22, 2014
When I originally wrote the transform feature I didn't think that the
XContentType of the reencoded source mattered.  It actually matters because
payloads for the completion suggester are stored and returned exactly
as encoded by this XContentType.

This revision changes the transform feature from always reencoding with smile
to always reencoding with the provided XContentType to support the completion
suggester.

Closes #8959
jpountz pushed a commit that referenced this issue Dec 22, 2014
When I originally wrote the transform feature I didn't think that the
XContentType of the reencoded source mattered.  It actually matters because
payloads for the completion suggester are stored and returned exactly
as encoded by this XContentType.

This revision changes the transform feature from always reencoding with smile
to always reencoding with the provided XContentType to support the completion
suggester.

Closes #8959
mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015
When I originally wrote the transform feature I didn't think that the
XContentType of the reencoded source mattered.  It actually matters because
payloads for the completion suggester are stored and returned exactly
as encoded by this XContentType.

This revision changes the transform feature from always reencoding with smile
to always reencoding with the provided XContentType to support the completion
suggester.

Closes elastic#8959
mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015
When I originally wrote the transform feature I didn't think that the
XContentType of the reencoded source mattered.  It actually matters because
payloads for the completion suggester are stored and returned exactly
as encoded by this XContentType.

This revision changes the transform feature from always reencoding with smile
to always reencoding with the provided XContentType to support the completion
suggester.

Closes elastic#8959
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants