Dictionary Order

[jeffsteinmetz]

Has anybody found an intuitive way to enforce the dictionary order used when creating a document? (I know about OrderedDict in python 2.7 - but that seems overkill for the elasticsearch use case - and I don't know if it even works with this ES Python client).

for example, indexing a document using python as:

doc = { "a":"a", "b":"b", "c":"c" }

doc may actually get sorted by python in any random order, and looking inside ElasticSearch, you may get

 { "a":"a", "c":"c", "b":"b" }

[honzakral]

Why do you need it in sorted order? Imho you only care about order when printing things out so I'd suggest you sort there.

Please let us know your use-case why you'd want a sorted dict, thanks!

[jeffsteinmetz]

Totally agree that within elastic search, order is irrelevant.
It is purely a usability desire - such as when viewing the raw json document in API call results, or elasticsearch head.

I also realize I could avoid using straight Python dictionaries and just build out the raw JSON string (albeit a little more brittle and less elegant looking code...)

During debugging, having a human readable document like this (the way it was programmed on the way in):

    "contact_first_name" : "kim",
    "contact_last_name" : "smith",
    "address":"555 main street",
    "address_city":"Minneapolis" ...

Is much easier than a scrambled document, that may have 20 to 30 fields, all in a random(ish) order. first_name may be 10+ fields away from last_name

    "address":"555 main street",
    "contact_first_name" : "kim",
    "address_city":"Minneapolis",
    "somefield1":"data1",
    "someotherfield2":"data2",
    "contact_last_name" : "smith",

[honzakral]

Unfortunately that is not possible with elasticsearch due to the nature of the data format. I'd suggest either having a debug print in python that will sort it for you when outputting the dictionary or a simple JS that will do the same for head/kopf/sense/...

[jeffsteinmetz]

(note: ... I'm just using Python as data cleaner a bulk loader into ES. Not for any retrieval.)

Perhaps you mean it is not possible in 'elasticsearch-py'. ?
Making sure I understand what you mean by "it is not possible".

It's actually possible with elasticsearch. This curl command writes the document as is (and will read back exactly as written). Thanks again - mainly trying to get clarification.

curl -XPUT 'http://localhost:9200/indexname/test/1' -d '{
    "contact_first_name" : "kim",
    "contact_last_name" : "smith",
    "address1":"555 main street",
    "address_city":"Minneapolis",
    "post_date" : "2009-11-15T14:12:12",
    "message" : "trying out Elasticsearch"
}'

[honzakral]

What elasticsearch does is it stores the json exactly - any client though will probably deserialize the json to it's native datastructure - some form of hash - that is typically unsorted.

If you really want to control how it looks in elasticsearch you can produce the json yourself, elasticsearch-py will then just pass it along, unless you are using the bulk helpers. In that case what you can do is provide elasticsearch-py with your own json serializer that will produce json to your specification - simply pass an object with the same signature as https://github.com/elasticsearch/elasticsearch-py/blob/master/elasticsearch/serializer.py#L8-L18

Hope this helps

[jeffsteinmetz]

I would be interested in learning how to make a serializer that works with an OrderedDict in python.

Yesterday I arrived at the same conclusion, and did pretty much as you suggested (building the JSON string manually). It isn't pretty but it works.
Here is a pseudo sample in Python. It assumes you want to control the field names, and you know the order of the fields in the CSV that you are bulk loading.

#!/usr/bin/env python

import csv
from elasticsearch import Elasticsearch
from elasticsearch.helpers import bulk, streaming_bulk

# change to your ES IP address
es = Elasticsearch(["10.1.1.1:9200"])

index_name = "myindex"
document_type = "contacts"

es.indices.create(
    index=index_name,
    body={
      'settings': {
        'number_of_shards': 5,
        'number_of_replicas': 1,
      }
    },
    # ignore already existing index
    ignore=400
)

reader = csv.reader(open("contacts.csv", "rb"))

j = 0
bulksize = 5000
actions = []

#skip first line with headers
reader.readline()

for row in reader:

    try:
        # generate source string from OrderedDict
        # this is kind of a hack, just so that document field order is maintained
        # in elastic search, similar to a curl PUT command sending a string.
        # elasticsearch-py uses a dictionary by default, but a python dictionary  
        # internally sorts its keys (psudo)randomly, so the docs come out in strange orders.        
        oSource = OrderedDict([
                    ("company_name", row[1]),
                    ("company_name_alt",row[2]),
                    ("contact_first_name",row[3]),
                    ("contact_middle_initial",row[4]),
                    ("contact_last_name",row[5]),
                    ("address_street",row[6]),   
                    ("address_city",row[7]),
                    ("address_state_province_abb",row[8]),
                    ("address_zipcode",row[9])
                    ])

        source = ""
        # build the source string manually
        # the last string will have a trailing comma
        # but since we tag on ID at the end in the action = {} below, comma is ok
        for k,v in oSource.items():
            source += '"' + k + '":"' + str(v) + '",'

        action = {
            "_index": index_name,
            "_type": document_type,
            "_id": row[0],
            "_source" : "{" +
                    source +
                    '"id":' + row[0] 
                +"}" 
            }
    except:
        pass

    if (action != ""): actions.append(action)
    j += 1

    if (j >= bulksize):
        bulk(es, actions)
        del actions[0:len(actions)]
        j = 0

if (len(actions) > 0):
    bulk(es, actions)
    del actions[0:len(actions)]