Question: Unable to get any datapoints from elasticsearch 2.2

[asitk]

Server: Graphana v2.6.0
Datasource :Elasticsearch: v2.2
Ubuntu 14.04

Noob issue. I created a dashboard with a Elasticsearch datasource. I set the version to 2.X. Here is my document

{
    "took": 26,
    "timed_out": false,
    "_shards": {
        "total": ? 5,
        "successful": ? 5,
        "failed": ? 0
    },
    "hits": {
        "total": ? 21473,
        "max_score": ? 1.0,
        "hits": [{
            "_index": "logs",
            "_type": "line_datanode",
            "_id": "AVNfKu3DElIP3IA713se",
            "_score": ? 1.0,
            "_source":

            {
                "timestamp": "1457590530104",
                "text": "\"2016-03-10 02:57:50,034 INFO  DataNode.clienttrace (BlockReceiver.java:run(909)) - src: /127.0.0.1:52717, dest: /127.0.0.1:50010, bytes: 4397, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1728807816_1, offset: 0, srvID: DS-492720592-127.0.1.1-50010-1457080791566, blockid: blk_-8983615958535416445_5688, duration: 338170\""
            }
        }]
    }
}

My query from Grafana is not returing any results. I am simply trying to plot a count all log records hence i just created a basic dashboard with no specific lucene query

POST + Response

{
    "search_type": "count",
    "ignore_unavailable": true,
    "index": "logs"
} {
    "size": 0,
    "query": {
        "filtered": {
            "query": {
                "query_string": {
                    "analyze_wildcard": true,
                    "query": "*"
                }
            },
            "filter": {
                "bool": {
                    "must": [{
                        "range": {
                            "@timestamp": {
                                "gte": "1457570145648",
                                "lte": "1457591745649",
                                "format": "epoch_millis"
                            }
                        }
                    }]
                }
            }
        }
    },
    "aggs": {
        "2": {
            "date_histogram": {
                "interval": "10s",
                "field": "@timestamp",
                "min_doc_count": 0,
                "extended_bounds": {
                    "min": "1457570145648",
                    "max": "1457591745649"
                },
                "format": "epoch_millis"
            },
            "aggs": {}
        }
    }
}
Response: {
    "responses": [{
        "took": 4,
        "timed_out": false,
        "_shards": {
            "total": 5,
            "successful": 5,
            "failed": 0
        },
        "hits": {
            "total": 0,
            "max_score": 0.0,
            "hits": []
        },
        "aggregations": {
            "2": {
                "buckets": []
            }
        }
    }]
}

I can see results when i query in elastic-search directly

TIA for any pointers!

[torkelo]

you timestamp field is named "timestamp" , but in your data source config you have specified "@timestamp" (notice the at sign)

[asitk]

Thanks!, I tried using just the timestamp before. This is the error i received

{
    "root_cause": [{
        "type": "illegal_argument_exception",
        "reason": "Expected numeric type on field [timestamp], but got [string]"
    }],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [{
        "shard": 0,
        "index": "logs",
        "node": "WjuujlQ_Qhu7ySDB5SN90g",
        "reason": {
            "type": "illegal_argument_exception",
            "reason": "Expected numeric type on field [timestamp], but got [string]"
        }
    }]
}

the timestamp is in millis. Looks like elasticsearch does not like the way the jsonobject is inserting the timestamp

{
    "timestamp": "1457590806299",
    "text": "\"2016-03-10 11:50:06,298 INFO  datanode.DataNode (FSDatasetAsyncDiskService.java:deleteAsync(143))  - Scheduling blk_ - 7567441152900287140 _6818 file / app / hadoop / tmp / dfs / data / current / subdir0 / blk_ - 7567441152900287140
    for deletion\""
}

[torkelo]

You need to fix you elasticsearch document timestamp, either by passing it as a number but I also think you need a custom index mapping informing elasticsearch how to interpret the timestamp

[asitk]

yup, works now after explicit indexing. Many thanks for the pointers :-)

[sanilveeravu]

hi asitk, can you guide on how you did the explicit indexing. I am getting the same error where the value is correct from logstash as a numeric but throws numeric exception when trying to plot the value in grafana

[sanilveeravu]

@asitk hi asitk, can you guide on how you did the explicit indexing. I am getting the same error where the value is correct from logstash as a numeric but throws numeric exception when trying to plot the value in grafana

[asitk]

Check out this doc https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html

[Ju2ender]

I have same problem too.
The key is timestamp in Elasticsearch just a string ("timestamp": "1488004637612") or a long value ("timestamp": 1488004637612), it's not a date!
So in Grafana's dashboard has no datapoints: x-asix is date type.

Solution

Timestamp's type should be "date", format should be "epoch_millis".

1. Get your type mapping: send GET request to http://[your server IP]:9200/[index]/[type]/_mapping

2. Delete old index: send DELETE request to http://[your server IP]:9200/[index]

3. Create index: send PUT request to http://[your server IP]:9200/[index]

4. Create timestamp's mapping: send PUT request to http://[your server IP]:9200/[index]/_mapping/[type]:

    {
        "properties": {
            "timestamp": {
                "type": "date",
                "format": "epoch_millis"
            }
        }
    }