fixes #4693 and fixes #4526: add Nested type bucket aggregation support for elasticsearch datasource

[arcolife]

EDIT: This PR was accidentally merged by the maintainer. Follow up at #4694 (comment)

---

This adds support for querying nested fields with elasticsearch as datasource; built on top of PR #4527

fixes #4693

Cheers.

[arcolife]

@torkelo Would look something like this. Your thoughts? (I'll change the Nested Query part to another dropdown list).

Also the arrangements of Group By / Then By, would matter, since date_histogram has to be the outermost agg. I first included nested options under Metric as a new option "Average (nested)" but then that wouldn't make sense, so I changed it to show as an option under bucket aggs.

[arcolife]

A query generated from this would look something like:

..
"aggs": {
    "4": {
      "date_histogram": {
        "interval": "1m",
        "field": "_timestamp",
        "min_doc_count": 0,
        "extended_bounds": {
          "min": "1434982853565",
          "max": "1434984034923"
        }
      },
      "aggs": {
        "2": {
          "nested": {
            "path": "disk"
          },
          "aggs": {
            "nested_aggs": {
              "filter": {
                "term": {
                  "disk.disk-device": "dev8-0"
                }
              },
              "aggs": {
                "1": {
                  "avg": {
                    "field": "disk.wr_sec"
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

And sample results:

..
"aggregations": {
    "4": {
        "buckets": [
            {
                "2": {
                    "doc_count": 9,
                    "nested_aggs": {
                        "1": {
                            "value": 90.4
                        },
                        "doc_count": 1
                    }
                },
                "key_as_string": "2015-06-22T14:20:00.000Z",
                "key": 1434982800000,
                "doc_count": 1
            }..
..

Notice extra levels of hierarchy in buckets.

[portante]

@arcolife, can you rebase this work?

[arcolife]

rebased. Thanks!
Also fixed a bug from one of the earlier commits of master. ref: a1faaa1#commitcomment-17295710

[arcolife]

added unit test for nested aggs: https://github.com/grafana/grafana/pull/4694/files#diff-c0af25ef637ff34b3c81f7ff8b78243cR122

@torkelo Please review. Thanks!

[arcolife]

@torkelo hey could you do a quick usability review and let me know what to change? I'd be glad if this gets merged sooner; really need this feature in official.. thanks!

[concaf]

+1 @arcolife, it would be very useful for our project too!

[krishnakalyan3]

+1 @arcolife Looks like a good feature.

[arcolife]

@torkelo Sarjitsu is a project powered by Grafana, where I wanna make use of the nested docs feature and it would really cool to have it integrated. I've blogged about it here: http://arcolife.github.io/blog/2016/06/06/sarjitsu-a-project-on-visualizing-your-systems-activity/

Would appreciate your quick feedback, so we could take this further. I was thinking we could release a separate build with this feature, and put it out there for testing. Thanks!

[arcolife]

@torkelo @alexanderzobnin This patch has been in a limbo since we last worked on #6043. Could you guys review this once? I've rebased to get rid of conflicts.
Let me know if there's something wrong with it and I could update this quickly. We're eagerly waiting for this feature, since Kibana doesn't do nested docs either (IIRC). Thanks.

[arcolife]

Ok CI check 1 failed, 1 passed. let me look at the failing checks and get back to you with updates.. (a lot has changed in grafana since last year). But do let me know when you would have time to review this.

[torkelo]

sorry, merged this by mistake (to many PR tabs open). Reverted the merge.

[arcolife]

@torkelo I see this as merged, while this was reverted. Do I need to reopen this patch in another PR or can this be resurrected?

[torkelo]

it is not possible to reopen a merged PR (that was later reverted)

[arcolife]

ok i'll add another PR then, piggy backing on this one.

[arcolife]

@torkelo ^ turns out I can't really add another PR because github shows me "This branch is even with grafana:master." when I click https://github.com/arcolife/grafana/tree/nested_agg_query due to the accidental merge on master earlier. Any clues on how we can get this back on track? thanks.
cc/ @portante

[arcolife]

okay so i've added a new PR mirroring the patches from this one. WIP now at #7863

[carlosvega]

Is this available in any Grafana release? How is it doing?

[lephix]

+1

[killpanda]

+1

[ali-bugdayci]

+++1

[arcolife]

@ali-bugdayci @killpanda @lephix @carlosvega @rburkhonov @gsaslis for updates on this, kindly refer to #7863 (comment)

Looking for feedback! thanks.

[ameed93]

@arcolife : could you please share the docker image if you have it?

[arcolife]

@ameed93 no docker image with me. After this PR was "accidentally merged", I had another one #7863 -> rebased a couple times and needed reviews from the devs which it never got.

[ameed93]

@arcolife: it's very sad how they treat this feature, its a very important one, I decided to create my custom graphs!
Just a question: can I build a docker image from #7863? since I tried to do that with this branch but that didn't work!

PS: I think if you publish #7863 to get more +1's, maybe they will take a look.

[arcolife]

@ameed93 sure go ahead (not so sure it'll work as expected, and then you'd have to go backwards with corresponding ES versions as well I suppose). Also I don't think that PR is complete since I was waiting for reviews before refactoring further. But I'm interested in the outcome nonetheless.

most of the folks have ended up flattening their nested structes in ES as a workaround. lol

not sure about state of affairs right now with Grafana if it has an editable ES query box that supports the nested types. It'll be harder for sure than UI boxes.

To that end, my work on nested fields autocomplete in query searh was wrapped around in #6043 and merged, so it shouldn't be too difficult to build a query, hopefully. Also #3772 and #4526