Elasticsearch: Allow users to override index pattern per query

[simianhacker]

What this PR does / why we need it:

This PR adds the ability to override the index pattern per Elasticsearch query. This would allow users to override the index pattern per query and have the added benefit of supporting multiple index patterns on the same visualization.

Which issue(s) this PR fixes:

Fixes #20674

Special notes for your reviewer:

I put this PR together as a possible solution for #20674, I'm open to other ideas for solving this problem. I've chosen this as my preferred solution because of the low surface area (code wise) and it provides the maximum benefit because it allows users to mix index patterns in the same visualization.

[dagwieers]

This is exactly what we need. Currently we have about 10 different Elastic datasources with the only difference being the index pattern. And we are adding more each few months.

Using the "Outer join" transformation we can enrich our query data with information from another "source".

[halfa]

That would be a welcome addition, opening the possibility to mix indices in a single dashboard

[dagwieers]

@simianhacker Just a heads-up that this branch now has conflicts

[simianhacker]

@Elfo404 This has been refactored to the React PR and is ready for review. I also fixed an issue where the Alias input field wasn't being populated correctly.

[dagwieers]

@simianhacker It is obvious, but important to realize that with this change access to indices is no longer controlled from the data source. So it is important to have Grafana's access properly fine-tuned in Elastic itself.

Which you should be doing anyhow, but may not be the case for everyone running both together today. I think we have to mention this specifically in the release notes when this is merged.

Also, we should probably improve the Elasticsearch data source documentation with guidance on what privileges would work for Grafana. Which is just read on the indices listed:

indices:
- names:
  - your-index-*
  - your-other-index-*
  privileges:
  - read
``

[Elfo404]

@dagwieers good point. There's also an existing issue regarding this: #5880.

with this change access to indices is no longer controlled from the data source.

That's not completely true as currently there are no checks on which indices can be accessed by using the proxy endpoint, that's actually why this PR works.

IMHO it should not be Grafana's (or this plugin's) responsibility to enforce this kind of "safety" measures since it will only provide a false sense of security. What do you think?

Anyway, I agree that the documentation will benefit from having a bit more information on that matter.

[stale]

This pull request has been automatically marked as stale because it has not had activity in the last 2 weeks. It will be closed in 30 days if no further activity occurs. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!

[dagwieers]

Please do not forget about this one 😉

[henrikno]

This is going to reduce the number of datasources we have to create by at least 50x.

[simianhacker]

@henrikno That's awesome to hear.

[stale]

This pull request has been automatically marked as stale because it has not had activity in the last 2 weeks. It will be closed in 30 days if no further activity occurs. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!

[hikerspath]

This was marked for v8, any updates on when this could go live? Would be SUPER useful to be sure. I could reduce about 35 datasources to 1

[CLAassistant]

All committers have signed the CLA.

[danielmotaleite]

This is really needed!
any hint when this will be merged

[hikerspath]

Likewise, very curious...

[Elfo404]

Sorry for the late update, this one slipped through a bit :)

we have some refactoring planned for the ES data source, moving most of the logic to the BE which will hopefully solve quite a few pending issues (especially when it comes to alerting).

That said, this PR is actually conflicting with some of the decisions and concerns that arose when discussing the feature request, so (also together with @simianhacker ) we decided to pause the work being done here and pick it up as soon as possible once the migration is completed.

I understand it's not the answer you were hoping for, but I want to confirm this PR has not been forgotten, we just need to do some prep work before moving forward.

[stale]

This pull request has been automatically marked as stale because it has not had activity in the last 2 weeks. It will be closed in 30 days if no further activity occurs. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!

[grafanabot]

This pull request has been automatically closed because it has not had activity in the last 2 weeks. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!

[cjabrantes]

Hi,

Can someone confirm this one? i was needed this, but seems its not available yet.

Can this also be used with remote cluster?

Thanks,