Elasticsearch: Create Raw Data metric to render raw JSON docs in columns in the new table panel #26233

ivanahuckova · 2020-07-10T11:49:58Z

What this PR does / why we need it:

Follows up on #24521. To be backwards compatible, I have created Raw Document v2 metric to process results as DataFrames and not SeriesList.

Blocker:
When switching between the Raw Document and Raw Document v2 table does not update as new request is not sent (because the request is the same).

Which issue(s) this PR fixes:

Fixes #24163

ivanahuckova · 2020-07-10T11:51:59Z

@marefr @torkelo Could you please check this out and share your thoughts? I have followed up on #24521 and created a new metric as suggested.

Also, do you have any suggestions on how to fix the blocker - when switching between the Raw Document and Raw Document v2 table does not update as new request is not sent (because the request is the same).

torkelo · 2020-07-10T14:01:50Z

not sure I understand why a new query is not issued, you will need to trigger a new query for the result processing to be re-executed

Raw Document v2 is not great. Maybe

Document fields
Raw data

and then change text label (not value) for the old Raw document to Raw document (legacy).

ivanahuckova · 2020-07-10T14:56:57Z

not sure I understand why a new query is not issued, you will need to trigger a new query for the result processing to be re-executed

Figured that out in query_ctrl we are checking if queries are the same. I have added exception for Raw Data and Raw Doocument.

Raw Document v2 is not great. Maybe
Document fields
Raw data
and then change text label (not value) for the old Raw document to Raw document (legacy).

Changed to Raw Data, as I feel it is more clear and for intuitive for users based on Raw Document naming. Also they are right next to each other, so users can easily try it out, when they see Raw document (legacy).

marefr

Have tested this. In general I wonder whether introducing a different Metric is the way to go. Worth consider having an option above or below Size when Raw Document is selected. Maybe a switch/boolean for legacy format that's true if this property is null/undefined and when adding a new query and selecting Raw Document this gets set to false per default. Doing this gives us possibility to also add some tooltip/description of what it means. Thought?

Found some bugs.

Using Raw Data I cannot edit size option:

Comparing with old table panel and Raw Document query it doesn't include _source as a column, but using Raw Data does which feels wrong:

… field as column

ivanahuckova · 2020-07-13T14:22:06Z

Using Raw Data I cannot edit size option

✅ Nice catch! Fixed in 2096cd0

Comparing with old table panel and Raw Document query it doesn't include _source as a column, but using Raw Data does which feels wrong

✅ Removed in 611b692

Have tested this. In general I wonder whether introducing a different Metric is the way to go. Worth consider having an option above or below Size when Raw Document is selected.

Well yeah, that is a really good question and I think I might like the solution that you have just proposed slightly better. I have created a new metric as both you and @torkelo have suggested it in #24521 (comment) and #24163 (comment), so I thought that there was a consensus that this would be the best option. But I try to create another PR with the other solution and we can just choose whichever we like more. But we should make sure that this fix goes to 7.1 as it was promised. 🙂

ivanahuckova · 2020-07-14T11:02:28Z

I have created a new PR for the "switch-idea" #26323.

This reverts commit 611b692.

…h source field as column" This reverts commit 31a9d5f.

marefr

Logs in Explore seems to be broken (works in master):

If this was caused by removing _source column I'm fine with skipping that change if it resolves explore/logs problem

into ivana/elastic-table

ivanahuckova · 2020-07-15T09:31:05Z

@marefr fixed in 3687e94

ivanahuckova · 2020-07-15T09:37:42Z

This reverts commit a102ab2.

aocenas · 2020-07-15T11:50:32Z

public/app/plugins/datasource/elasticsearch/query_ctrl.ts

@@ -51,9 +51,15 @@ export class ElasticQueryCtrl extends QueryCtrl {
  }

  queryUpdated() {
+    // As Raw Data and Raw Document have the same request, we need to run refresh if they are updated
+    const isPossiblyRawDataSwitch = this.target.metrics.some(


Not sure I get this logic here. We are checking only new ones so we do not actually compare to old one. Probably works but seems unintuitive here. Can we just use something similar to what we already have like this.oldTargetMetric and then compare it to new one?

Probably a bit nitpicky, can be skipped if this should be time consuming, but I think it would make more clear what we are actually trying to figure out here.

Will create another PR for this.

All requested changes by Marcus are implemented. PR was reviewed by Andrej as Marcus is busy with other bug that directly affects 7.1 release.

…ns in the new table panel (#26233) * test * WIP: Create v2 version * Update tests, remove conosole logs, refactor * Remove incorrect types * Update type * Rename legacy and new metrics * Update * Run request when Raw Data tto Raw Document switch * Fix size updating * Remove _source field from table results as we are showing each source field as column * Remove _source just for metrics, not logs * Revert "Remove _source just for metrics, not logs" This reverts commit 611b692. * Revert "Remove _source field from table results as we are showing each source field as column" This reverts commit 31a9d5f. * Add vis preference for logs * Update visualisation to logs * Revert "Revert "Remove _source just for metrics"" This reverts commit a102ab2. Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com> (cherry picked from commit 3fd8104)

ivanahuckova added 4 commits July 10, 2020 12:49

test

1088740

WIP: Create v2 version

712c061

Update tests, remove conosole logs, refactor

d61e334

Remove incorrect types

9407731

ivanahuckova requested a review from a team as a code owner July 10, 2020 11:49

ivanahuckova requested review from davkal and removed request for a team and davkal July 10, 2020 11:49

Update type

591d2b0

ivanahuckova requested review from aocenas, torkelo, kaydelaney and marefr July 10, 2020 12:57

ivanahuckova added 3 commits July 10, 2020 16:09

Rename legacy and new metrics

ae0681f

Update

b2b04dd

Run request when Raw Data tto Raw Document switch

b6e7609

ivanahuckova changed the title ~~WIP - Elasticsearch: Fixes raw document query render in table panel~~ Elasticsearch: Fixes raw document query render in table panel Jul 10, 2020

ivanahuckova self-assigned this Jul 13, 2020

ivanahuckova added the datasource/Elasticsearch label Jul 13, 2020

ivanahuckova changed the title ~~Elasticsearch: Fixes raw document query render in table panel~~ Elasticsearch: Create Raw Doc metric to render raw JSON docs in columns in the new table panel Jul 13, 2020

ivanahuckova added the add to changelog label Jul 13, 2020

ivanahuckova added this to Under review in Observability (deprecated, use Observability Squad) via automation Jul 13, 2020

ivanahuckova added this to the 7.1 milestone Jul 13, 2020

marefr requested changes Jul 13, 2020

View reviewed changes

ivanahuckova added 2 commits July 13, 2020 15:23

Fix size updating

2096cd0

Remove _source field from table results as we are showing each source…

31a9d5f

… field as column

marefr and others added 3 commits July 15, 2020 10:45

Merge remote-tracking branch 'origin/master' into ivana/elastic-table

593cc81

Revert "Remove _source just for metrics, not logs"

a102ab2

This reverts commit 611b692.

Revert "Remove _source field from table results as we are showing eac…

957cf7d

…h source field as column" This reverts commit 31a9d5f.

marefr previously requested changes Jul 15, 2020

View reviewed changes

ivanahuckova modified the milestones: 7.1, 7.2 Jul 15, 2020

ivanahuckova added 3 commits July 15, 2020 11:24

Add vis preference for logs

e225b76

Update visualisation to logs

3687e94

Merge branch 'ivana/elastic-table' of https://github.com/grafana/grafana

f474ae0

into ivana/elastic-table

ivanahuckova modified the milestones: 7.2, 7.1 Jul 15, 2020

ivanahuckova added the cherry-pick needed label Jul 15, 2020

ivanahuckova requested a review from marefr July 15, 2020 09:38

Revert "Revert "Remove _source just for metrics""

a4111c4

This reverts commit a102ab2.

ivanahuckova mentioned this pull request Jul 15, 2020

Elasticsearch: fixes raw document query render in table panel #24521

Closed

aocenas approved these changes Jul 15, 2020

View reviewed changes

ivanahuckova merged commit 3fd8104 into master Jul 15, 2020

Observability (deprecated, use Observability Squad) automation moved this from Under review to Done Jul 15, 2020

ivanahuckova deleted the ivana/elastic-table branch July 15, 2020 13:20

dprokop mentioned this pull request Jul 16, 2020

Cherry picks for 7.1.0 #26368

Merged

dprokop removed the cherry-pick needed label Jul 16, 2020

dprokop mentioned this pull request Jul 16, 2020

Update changelog for 7.1.0 #26373

Merged

ivanahuckova changed the title ~~Elasticsearch: Create Raw Doc metric to render raw JSON docs in columns in the new table panel~~ Elasticsearch: Create Raw Data metric to render raw JSON docs in columns in the new table panel Aug 10, 2020

gabor mentioned this pull request Jan 26, 2023

elasticsearch: mark "raw document (legacy)" mode as deprecated #62197

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Elasticsearch: Create Raw Data metric to render raw JSON docs in columns in the new table panel #26233

Elasticsearch: Create Raw Data metric to render raw JSON docs in columns in the new table panel #26233

ivanahuckova commented Jul 10, 2020 •

edited

ivanahuckova commented Jul 10, 2020

torkelo commented Jul 10, 2020

ivanahuckova commented Jul 10, 2020

marefr left a comment

ivanahuckova commented Jul 13, 2020 •

edited

ivanahuckova commented Jul 14, 2020 •

edited

marefr left a comment •

edited

ivanahuckova commented Jul 15, 2020

ivanahuckova commented Jul 15, 2020

aocenas Jul 15, 2020

ivanahuckova Jul 15, 2020

Elasticsearch: Create Raw Data metric to render raw JSON docs in columns in the new table panel #26233

Elasticsearch: Create Raw Data metric to render raw JSON docs in columns in the new table panel #26233

Conversation

ivanahuckova commented Jul 10, 2020 • edited

ivanahuckova commented Jul 10, 2020

torkelo commented Jul 10, 2020

ivanahuckova commented Jul 10, 2020

marefr left a comment

Choose a reason for hiding this comment

ivanahuckova commented Jul 13, 2020 • edited

ivanahuckova commented Jul 14, 2020 • edited

marefr left a comment • edited

Choose a reason for hiding this comment

ivanahuckova commented Jul 15, 2020

ivanahuckova commented Jul 15, 2020

aocenas Jul 15, 2020

Choose a reason for hiding this comment

ivanahuckova Jul 15, 2020

Choose a reason for hiding this comment

ivanahuckova commented Jul 10, 2020 •

edited

ivanahuckova commented Jul 13, 2020 •

edited

ivanahuckova commented Jul 14, 2020 •

edited

marefr left a comment •

edited