"No results found" in TSVB Markdown panels that use .kibana index pattern with no time field

[GrahamHannington]

Kibana version: 7.13.2

Elasticsearch version: 7.13.2

Server OS version: Linux Ubuntu 20.04

Browser version: Google Chrome Version 91.0.4472.77 (Official Build) (64-bit)

Browser OS version: Microsoft Windows 10 Enterprise, Version 1909 (OS Build 18363.1556)

Original install method: Docker image (sebp/elk)

Describe the bug: In 7.13.1 and 7.13.2 (and possibly earlier), TSVB Markdown panels that use the .kibana index pattern (why? see "Any additional context") show "No results found", where, as recently as 7.11.2, they showed content.

Steps to reproduce:

1. Create .kibana as an index pattern in Kibana.
2. Create a TSVB Markdown visualization with static Markdown content only (no expressions to refer to search results).
3. Set the index pattern to .kibana, with no time field.

Expected behavior: Such panels should show content, regardless of time range or filters, as they did in 7.11.2.

Screenshots:

This used to work (in 7.11.2):

(I'm deliberately not including custom-CSS-formatted Markdown content in this screenshot, because it contains proprietary terms.)

Any additional context:

I've developed a set of Kibana dashboards. At the top of each dashboard is a TSVB Markdown panel that contains a row of links to the dashboards. All of the dashboards refer to the same visualization; the same single saved object in the visualization library.

If the "standard" Markdown viz offered custom CSS, I wouldn't use a TSVB viz for this purpose. See issue #13313.

If Kibana offered a better way to present links to related dashboards, I wouldn't use a Markdown panel at all for this purpose. See issue #99740.

As it is, I have to use a TSVB viz, and that means specifying an index pattern, even for static Markdown content.

Filters that a user creates in one dashboard persist when they click a link to another dashboard. This is deliberate and desirable. However, the dashboards use different index patterns. Some fields are common across these index patterns, some aren't. So I've switched on the Kibana advanced setting courier:ignoreFilterIfFieldNotInIndex.

Problem: The index pattern specified in the TSVB Markdown panel (of dashboard links) that is used in all of these dashboards can interfere with the behavior of that setting. For details, see the Elastic Kibana discussion forum topic "Omit pinned filters that are inappropriate to a dashboard?".

Workaround (worked in 7.11.2): Specify .kibana as the index pattern for the TSVB.

Problem: This workaround no longer works in 7.13.1 and 7.13.2.

I want to present panels with static Markdown content formatted using custom CSS. That is all.

I don't want those panels to be affected by filters or time range. And I don't want such panels—in particular, the index patterns specified in TSVB Markdown panels—to affect the behavior or appearance of filters (again, for details, see "Omit pinned filters that are inappropriate to a dashboard?").

My other use cases for TSVB Markdown panels include "tiles" on a "home"/entrypoint dashboard for this set of dashboards. I don't want those tiles to be sensitive to filters or time range. They should always show their content.

I'm happy to acknowledge that specifying .kibana as the index pattern for these panels was a dirty workaround. But it used to work, and now it doesn't, and in 7.13.2 I'm left staring at "No results found" where in 7.11.2 I used to see custom-CSS-formatted content.

[stratoula]

@GrahamHannington if you go to the panel options, which is the status of this setting (Use only kibana indexes)?

If it is off, can you enable it and tell me if you see the .kibana on the dropdown?

[GrahamHannington]

In my Kibana 7.13.2 instance, that "Use only Kibana index patterns" setting is off. (I'm sure you know this, but just so you know I know this 🙂: in Kibana 7.11.2, where I am successfully using .kibana as an index pattern in this context, there is no such setting on the Index pattern dropdown.)

When I enable it, I see .kibana in the dropdown:

I quite deliberately created an index pattern in Kibana for .kibana. I originally did that because I wanted to visualize data in that index relating to saved objects such as tags: for details, see the Elastic Kibana discussion forum topic "Create visualizations based on Kibana saved objects, including tags?".

Only later, after I'd already experimentally created a .kibana index pattern, did it occur to me to try using that index pattern as a workaround for the issues that I was experiencing with TSVB Markdown panels, as described in the discussion forum topic "Omit pinned filters that are inappropriate to a dashboard?".

[stratoula]

Interesting case indeed! I have never tried that :D

Just FYI (to explain the change that you see with the dropdown), on 7.13.0 we introduced the dual mode on TSVB #91367

This enables TSVB to work both for indexes and index patterns. Note here that it is recommended to work with index patterns. Working with index patterns unlocks many cool features such as runtime fields, URL drilldowns and more features to come.

The question is (as I have never used .kibana as an index pattern), if you change the mode and select the .kibana index pattern, does it render the markdown?

[stratoula]

Yes I have tested it, it should work for markdown. @GrahamHannington thanx for reporting this, we will try to fix t for 7.13.3

@alexwizp it seems that we missed markdown.

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[GrahamHannington]

@stratoula asked:

if you change the mode and select the .kibana index pattern, does it render the markdown?

Unfortunately not (in my 7.13.2 instance).

[GrahamHannington]

I would be happy if 7.13.3 "unbroke" my existing TSVB Markdown panels that use the .kibana index pattern for this purpose; happier still if, in the near future, Kibana rendered this "workaround" obsolete. (Frankly, to me, although I'm happy it worked, I consider it to be an ugly kludge.)

[stratoula]

@stratoula asked:

if you change the mode and select the .kibana index pattern, does it render the markdown?

Unfortunately not (in my 7.13.2 instance).

Interesting, I just created the .kibana index pattern with updated_at as a timefield and my markdown works 🤔

We should def fix the problem with markdown and no timefield present bug. Now about the .kibana problem, I am not sure if it is related to TSVB tbh. Using .kibana as an index pattern is quite hacky, I agree with you

[GrahamHannington]

@stratoula wrote:

Interesting, I just created the .kibana index pattern with updated_at as a timefield and my markdown works 🤔

Back in 7.11.2, when I originally created the .kibana index pattern, I didn't "select a primary time field for use with the global time filter", because I wasn't interested in using time-based data in that index pattern. And then, later, when I used that index pattern for a TSVB Markdown panel, I definitely didn't want it to be sensitive to the global time filter; I didn't want the global time filter determining whether or not the Markdown panel displayed its contents, depending on, say, whether Kibana saved objects had been updated in that time range. Hence, I deliberately did not specify a time field in the TSVB panel.

Now, in 7.13.2, in the Kibana UI, I went to Stack Management > Kibana: Index Patterns, and tried to find a way to specify that time field on the existing index pattern. I couldn't find a way to do that, so I deleted the index pattern, then re-created it, this time selecting updated_at as the time field.

I should have seen the next part coming, but it's late (11 PM local time), and I'm tired: when I returned to edit a TSVB Markdown viz that used that .kibana index pattern, the viz showed the error "Could not locate that index-pattern (id: dab70220-9cc1-11eb-b38d-7b8e5ab9c939)". The "click here to re-create it" simply took me to the index patterns page, without any context that might supply that ID. And the Kibana UI didn't offer a menu option to edit the visualization, probably because of that error.

I can't see a way in the Kibana UI to edit the JSON definition of the index pattern to change its ID (if that's even possible) to match the original that I deleted. I could look into deleting the index pattern again, and this time re-creating it via an API call with a specific ID (if that's even possible), but... it's late. I'm giving up for the night.

In case you're wondering: I haven't completely destroyed my environment doing this. I'm performing this work in a 7.13.2 "sandbox" Elastic Stack instance, completely separate from my still-working 7.11.2 instance and its saved objects.

[alexwizp]

@GrahamHannington let me clarify, you are using TSVB Markdown to display some static styled text , right? I want to make it clear that all visualizations in TSVB are based on date_histogram / auto_date_histogram aggregation. This means that the value for the field must be filled in.

I checked how the TSVB behaves in version 7.12.1 and yes instead of No Data user see some empty values. It was an issue with hardcoded default field name which was fixed here - DEFAULT_TIME_FIELD was removed.

It doesn't make much sense to return this hardcoded value because it will confuse the user more.

[GrahamHannington]

I wrote:

And the Kibana UI didn't offer a menu option to edit the visualization, probably because of that error.

The dashboard didn't allow me to edit the viz with that error, but I could edit it independently of the dashboard, via the saved objects list.

Tired and. hence, a little reckless, I used the Kibana UI to "inspect" the underlying JSON definition of the TSVB Markdown viz, and replaced the original index pattern ID with the new ID.

That worked: when I returned to a dashboard containing that viz, the Markdown content appeared! And, when I edited the viz from that dashboard, the Time field dropdown had been populated with the updated_at field:

Moreover, the Markdown content now displays.

Even better: the TSVB Markdown panel, even with that time field specified, doesn't seem to be affected by the global time filter. I specified a time filter spanning two days in 2001: while the "real" data-based visualizations displayed the message "No results found", the TSVB Markdown panel still displayed its contents.

This is all good news.

Do you have any tips on adding updated_at as the time field to an existing .kibana index pattern saved object, so I don't hit that "different ID" issue?

[GrahamHannington]

you are using TSVB Markdown to display some static styled text , right?

Yes.

[alexwizp]

ok ... i think you can create a static Runtime Field. But I want to remind that this is not the correct use of TSVB and Runtime Fields.

1. Open Stack Management -> Index patterns -> .kibana

2. Press Add Field

3. Create a static Date field without value
   

4. Update your TSVB visualization to use that field

Please close that issue if my suggestion help you.

[GrahamHannington]

@stratoula wrote:

We should def fix the problem with markdown and no timefield present bug.

Is this still your position?

I ask because @alexwizp later wrote:

It doesn't make much sense to return this hardcoded value because it will confuse the user more.

and I'm unsure whether that statement by @alexwizp affects (supersedes) the earlier statement by @stratoula.

If you intend to fix the problem (with Markdown and no time field present) then I can simply wait for the fix without changing my saved objects.

If you are not going to fix the problem, then I will need to change my saved objects.

I'm not annoyed or anything (on the contrary, I sincerely appreciate your time): I'm just not clear on your position.

To recap:

• In 7.11.2, I have several TSVB Markdown panels, containing static Markdown only, that specify .kibana as the index pattern, with no time field. That works: the panels display their content.
• In 7.13.1 and 7.13.2 (possibly earlier), those panels display "No results found".
• In 7.13.2 (and possibly earlier), if I re-create the .kibana index pattern and, and at the time of creating that index pattern, "select a primary time field for use with the global time filter", such as updated_at, and use that new index pattern, then the problem goes away: the panels again display their content.

[GrahamHannington]

@alexwizp ,

While I sincerely appreciate the suggestion, I strongly dislike the combination of creating a runtime field and having to edit each TSVB Markdown viz to specify that runtime field. In particular, creating a runtime field extends the scope of the existing kludge.

I would much rather specify an existing field in the index pattern, such as updated_at, as the timeFieldName for the index pattern.

As I've described, I've looked in vain in the Kibana UI for a way to directly edit the underlying JSON definition of an index pattern, in the same way that you can "inspect" and then edit the JSON of a visualization. So I've used the corresponding Kibana Update index pattern API:

curl -X POST -H "Content-Type: application/json" -H "kbn-xsrf: true" -d @update-index-pattern-with-time-field.json myelastichostname:15601/s/sandbox/api/index_patterns/index_pattern/dab70220-9cc1-11eb-b38d-7b8e5ab9c939

where the file @update-index-pattern-with-time-field.json contains:

{
  "index_pattern": {
    "timeFieldName": "updated_at"
  }
}

That appears to have worked. (I originally thought it didn't, but that was user—my—error: I'd omitted the -H "Content-Type: application/json" option from the POST.)

[stratoula]

@GrahamHannington what I meant about the markdown and no timefield present:
TSVB is a timeseries visualization, it is developed to be used for data with a time field. Before 7.13 (that we introduced the dual-mode), the user could create a TSVB visualization without applying a timefield because silently it would apply the time field of the index pattern, even though the user didn't apply it. Now when a user upgrades his TSVB markdown viz to 7.13 and he hasn't applied a time field, the Markdown returns no data and the user has to apply the timefield in order to see data. This is something that we should fix.

Your case is a bit different. You had created an index pattern without a timefield and it worked but this is not how TSVB should work. As Alexey said TSVB is based on date_histogram / auto_date_histogram aggregation. So as we have a workaround for your case I suggest only fixing the bug that I mentioned above.

I hope I clarified what I meant :D

[GrahamHannington]

Hi @stratoula ,

Understood, thanks for the clarification.

For now, then, I'll go with the workaround: update the .kibana index pattern to use the updated_at field as its timeFieldName, and ensure my TSVB Markdown visualizations refer to that time field.

I look forward to not having to use TSVB to present custom-styled static Markdown. (For example, a fix for #13313, and preferably also #99740.)

Thanks for your help!

[nicpenning]

@GrahamHannington if you go to the panel options, which is the status of this setting (Use only kibana indexes)?

If it is off, can you enable it and tell me if you see the .kibana on the dropdown?

Any idea why this is the default?

This broke most of our TSVB visualizations because filebeat-* is a Kibana index pattern not an Elasticsearch index.

[alexwizp]

@nicpenning Yes, we now have a split between ES and Kibana indexes. From my point of view, this solved the most confusing part of TSVB and gave us the opportunity to add some features that are only available for Kibana indexes.
I just curious why you don't see any data if you select a ES index ...

[nicpenning]

I don't know why either 😞 that's what's confusing.

[alexwizp]

ok... is it possible to share somehow your visualization configuration (screen or exported saved object)?. We can try to reproduce it locally based on filebeat index.

[nicpenning]

I can try and do that for you.

[nicpenning]

@alexwizp could you try the tsvb/dashboard from here?

https://github.com/nicpenning/Nessus-ES