Exporting and importing dashboards with index pattern IDs

[cwurm]

At some point, Kibana changed how it exports dashboards (see comments here and here). Where previously, the export would contain a searchSourceJSON like this:

"searchSourceJSON": {
    "filter": [],
    "index": "packetbeat-*",
    "query": {
        "query_string": {
        "analyze_wildcard": true,
        "query": "*"
        }
    }
}

It is now (7.0 at least):

"searchSourceJSON": {
    "filter": [],
    "index": "dfce2b60-18c7-11e9-9094-c50574723088",
    "query": {
        "language": "kuery",
        "query": ""
    }
}

index is the ID of the Kibana index pattern object, so different for each instance.

This presents us with a problem when distributing dashboards with our Beats. Where previously we could just rely on e.g. the packetbeat-* index pattern to be present already when importing a dashboard and used automatically, with IDs this does not work anymore. We don't know the existing IDs, and we don't want to import a new index pattern every time we import a dashboard.

Some options:

1. Since Kibana still seems to accept the old format, we could modify our dashboard exporter to overwrite the index field with the pattern. I'm not sure how long that would work though, I assume at some point Kibana might want to deprecate/remove specifying the search pattern in that field.
2. We could modify our dashboard importer to query first for the presence and ID of the specific index pattern (e.g. packetbeat-*) and then overwrite the index field with that. To find the index pattern ID we could probably use GET /api/saved_objects/_find.

I like 2 because it's a longer term solution, but I did want to see if there are other opinions, and maybe if somebody from the Kibana team can confirm if this is the way to go.

/cc @sayden @ruflin @jsoriano - since you hit this as well in #10076
/cc @elastic/kibana-platform - I see you're working on some improvements to the APIs in elastic/kibana#27203, but from what I see it has more to do with automatically including and resolving dependencies and wouldn't help here. In any case, I'd be curious on your opinion of how we should handle this.

[epixa]

You can explicitly set ids when you create an index pattern. The option is hidden behind an "advanced" toggle.

[cwurm]

(Somebody on the Beats team please correct me if I'm wrong)

So I don't think that we create index patterns in the Beats, at least I don't see where just now. We create index mapping templates and then import dashboards. I guess now we might have to create index patterns as well?

All this is automated (user just runs auditbeat setup to set everything up), so if we have to do things like create index patterns with custom IDs, or retrieve IDs of existing patterns we'd have to be able to do it through APIs.

@ruflin or @andrewkroh - you probably know better what we do?

[cwurm]

So I've done a bit more research.

The premise of all this is that we want to make it simple for users to get started with any Beat. To that end, each Beat ships with an index pattern and one or (usually) more dashboards.

The way it currently works we get these two objects like this:

• The dashboards are exported from Kibana using GET /api/kibana/dashboards/export and stripped of their index pattern. Files like this one are checked in to the Beats repo.
• The index pattern is generated from a fields.yml file containing all fields of a Beat. A fields.yml looks like this and the resulting index pattern like this.

Users can then use the beat setup command to load the index template and dashboards using POST /api/kibana/dashboards/import. Just before loading, the code will replace the id and attributes.title fields in the index pattern, and the searchSourceJSON.index and visState.params.index_pattern fields in the dashboards with the value of setup.dashboards.index, if set.

By default, setup it will load everything, but it's possible to load only the template (--template) or only dashboards (--dashboards).

With searchSourceJSON.index now an ID specific to the Kibana instance, the exported dashboard can no longer be imported into another instance without changing that field.

Maybe this could be addressed by defaulting setup.dashboards.index to the default index for the Beat (e.g. auditbeat-*). What would then no longer work (but works now) is for the user to create their own index pattern in Kibana and importing our dashboards out-of-the-box. The user would have to find the index pattern ID and specify it as setup.dashboards.index.

I tried setting setup.dashboards.index="auditbeat-*" just now and it doesn't seem to quite work. But I'll investigate more.

[cwurm]

Mistake in my setup, setting setup.dashboards.index="auditbeat-*" or more generally defaulting it to {{beatName}}-* seems to work.

[cwurm]

What happened is that I set up the index pattern manually (causing it to have a random ID) and set up a dashboard on top of it. If I had used the generated index pattern all would have been fine.

Closing this.

[jsoriano]

Only this week I have seen two reports caused by this:

• Could not locate that index-pattern (id: winlogbeat-*), [click here to re-create it](#/management/kibana/index) #10923
• https://discuss.elastic.co/t/problem-with-dashboard-metricbeat-apache-overview-full/169921

Was this already changed in Kibana 6.6?

I wonder if we can do something to improve the experience here.