Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Proposal to rename Kibana "Index Patterns" #44955
Today in Kibana there are two very different things which are both called "index patterns".
The string based index pattern and object based index pattern are two very different things, and have very different use cases. Yet they are related and it is easy to mix the two up in conversations and in documentation.
The string based index pattern is very aptly named: it is a pattern that matches indices. Object based index patterns OTOH are not. They are not patterns, rather descriptions of fields in an index.
I therefore propose that the object based index patterns are renamed. Non-exhaustive list of suggestions for names (feel free to come up with others - this is just a starting point):
Isn't this just a waste of time?
I must admit: I don't know exactly how much work this change requires. I can imagine there is a tonne of places in the product that needs to be changed. On top of that there is documentation. And then we need to re-educate the community. I can completely understand if this feels like too much work for too little value. I've been thinking that for two years - until now :)
I agree with @sqren that I'm not really sure how difficult this work would be to rename, but overall I'm a huge +1 for clearing up this naming confusion. It makes talking about ES queries from a Kibana context so difficult and comes up in almost every project I've worked on here.
tl;dr from this one person:
I agree with the reasons why the current name is confusing. Also, that names, esp. for such central things, are important to get right. names for central structures are preferably held constant - it helps when people run into docs, forum and SO discussions and Elastic versions over many years. By extension I also agree that now is the best time to discuss misnomers since we haven't yet invented time travel
Re the suggested names: it doesn't feel like a mapping. What is being mapped, from what to what? Also even if it can be seen as a mapping, it might not be the most specific term. We tend to use Mapping for various things already in the entire stack so it has a somewhat diluted meaning.
Maybe someone could define, as briefly yet fully accurately, what these things are, and what goals they serve, or point to an existing one?
The term "object based index pattern" is something Google couldn't find. But our doc did mention about rollup index patterns. Also, the string-based index pattern is not quite equal to
So I'd be in favor of not looking at the name in isolation, as name should be apt; which requires a clear concept and definition; which, if my reading of eg. the above #35481 and my confusion is any hint, isn't yet worked out.
Sticking to just the string pattern based index patterns, there seem to be other potential issues:
You are completely right, and it was probably not a good idea of me to suggest names this early in the process. The important thing is that we can start the conversation now about what index patterns are, and then afterwards we can discuss alternative names.
The reason I suggested "mapping" is because the object based index pattern is similar to elasticsearch's concept of Mapping which is a description of the fields in an index.
I'm not sure what you mean here. Isn't that exactly the problem that the term "index pattern" has more than one meaning? I'm proposing that we stop using the term "index pattern" for anything but the string based form.
++ this definitely adds to the confusion. When I realized that it was mostly built as an extension of the field capabilities API it became a little less blurry but there are still a lot of things I do not understand.
I was hoping for this reaction: "We can't rename it because we don't know what it is". If we make it a goal to find a more appropriate name, and this forces us to define (and perhaps re-define) what Kibana index patterns inherently are, that might be a very good side-effect.
You list some issues with string based index patterns:
I don't see how that is a problem
Isn't this conflating string based index patterns with object based index patterns again? If we agree that (string based) index patterns are just strings, then we can come up with other abstractions that do what you suggest. For instance rich structures that have stable identifiers orthogonal to the underlying indices that it references via string based index patterns. The important thing is that we don't call this rich structure an "index pattern" like today.
You might be right. I am trying to decrease the scope by not solving all the pain points related to index patterns but just defining what it is an thereafter renaming it to avoid the ambiguity that currently exists.
Yes I thought so, and appreciate the appeal to analogy or familiarity, though
I agree with what you say here, my note you cite was in response to "The string based index pattern is very aptly named" - it's aptly named only in a superficial sense, precisely because of what you say here, and what I mentioned later in my comment (it's also a hub for a LOT of things eg. scripted fields, besides just being a pattern that matches indices)
Haha I looked at index patterns in terms of what they currently are (rather than assuming that first it was just proper matching patterns only and then more functions became coupled to it) and thought of it as a non-separable part of the confusion, rather than an add-on aspect :-) So to me, what latches on is an integral part of the problem, even if the solution might be the very uncoupling (or cohesion/integration) of these things.
Indeed, looks like others' and my responses are in effect, what you expected :-)
I tried to address it - it couples the name (and id/reference) with the content, which is a constraint that has no benefit that I know of, but excludes valid use cases, as a "schema" evolves over time:
"[wishlist] Giving index patterns a name: It's possible to use multiple patterns like filebeat-,metricbeat-,apm-* for one index pattern. It would be nice if then a name like o11y could be given to it instead of showing up like the above in the Discovery UI"
I don't think so (although as I mentioned I'm not 100% clear on what you mean by the object based index patters so apologies if I misspeak), you can create a string based index pattern and then go ahead and associate them with field formatters, scripted fields etc. Please correct me if a string based index pattern is just exclusively a subset of the set of indices in ES. Eg. this writes about field formatting, which decidedly does not fit into the concept of "subset of indices" that a pattern like
As I tried to imperfectly convey, I don't identify with this definition for the current situation, as they are a lot more. Again, it's from the viewpoint of looking at things as they currently are. The specification of the index subset (eg.
But it's possible, and feels totally appealing, that the rethinking of the concepts leads to a much less tightly coupled set of relations, where indeed, the subsetting of the indices is orthogonal to, or at least properly separated from, the concepts of field selection, field formatting, scripted fields etc.
In this case, I think the meaning still shouldn't be conflated with the means. The meaning (value) is a subset of ES indices. The means of subsetting the ES indices currently is that the user can specify a matcher string. But let's keep these two things separate: it may well be the case in the future that on the Kibana side, or even in ES, we introduce other ways for specifying sets of indices.
Totally! I feel like we're pretty much on the same page in all aspects that matter here, and any clarification from me may be due to the fact that you're probably ahead in thinking about how to disassociate (extract out) currently attached things from "index patterns" while I try to avoid implying a solution as your issue (including its title) is about naming the thing, while the linked issue is about what it should stand for.
I think this is long overdue @sqren. Thank you for raising this!
As a user, I define a Kibana's index pattern based on its properties and behavior:
When I look at this list of properties and behaviors, the one essential characteristic that jumps out at me is the last one. These apps depend on the Kibana index pattern to provide the data that they consume and visualize. And "search" is the mechanism for retrieving data. Because this thing is the source of this search capability, I tentatively offer "search source" as a name.
"Search source" seems to me that it would be specific enough to be distinct from other terms and support SEO. It also seems like it would be flexible enough to allow us to add new capabilities to it over time without requiring a name change. We already support a special type of rollup index pattern for searching rollup data; this would become a "rollup search source". Maybe we'll add other types of data in the future, and the term "search source" would accommodate that.
One minor concern is that "search source" is already a term used in the Courier code, but that shouldn't be difficult to rename and I think we're moving away from using Courier in general anyway.
Yeah, we went over quite a few things. A zoom call sounds like a good way to move forward. I'll set something up.
I can definitely see "search source" work as a name. It is a little generic/vague which might both be an advantage (it's easy to evolve the concept over time without another rename) however it might still be difficult to grasp for new-comers. Either way it's much better than status quo since it removes the ambiguity around "index pattern". So I'm tentatively +1 :)
As this is going in the direction of "rethinking" Index Patterns I would like to throw this ES issue in here: elastic/elasticsearch#33267 It might be that some of the data above actually belongs in Elasticsearch directly which could simplify things and we need less names.
I like the name "search source" as long as we want to bake in the word "search" (it feels like a response to eg. a SQL query can be seen as search, but maybe it's a slight broadening of the concept, still perfectly OK though). As a non-native speaker it's a bit hard for me to pronounce it, any opinion on "data source"? Vega and other tools use it.
Related: it's not just pointing to source data; it's also an augmentation of source data with metadata that ES doesn't manage as part of indices or related structures. In a more traditional, eg. relational database, many of the schema info (metadata) would be an integral part of the database itself.
An interesting question is whether everything that is needed is ideally resided within Kibana, or would better belong into ES. So it's worth checking if the current boundaries between ES and Kibana are considered optimal.
I assume it could work such that calculated fields ("scripted" fields), field formatters etc. could be shared among data / search sources.