saved object tagging phase 1

[nreese]

Create a new saved object type for storing tags

{
  "type": "tag",
  "updated_at": "2017-11-01T11:29:45.060Z",
  "tag": {
     "name": <string containing tag name that is displayed in tag pill, limited to 50 characters (like github)>,
     "color": <hex string - tag pill color>
   }
}

New kibana tag API

We will not be storing tag counts in the tag saved object. The tag counts will be generated on the fly by querying all saved objects, building a snapshot of counts, and adding the generated counts with the tag saved object results . This will avoid all the complexities of managing tag counts during saved object writes.

• get
  • query string parameter to include counts
• delete - delete tag saved object, remove tag from all other other saved objects

Saved object API updates

• add tag to saved object (saved search, visualization, dashboard)
• delete tag from saved object
• Update read to translate tag ids to tag names?

Manage tags on visualizations, saved-searches, and dashboards

• Tag management is done in save UI
• Ability to create new tags in save UI.
• Link to tag CRUD GUI for deleting/editing tags.

Use tags to filter visualizations, saved-searches, and dashboards lists

• dashboard list view
• dashboard add panels view
• visualize list view
• saved search list view
• index pattern

Questions

• Where should tags live in saved objects? Should tags live at the top level or should they live inside each saved object type?
• Where will tag ids get converted to tag name? Where should tag joining be done? In the saved object API? On the client?
• Where should tag CRUD GUI live? Should this be placed under management? Is this just going to be a pop-out?
• Tag colors: Should any color be allowed? Should there just be a static list of colors?

[nreese]

@elastic/kibana-operations @elastic/kibana-design @elastic/kibana-management Do you have any opinions about the questions above or the tentative plans for object tagging?

[snide]

Tag colors: Should any color be allowed? Should there just be a static list of colors?

Likely a mix. I'm planning to use the viz color blind palette (which is 10 colors) for the tags. We've got the color picker component as well though, so I'll make sure the badge component can how accept a hex color as a prop as well.

[AlonaNadler]

Adding a few to @nreese list

What are we trying to solve?

• Since all the dashboards/visualizations are in the same lists, users can’t tell easily which dashboard is relevant for their role, which makes it harder to find the relevant dashboards
• Users sometimes delete and edit dashboards that are not theirs since it’s not immediately clear in Kibana that these objects are global

Tags dedicated page(pop-up) : this page will have all functions related to managing tags:
-Create a new tag:

• Type tag name
• Select Tag color
• (Optional) tag’s description

-Edit a tag name:

• Edit color
• Edit name
• Delete a tag

-List number of objects in the tag (#visualizations, #dashboards, etc..) - list of tags with a summary of the number of objects from each type
-Search for a tag - search for a tag by name including full or partial match.

Tag an object: When saving an object (for new and existing object) user can select to tag the object with one or multiple tags from a list of existing tags (tagging an object is optional).

-Remove a tag from an object can also be done by clicking save (except for index pattern, TBD)

Filtering by a tag:

• Users can filter the objects by specific tag by filtering existing tab or by searching for the tag name
• Filter multiple tags - using OR (instead of AND)
• Persist last selection of tags - user selected Sales tag and Marketing tags in dashboard, next time he logs in to dashboards the list will be filtered by Sales or Marketing

[trevan]

@AlonaNadler, that document is not publicly available. Can it be?

[AlonaNadler]

@nreese I added index pattern to your lists of objects that can tagged

[nreese]

cc @elastic/kibana-platform

[chrisronline]

The GUI should probably live in management in the same area that saved objects are managed.

I'm not sure about the storage. I'm hoping we can just store them within each saved object itself instead of duplicating a list at some higher level. In the management UI, we can just search all saved objects (this shouldn't be too costly based on what I've heard about saved object counts, but someone feel free to contradict me here) and that will be the list we display.

[tylersmalley]

My immediate concern is the addition of a new type as an inter-object dependency. A seemingly large percentage of our users use the export/import as a backup/restore solution. How will we handle this? We will also rely on client-side joins for this data, increasing the complexity of creating a new object with a tag.

I am wondering if we could denormalize this data on the objects themselves and completely eliminate the need of adding a new type.

Example:

{
  "type": "visualization",
  "updated_at": "2017-11-01T11:29:45.060Z",
  "tags": [{
     "name": <name>,
     "color": <hex>
   }],
  "visualization": {
    ..
  }
}

Getting a list of available tags should be easy. The only question I have is changing a tag name or color. I believe this should be possible with the update_by_query using a script, but we would need to verify.

[bmcconaghy]

So it seems like we could just have tags as the strings themselves and then use ES foo to build cached list of available tags for UI purposes. If we are going to be querying the docs anyway for tags, I don't see why we need the tag objects.

[epixa]

I'd recommend not overthinking the featureset of tags. Things like colors are cool, and I see the use case, but there's a lot of power in simply putting strings on a saved object that we can filter on, and we should explore all that has to offer before adopting a more complicated structure.

Changing the name of an existing tag will happen, but it's not something a person is doing daily. A bulk update here isn't crazy.

Also, we don't need to store metadata in json objects in Elasticsearch. ES is just a storage engine for us, so if we want to serialize metadata alongside the tag name, we can just do that in Kibana and store the result as a string. This falls down if we want to search against that data, but that's not what we're talking about here - "give me all the blue tags" is just... not very interesting.

Example: my tag name{"color":"#0000ff"}

This means we can go forward with tag names and build features around them such as security (grant access to all objects with tags matching my tag name) without preventing us from attaching meta data if and when we desire.

[kobelb]

Changing the name of an existing tag will happen, but it's not something a person is doing daily. A bulk update here isn't crazy.

What will happen here once we introduce object level security and the user won't potentially be able to update "all" saved objects that have this tag?

Not having a dedicated saved object type for tags, and the relationship, will make a tag management screen increasingly complicated. Additionally, providing autocomplete functionality when a user goes to tag an existing object would require us retrieving all saved objects, and parsing their tags collection.

[epixa]

What will happen here once we introduce object level security and the user won't potentially be able to update "all" saved objects that have this tag?

Without an elevated set of privileges, a user shouldn't be able to globally rename a tag if they don't have access to the places where it is used. If we want users to modify tags without some elevated privilege, then one user renaming a tag for their own objects shouldn't impact the tags used by objects they don't have access to. It should effectively introduce a new tag and all the saved objects the user doesn't have access to would remain undisturbed.

If we want administrative tag management that affect things globally regardless of OLS, then an elevated privilege is required. We could piggy back on the manage privilege, or we could create our own custom privilege. In either case, the Kibana APIs would need to check this privilege with the has_privilege API, and then use the internal server user for these global operations.

Having a global list of tags separate from the objects themselves means we have nested objects, which means we have the same sort of security challenges that we have with visualizations on dashboards today.

Not having a dedicated saved object type for tags, and the relationship, will make a tag management screen increasingly complicated.

I agree. There are trade offs here either way.

Additionally, providing autocomplete functionality when a user goes to tag an existing object would require us retrieving all saved objects, and parsing their tags collection.

We should be able to do this with term aggs rather than a search for raw documents. I'm pretty sure this is how we handle autocomplete elsewhere as well.

[snide]

Mocks from this morning's meeting. Note that all components in this mock should be available already in EUI. The filtering / search mechanics are in flight with elastic/eui#379

Todo

Make a mock for adding a visualization to a dashboard by utilizing tags.

[tylersmalley]

@epixa, thinking through this some more -

If we have the tags denormalized on the objects, it's entirely possible to have tags with differing meta-data. You could have a tag with a particular color then import a document containing the same tag with a different color. This is something we could handle in the object import as we could see what tags there are and ensure they are consistent. Just something we might need to handle if we go this route.

[epixa]

@tylersmalley That's a good point, and it isn't limited to import. If a person only had access to 50% of the objects that has a tag and then they went and changed the color of that tag, it would only update the 50% of objects that they have access to, which leaves a person that sees 100% of the objects in an awkward state.

[nreese]

@epixa @kobelb @tylersmalley
I have a POC tags API demonstrating a denormalized storage of tags. Can you take a look and see if this is a design that we want. Once I get the thumbs up, then I will move forward with productizing the tags API.

Known problems

• You could get into a state where a single tag label has different colors. This can happen by importing saved objects that have tags with the same labels but different colors. This problem is not unique to denormalization. This can also happen once object level security is implemented and updates only update tag objects that the user has permission to update.

[kobelb]

Without an elevated set of privileges, a user shouldn't be able to globally rename a tag if they don't have access to the places where it is used. If we want users to modify tags without some elevated privilege, then one user renaming a tag for their own objects shouldn't impact the tags used by objects they don't have access to. It should effectively introduce a new tag and all the saved objects the user doesn't have access to would remain undisturbed.

If we want administrative tag management that affect things globally regardless of OLS, then an elevated privilege is required. We could piggy back on the manage privilege, or we could create our own custom privilege. In either case, the Kibana APIs would need to check this privilege with the has_privilege API, and then use the internal server user for these global operations.

The corollary with Dashboard/Visualizations would stipulate that users can no longer edit Visualizations if they're added to a Dashboard that they don't have edit permissions, this doesn't seem like something that we'd want to do. I think we should allow Visualizations in this situation to be updated, and defer this decision to the Dashboard author to whether they want to add a Visualization that not only they have edit rights to. The same applies to tags, they could create a tag with the proper ACL to safe-guard this from occurring, but I think trying to automatically enforce this safe-guard is increasingly problematic.

[stacey-gammon]

If we go with the denormalized, string on object approach, how does a tag first get created?
According to the mocks, looks like non-admin users aren't supposed to create tags, but a tag can't first be created without an object existing with that tag. If we allowed anyone to create tags in the UI, this comes with it's own set of issues. And even if we don't let users create tags from the UI, they could still create tags (and also accidentally delete them if editing/removing the last unique tag on an object) by editing the data on their saved object directly.

I think the denormalized approach is going to create issues for us down the line. Remember how difficult it was to change dashboard ids from a user readable title string to a guid? I think it'd be easier to start with using a guid to attach and associate separate tag objects from the get go. Not only does this prevent the mismatched colors issue and the issue mentioned above, but I also have concerns about tags becoming associated by accident (there could be two different users/systems, e.g. beats, tagging objects with the same string but different meanings).

[epixa]

I had three primary concerns around tags being separate objects, but this and related discussions have shown that they all have mitigations:

Concern 1:
We know people ultimately want to secure tags, and nested objects make securing them harder. If they denormalized on the objects themselves, we already have a security solution that works for read-only users in the form of document level security.

While managing it becomes a bit more cumbersome, with the proper API/UI in Kibana, DLS is still a viable approach based on tag ids. Plus, if OLS is built directly in Kibana, then we can build a secure tag layer over that without issue. In the latter case, we wouldn't want DLS anyway.

Concern 2:
Nested objects are complicated, and every time we introduce one we have to deal with a bunch of new concerns like import/export. Historically, we have done a very bad job at managing these relationships in a sane and repeatable way.

We must fix this problem in phase 1 of any authorization solution we build into Kibana, so nested objects are here to stay and will only have improved support going forward. Maybe we can even tag a swing at the initial groundwork for this change in the context of this tag work, but even if we don't, the tag implementation will benefit from whatever we come up with in the future.

Concern 3:
Elasticsearch has no mechanisms for relational integrity, so tag ids can be orphaned.

We can ignore missing ids in our searches and even build a mechanism to automatically patch objects with missing tag ids when they were discovered or as a background task if we wanted to. In practice, we probably don't need to since tags won't be changing constantly, so the built-up cruft won't be very significant.

---

With those concerns addressed, I look to the concerns raised about the denormalized approach, and I can't come up with any reasonable mitigations.

Concern 1:
Tags will have to be treated as ephemeral if they can only exist if they are associated to at least one saved object, which means we can't really have a standalone CRUD interface for them.

The only real mitigation I can think of here is maintaining a totally separate list of tags as the source for autocompletion and whatnot, but then we basically have all the same problems that we'd have with the separate object approach, only things would be even more complicated since the relationship between the objects would be implicit based on name rather than a concrete id.

Concern 2:
Since tags are ephemeral, securing them is both challenging and inflexible since you can only really do so by name matching. For example, a person could rename one of their tags and accidentally expose that tag to a bunch more people because they didn't realize it matched some other security permission.

There isn't any mitigation here that I can think of.

Concern 3:
With secure tags in place, tag metadata like color will diverge over time between security boundaries. This is totally OK if there is never an overlap in security boundaries for only some users across certain tags, but it becomes a UX issue when such an overlap exists. Consider the scenario where there are two tags, production and marketing. John only has access to marketing, but Susan has access to both tags. John changes the color of the marketing tag, but that only updates the objects he can see, which are all the marketing objects that do not have a production tag. Susan, with access to all objects in both tags, now sees marketing tags in her list that have different colors. If she fixes that issue on her end, then John is now faced with the same problem on his end.

The only practical mitigation I can think of here is to not allow tags to have metadata.

---

All around, it seems like the tags-as-separate-objects approach is more flexible and easier to reason about. If we go down the denormalized route, then we have at least a couple big challenges without obvious solutions.

[AlonaNadler]

Summary from today's meeting:

• As a result of the discussion above, Tags will be a normalized object in Kibana
• Management team will work on the new import/export API which will allow import/export tags with all the dependencies via the UI
• Sharing team will focus on the core of tags feature without the API
• Tags first phase will not include the ability to export/import this capability will come immediately after

[stacey-gammon]

Tags first phase will not include the ability to export/import this capability will come immediately after

To clarify, I think we are going to add the ability to export/import in the GUI, but we'll do so using the same mechanism used today which I think will be very easy. I say that without looking at the code, but I'm guessing all you have to do is add a new type string somewhere and the tab and functionality for importing/exporting "tags" automatically appears in the saved objects management page. Does that sound right @bmcconaghy? Is this in line with your thinking too @nreese?

[nreese]

I created #16831 to track some of the research I had done for creating an export/import API

[alexfrancoeur]

Don't mind me, just subscribing

[timroes]

This effort currently is stalled until Spaces are implemented. At this point the topic could be picked up again or discussed. Since the topic itself belongs more to Management and Platform team who own the Saved Objects (and the management aspect of that), I moved it to their label. Also once Spaces would be implemented and this topic should be picked up again, we anyway need to further discuss who can then work at this.

[rayafratkina]

When we get back to this work, we need to reconcile with Canvas static tags created in #28779

[streamich]

cc @elastic-jb @elastic/kibana-app-arch

[streamich]

I've been working on a PoC during my Space Time to add tagging support to Kibana. Below I would like to present the findings and structure I ended up with, after considering all comments in this thread, all other Kibana GitHub issues about tagging and a number of other requirements that you can see below in the "Product Specification" section.

Tags are Basic+ feature which means it has to be a plugin in X-Pack. In the PoC I implemented it as tags plugin in X-Pack.

Demo

Empty management app Tags section:

Tag creation form:

Creation of key-value tag:

Progression from empty Tags section in management app to populated one:

Example plugin with examples for tag integration for developers:

Integration of <TagListEditable> component on a dashboard:

Dashboard listing view with attached tags:

Filtering by Environment:Production tag in dashboard listing table:

Visualization listing view with attached tags:

Tags app which displays all resources (any resource type) to which specific tags were attached:

Product Specification

Below are product specifications which I received from @alexfrancoeur while working on this PoC as well as extra requirements from myself.

• Objects should be able to support one or multiple tags ✅
• Tags are mutable - strings and colors can change, but the id of that tag is still persisted with the associated objects ✅
• Tags should make it easy to group and sort on objects, whether that's in dashboards, Elastic packages with groupings of ML job configs, alerts, etc. or saved object management for quick export or copy to space ✅
• Tags should be consistent across solutions and applications within Kibana with a reusable and backwards compatible service that developers can trust ✅
• Central area for management, metadata management, full CRUD capabilities, support for RBAC ✅
• Key-value tags ✅
• Attach tag to anything ✅
• REST HTTP API ✅
• GitHub-like “label” model ✅
• Find all resources (of any type) by tag ✅
• Default tags
• Color blind tags
• Path tags
• Emojis?
• Referential integrity

Key-value tags

Key-value tags are ones where user for a given key creates multiple tags with different value. This allows grouping tags by value.

Attach tag to anything

Not everything in Kibana is a Saved Object. This PoC allows attaching tags to anything.

• Some things are stored not as a saved object.
• Some things are stored in-line on a saved object: Time-to-Visualize visualizations, drilldowns.

REST HTTP API

All tag persistance logic is implemented in one place on the server-side and exposed through REST HTTP API to outside for consumption by Kibana app and other users.

• Using Saved Objects client on frontend is bad practice.
• Expose semantic REST API for Kibana and third parties to use.

GitHub “label” model

The implementation follows GitHub's label model, where you first select one or more tags using full-text search. Once tags are selected the resources are filtered using exact match by tag ID.

Default tags

There could be a set of Default tags, which are not actually used for tagging but used for populating a new space with tags. Request for security team to implement listener for new space creation. We could listen for a new space creation and populate that space with tags. This is similar to GitHub's default tags for organization.

Color-blind support

Tags could have a pattern for color blind people to better discriminate between color shades. This was not implemented in this PoC, but @elastic/eui-design team could add support for that in EUI.

Technical Architecture

Tagging is implemented in multiple layers, where each layer is completely isolated and consumes functionality only from the one layer below.

Saved object client

• Managed by platform team
• Provides ability to persist/manage “objects” used in Kibana

Tags saved objects

• Adds two new saved objects tag and tag_attachment.
  • tag_attachment is used for attaching tags to anything, not just saved objects, and to filter for all Kibana resources across any type.
• RBAC/spaces/security/migrations should be possible to support as tags themselves are saved objects and saved object client provides that functionality.
• Referential integrity is not supported nor used in Kibana. I've started a GitHub issue to discuss referential integrity in Kibana. Although saved object client does not support referential integrity, tags client could make best effort to maintain tag data integrity; and implemented referential integrity once that is provided by saved object service.

Tags client

• Implements CRUD+Search ITagsClient interface for tag and tag_attachments saved objects.
• Can attach tags to anything, not just other saved objects. Uses Kibana global IDs for that.
• Exposes subscription attachments$ for other plugins to react to changes of tag attachments to some resource.

REST HTTP API

• Exposes ITagsClient interface through HTTP.
• Saved objects are used only on the server. It is internal implementation detail that tags are stored in saved objects. More generally, I think we could start a discussion about deprecation saved objects client on the frontend, as there are many benefits of having it only on the server (@rudolf).
• API is almost identical to GitHub's label API.

The HTTP endpoints tags plugin exposes is its Public API:

• Third parties can use the HTTP API to securely manage tags in Kibana.
• Similar to alerting, they implement all saved object logic only in one place—on server—and expose HTTP API.

Browser tags client

• On the browser the same ITagsClient interface is implemented, but it works through HTTP API, not frontend saved object client.
• The client offers stateless CRUD+ functionality, simply redirects requests to the REST API.

Tags manager

Tags manager is implemented on top of the tags client, and has the following properties:

• Reactive state management for tags UI. Everything is exposed as observable. And results can come synchronously from the case or async from the server. Or, both, first synchronously from the cache then the latest correct value async from the server.
• Stateful — it acts as a cache, it remembers all tags and tag attachments it has ever seen for rendering UI synchronously.
• Optimistic — it can execute operations such as "create" or "delete" optimistically; meaning that the operation is performed synchronously on the client and UI immediately updates and rolls back that operation in case it failed on the sever.

React components

The tags plugin provides a range of React components on top of which the UI functionality of the tags plugin and integrations can be built.

UI

A collection of apps use the React components and Tags Manager provided by thet tags plugin.

• Management app
• Tags app
• Examples plugin
• Integrations
  • Dashboard, Visualize, Index patterns, maps, etc..

Integrations

Once we have tags we now need to integrate them into various places in Kibana.

It is relatively easy to integrate tags into other X-Pack plugins as those can simply depend on tags plugin. For example, alerting plugin could "require" the tags plugin:

It gets harder when tags need to be integrated into an OSS plugin. For example, dashboard plugin could provided extension points and dashboard_enhanced plugin from X-Pack could implement those extension points using components it gets from tags plugin. This requires us to have "shadow" dashboard_enhanced plugin in X-Pack for dashboard plugin, whose sole purpose is to implement dashboard tagging extension points with functionality from tags plugin:

This quickly adds up as for every OSS tags integration we would need to follow a similar pattern. For example, for visualize plugin we would need something similar:

Another approach is to have an OSS enhancements (or tags_oss) plugin which is "empty", it would be just a registry of X-Pack services. When tags plugin loads it would populate the enhancements plugin, and dashboard plugin consumes the enhancements plugin as any other OSS plugin:

cc @nreese @stacey-gammon @peterschretlen

[cjcenizal]

@streamich Awesome!!! FWIW, your last option for integrating the tags plugin in OSS is similar to how I envisioned implementing per-user settings in both X-Pack and OSS per #17888 (comment). Essentially in this case, the OSS plugin exposes a contract backed by a no-op implementation. The X-Pack plugin can overwrite the no-op implementation with its own implementation via registration methods. Consumers can check isRegistered or isOperational methods on the OSS plugin to enable or disable tag-based features.

[alexfrancoeur]

This issue has been open for quite some time and as a result, a lot of the discussion is outdated. We've decided to close out this issue and #16495 in favor of a new meta issue focused on the initial implementation of saved object tagging.

PS. Thank you so much for sharing the results of your space time @streamich! Please feel free to participate in the new issue as much as you'd like.