Request for input: tags/labels for organizing definitions

[sryza]

We’re considering introducing additional constructs for organizing assets and jobs.

The status quo

• Assets can be organized into asset groups, which live inside code locations. Each asset lives in exactly one group and exactly one code location.
• Jobs can be organized into code locations. Each job lives in exactly one code location.
  • Each job also has a tags attribute, but this should probably actually be called run_tags, because the tags are applied to runs of the job, not the job definition itself. They’re meant to be used for purposes like controlling what queues the runs end up in, not for organizing the job definitions.

What we’re investigating

We’re considering introducing an attribute named either “tags” or “labels”, for organizing assets and jobs. Unlike asset groups and code locations, an asset or job would be able to have multiple of these tags or labels.

We’re trying to make decisions like:

• Should the tags/labels on an asset/job be a set of strings or a dictionary key/value pairs?
• Should they be called “tags” (which perhaps implies they work similarly to run tags) or “labels” (which leaves more room for them to be purely organizational, but also means more concepts floating around)?

Questions for you

If we offered tags/labels for organizing assets/jobs, how would you use them for? I.e. what are some specific examples of tags/labels that you would apply to your assets or jobs?

What are the situations where it would useful to search, filter, or group by these tags?

[ion-elgreco]

This would be very useful because we deploy every feature branch as a new code location, so in DEV dagster we can have multiple feature branches at once, it would be nice to seperate all runs per code deployment this way

[rokity]

@sryza Maybe take a look at LakeFormation tag access control. To take insipiration https://docs.aws.amazon.com/lake-formation/latest/dg/tag-based-access-control.html .

[dduong1603]

We are looking for ways to organize our assets into different subgroups that may share some common assets, so labels definitely make sense, especially when you can filter/search for them (either single label or multiple labels at a time).

Maybe have a setting on the UI for default labels (and customize that further for different logins/roles) instead of showing everything on load right now, especially on a large asset graph?

[sryza]

Do you have examples of what these subgroups would be that you'd be comfortable sharing? Just trying to make this more concrete in my mind.

[dduong1603]

I can't share the specifics...but we have multiple products - each with their own assets maintained by their own teams - but they also source from some common upstream assets (like users and payments) that are maintained by a central team. Right now we put the assets into groups, but if they want to check the status of all the relevant assets (from their teams and some from the central team) there is no easy way to filter/search for them and look at them in a single viewport since each asset can only belong to a single group.

[jburnich]

In my use case, a tagging system would save me time in selecting my assets in a grouped way when I want to build backfills (with the current version, I select assets 1 by 1).

What type of logical operations will this tagging system be based on? OR ? AND ? NOT ?
In some cases it could be interesting to combine these operations to make complex selections.
With a key/value system, let's imagine I want to select all assets of type contacts except those of clients A and B
(data_type = "contacts") AND NOT ((client = "A") OR (client = "B"))

I prefer the term "labels", which would make it easier not to mix up the different concepts.

[moleary-gsa]

A simple string label / list of string labels would be very helpful for asset cataloguing / visualization for us.

Our asset groups get very large and contain many similar pipelines so it's difficult at a glance to see them, if labels could be applied in the context of the asset DAG it would help with that.

I know that's not a straight forward addition though because it would need the addition of an additional UI element to group labels and that would maybe need to be separate to grouping if they aren't hierarchical (group -> label)

[sryza]

Would you prefer groups to be hierarchical?

[moleary-gsa]

That could work. It be nice in the DAG view too.

I wonder if that would be a hard thing to visualize if the nesting got too deep though?

The idea of having arbitrary key / value which could be used to filter the DAG would also be nice 😊

But if I think of AWS services for example. The hierarchical grouping of resources is more often than not good enough to navigate and the tags often feel redundant.

If I had to have one I'd have the grouping

[danielgafni]

Personally, I would love groups being hierarchical

[wintersrd]

This would be hugely valuable for our use cases as well, to be able to add arbitrary labels that could be filtered/queried against (and ideally displayed on each asset/op in the same manner as a compute_kind label). Being able to add a list of tags as a list is sufficient. The example I ran into today was a simple pipeline of source asset -> asset (snowflake) -> asset (MySQL). Just some basic metadata examples that are part of the asset definition for us:

• what is the source system type?
• Who is the business owner / data provider of the source?
• Are the sources or final assets internal or external of our environment?
• Is the final asset production facing or analytical?
• What are the SLA priorities of the assets?

As the nature of meaningful metadata varies by organization, having a strings (and a way to filter, search, etc) provides sufficient flexibility for each user's needs

[sryza]

It sounds like, in your case, key-values might be useful? I.e. imagining a set of tags/labels on a particular asset like {source_system=source", data_provider=marketing, external=true, sla_priority=high}?

[wintersrd]

It can be, so long as the keys are completely flexible. Otherwise a list-based approach is usable

[danielgafni]

We have lots of very similar assets produced by asset factories which usually have the same name (last part of the key). Due to our asset graph structure they are currently in one group, and it's really hard to put them in different groups as it would harm graph readability.

So we end up with lots of identically looking assets in a group. Their description even starts the same, so zooming in won't help to separate them at glance.

Having different sets of labels displayed on them would really help.

[tcjennings]

It would be useful to tag/label assets and jobs that use a resource, then use that tag/label in an AssetSelection so one sensor monitoring some resource, like a database table, could issue run requests for those assets. As far as I know, AssetSelection currently only works on assets by group, name, or relationship.

[danielgafni]

And limit Run/op concurrency based on these tags!

[ion-elgreco]

@sryza could this potentially also allow op_tags to be adjusted afterwards on an asset definition? We want to define the resources of each operation after they are in the State of an AssetDefinition. The reason is during our generation of the defs we define the range of the data that we are going to do runs on, this is also closely linked to how many compute resources we need.

Currently the only way to adjust them afterwards is overriding the asset.node_def._tags and asset.op._tags but this is not a public API

[IgnorantWalking]

Hi.

Job/asset labelling would also be a super helpful feature for us within the scope of a code-location, but it would be awesome if we could extend the concept to a whole deployment scope. In this issue on the dagster UI board: we summarize part of our use case and proposal:

• hierarchical assets group to better organize/display complex asset definitions within a code-location.
• complementary, labelling at assets/jobs level to allow defining what we called in the issue "definitions group"; a deployment level grouping defined as the result of a filter query applied over all the deployment code location contents: filtering by a tag, a label, by job or asset name, etc; or defined by a split expression: split the deployment contents by the values of a specific tag, for instance. Similar to the config we apply to define the queued run coordinator queues, but focused on the GUI representation. Of course, if the labelling feature is included, it would be perfect to support this, and the "definition group" could be restricted only to using these labels.

For us, it makes sense to distinguish between the asset definition itself (the code location and the group/subgroup it belongs to) and the "virtual definitions group" that can be declared using labelling to create a collection of assets/jobs not based on the infrastructure required to materialize them (the code locations) but some higher-level business concept.

Real scenarios for us that will justify the feature:

• split the deployment contents by use case, business vertical or affected stakeholders
• split the deployment by customer in a multitenancy solution: exact jobs/assets definition but multiple instances created, as many as different tenants exist; it would be awesome to be able to group all of them (defined on different code-locations) under the same virtual groups.

[carterfigma]

Is using dbt tags to filter assets via the UI on the roadmap?

[cnolanminich]

Yes -- it was actually released this afternoon as part of 1.7 🎉: https://github.com/dagster-io/dagster/releases/tag/1.7.0

[sryza]

Closing this discussion as we now have asset tags: https://docs.dagster.io/concepts/metadata-tags/tags#tags