Tags for sources and columns (#1906, #1586)

[beckjake]

Fixes #1906
Fixes #1586 (mostly)

Add support for tags on sources and columns.

Source tags can be applied at both the 'source' and 'table' level, and a source's tags are the combination of the two.

Column tags are applied at the column level and are only applied to tests of that column (and they apply to all tests of that column!). That means if you have no tests, column tags do nothing.

There's no way to tag a subset of a column's tests, and no way to tag tests defined at the table level beyond the usual behavior of selecting all models downstream from a model/source

[drewbanin]

@beckjake this overall looks very good to me!

It would be great to bubble column-level tags up to their parent models. I'd love to be able to run

dbt run -m tag:pii+

to re-run all models that contain columns containing PII data, for instance. Would something like this require a big re-architecting of how we parse nodes?

I do see that the columns dict in model nodes contains a tags attribute. Can we either:

• collect column-level tags into a node-level column_tags list, then check that in node selection?
• loop through column tags when selecting nodes to run, or
• something else?

My preference would be the first one (adding a column_tags list to model, seed, and snapshot nodes) as it would make rendering column-level tags in the docs a little bit easier IMO.

What do you think?

[beckjake]

collect column-level tags into a node-level column_tags list, then check that in node selection?

We can do that, but then you can't run tests on a given column/set of columns alone, and instead can only run tests on all models that have any columns with that tag. Is that ok?

Edit: I guess we could special-case selection so tests worked even more differently from run and then we could have it both ways. That seems unintuitive to me though.

[drewbanin]

We can do that, but then you can't run tests on a given column/set of columns alone, and instead can only run tests on all models that have any columns with that tag. Is that ok?

This is a really great point - I hadn't considered this! I do think special casing selection would work here, but I'm not super inclined to make that code any more complicated than it already is.

My instinct is just that dbt test -m tag:abc and dbt test -m tag:abc should both select resources with the same grain (ie. at the model/node level). While I do think there's certainly merit to allowing tests on specific columns to be selected (either by column name, or by column tag, or similar), I think that should probably be a different flag or selector. We can implement something like that in the future. Let me know how you feel about that?

[drewbanin]

Ok, following up on our IRL chat:

• ignore my comment above
• preserve the behavior defined here
• also support tags on tests (with the same syntax as severity)

[drewbanin]

One comment about coercing user-supplied values, but this LVGTM otherwise!

[drewbanin]

LGTM when the tests pass!