Allow the agent to not apply selected global tags on some inputs.

[pberlowski]

Feature Request

When setting up multi-layer telegraf system, we often find ourselves in a situation where we expect global tags to be applied on all but one plugin. An example of such situation is a proxy-telegraf where we need to apply the "hostname" for all the internal and hardware plugins but we need it excluded from the http_listener input. If the measurement comes via HTTP without the "host" tag, we want to honor that. On top of that the capability to "tagexclude" doesn't work in this case as we expect some metrics to come in with the "host" present and we want to retain it.

Proposal:

Create a configuration on the running_input to exclude some global tags.

Current behavior:

All global tags are applied on every input.

Desired behavior:

It is possible to configure an input to not apply specific global tags.

Use case:

Telegraf agent accepts measurements submitted via http (used as influxdb proxy). It doesn't apply the hostname tag to the measurements that are proxied and don't have it, but it applies hostname tag to its internal telemetry.

[danielnelson]

The host tag set by hostname = "" in the agent section of the configuration should work this way. It is only set if there is no existing host tag.

Another possible option is to use the override processor instead of the global_tags section to set the tag. This processor runs after the inputs and could use an attribute of the metrics to decide if it should set the tag. The processor only runs on the metrics that are allowed in via namepass/namedrop and tagpass/tagdrop. For example this sets the tag hostname=howdy on all metrics with the measurement cpu:

[[processors.override]]
  namepass = ["cpu"]
  [processors.override.tags]
    hostname = "howdy"

[pberlowski]

Thanks for looking at this @danielnelson. We've investigated the solutions you posted and found that they don't work or add some undesired overhead in the config.

Also, I might not have been detailed enough on the use-cases. Here's more information on what we're trying to achieve.

Baseline:

Use case 1: Metric is received via HTTP without "host"

1. The agent receives a foo metric via HTTP.
2. The agent does not apply host to the metric even though it's missing.
3. The agent emits the metric without host tag.

Use case 2: Metric is received via HTTP with "host"

1. The agent receives a bar metric via HTTP.
2. The agent does not change the value of the host tag.
3. The agent emits the metric with the original host tag.

Use case 3: Metric is collected via a cpu input without "host"

1. The agent collects a baz metric via a cpu input.
2. The agent applies a host tag using global "hostname" value.
3. The agent emits the metric with the new host tag.

Big picture:

Use case 4: The agent is running a production config

1. The agent is configured with hostname="our.hostname.fqdn" and atom="distinct_atom" global tags
2. The agent is configured with http_listener input that is expected to "not interact" with the host tag and the custom global atom tag. *(not interact => Use Case 1, Use Case 2)
3. The agent is configured with cpu, mem, disk, diskio, net, nstat, processes and internal plugins that are expected to normally apply the global tags.

Proposed solution (as per PR #6348)

In my understanding, the general purpose of the global tags is to reduce the repetition on the config and thus allow for more clarity. Allowing one input to skip some of those tags on demand seems a high value alternative to having to move the global tags into local tagging for each other input plugin. It creates less repetition and doesn't incur a processing/config overhead that the alternative solution would create.

Config example under the proposed change is:

[global_tags]
  atom="distinct_atom"

[agent]
  hostname="host.fqdn"
  omit_hostname=false

[[inputs.influxdb_listener]]
  service_address = ":8186"
  read_timeout = "10s"
  write_timeout = "10s"
  ignore_default_tags=["atom", "host"]

[[outputs.influxdb]]

Identified workaround

A workaround that we have identified involves marking the ingress plugin on each point and then enacting the semi-global tags based on that marking.

[[inputs.influxdb_listener]]
  service_address = ":8186"
  read_timeout = "10s"
  write_timeout = "10s"
  # Mark points collected via influxdb proxy
  [inputs.influxdb_listener.tags]
    ingress="influxdb_listener"
  
[[processors.override]]
  #Tag points collected through non-influxdb inputs
  [[processors.override.tagdrop]]
    ingress=["influxdb_listener"]
  [processors.override.tags]
    hostname="host.fqdn"
    atom="distinct_atom"

[[outputs.influxdb]]
  tagexclude=["ingress"]

The above workaround is functional but involves an additional processing step for the metrics and (in my opinion) adds a non-intuitive config. To me the proposed solution is siginificantly cleaner.

[danielnelson]

This is a little quirky, but how about this workaround:

[[inputs.influxdb_listener]]
  [inputs.influxdb_listener.tags]
    host = ""
    atom = ""

This takes advantage of the current precedence from high to low of: plugin dynamic tags, plugin config tags, global tags. Also depends on empty string tags being omitted from the output.

I think this method may not be well enough defined to be depended on into the future, I don't entirely trust myself not to change it. (related #3362)

So another option, though rather verbose, is to add the tags to every plugin other than influxdb_listener, and use the $HOST environment variable:

[agent]
  omit_hostname = true
[[inputs.cpu]]
  [inputs.cpu.tags]
    host = "$HOST"
    atom = "howdy"
[[inputs.mem]]
  [inputs.mem.tags]
    host = "$HOST"
    atom = "howdy"

One more issue you might want to keep an eye on is #1778, I think a solution for that issue would also meet your requirements. I should say that I don't see that issue being implemented in the short term.

Ultimately with the PR, we don't add new plugin level options unless we are very confident in the design, and I feel that the proposed pull request complicates things by adding tags as global and then removing them. I would rather have adding a tag be only an additive process. Also this is a rare enough requirement that I'm not inclined to add a new option specifically for it.

[pberlowski]

Frankly, if we're looking at the "quirky" workaround you proposed, then there's also an option to create an explicit value to be dropped on the "modify" step.

[[inputs.influxdb_listener]]
  [inputs.influxdb_listener.tags]
    host = "!ignored"
    atom = "!ignored"
  [inputs.influxdb_listener.tagdrop]
    host = "!ignored"
    atom = "!ignored"   

This may (in fact) be the most concise option.

Having said that, I still feel like these are "hacks" to get around the explicit setting the semi-global values per-plugin.

I fully understand your point about this an unusual back-and-forth in the configuration, although I'm a bit surprised at this being a rare requirement. In our case this came up very naturally as we're trying to achieve a situation where a telegraf agent can be used as a transpartent proxy for the measurement. We must have the internal telemetry for the proxy, but we can't allow the proxy to change the measurements.

Regardless, you have a valid point about wanting to keep the tags additive. I see an alternative that can get us there and that's creating "named arrays" of tags and allowing plugins to use these. That would be a compromise between having to set at each plugin, but also having a single point of change if necessary. What do you think?

Example:

[global_tags]
  foo = "bar"

[tag_groups]
  [[tag_groups.internal]]
    host="$HOST"
    atom="distinct_atom"

[[inputs.internal]]
  tag_groups=["internal"]

[danielnelson]

I believe the above idea will throw away the entire metric if it matches the tagdrop, and tagexclude doesn't allow you to match based on value so it can't be substituted.

I fully understand your point about this an unusual back-and-forth in the configuration, although I'm a bit surprised at this being a rare requirement.

You setup is typical, but most are content to set the host tag to the proxy Telegraf if it is unset in the incoming data. If you think of the host tag as being the telegraf_host tag, which is what it most accurately describes, then this works.

Having said that, I still feel like these are "hacks" to get around the explicit setting the semi-global values per-plugin.

Agreed these are hacks, but I'm beginning to think semi-global tags aren't something we really want. There are certainly new configuration constructs we could add, but I think something like this example is reasonable. I haven't tested it, but it is essentially your original workaround with 2 override processors to do the tag work, keeping the exclude out of the output:

[[inputs.influxdb_listener]]
  [inputs.influxdb_listener.tags]
    ingress="influxdb_listener"

[[processors.override]]
  order = 1
  [processors.override.tagdrop]
    ingress=["influxdb_listener"]
  [processors.override.tags]
    hostname="host.fqdn"
    atom="distinct_atom"

[[processors.override]]
  order = 2
  tagexclude = ["ingress"]

[pberlowski]

I think that we're managing without this feature and so it can be closed.