-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow the agent to not apply selected global tags on some inputs. #6347
Comments
The Another possible option is to use the
|
Thanks for looking at this @danielnelson. We've investigated the solutions you posted and found that they don't work or add some undesired overhead in the config. Also, I might not have been detailed enough on the use-cases. Here's more information on what we're trying to achieve. Baseline:Use case 1: Metric is received via HTTP without "host"
Use case 2: Metric is received via HTTP with "host"
Use case 3: Metric is collected via a cpu input without "host"
Big picture:Use case 4: The agent is running a production config
Proposed solution (as per PR #6348)In my understanding, the general purpose of the global tags is to reduce the repetition on the config and thus allow for more clarity. Allowing one input to skip some of those tags on demand seems a high value alternative to having to move the global tags into local tagging for each other input plugin. It creates less repetition and doesn't incur a processing/config overhead that the alternative solution would create. Config example under the proposed change is:
Identified workaroundA workaround that we have identified involves marking the ingress plugin on each point and then enacting the semi-global tags based on that marking.
The above workaround is functional but involves an additional processing step for the metrics and (in my opinion) adds a non-intuitive config. To me the proposed solution is siginificantly cleaner. |
This is a little quirky, but how about this workaround: [[inputs.influxdb_listener]]
[inputs.influxdb_listener.tags]
host = ""
atom = "" This takes advantage of the current precedence from high to low of: plugin dynamic tags, plugin config tags, global tags. Also depends on empty string tags being omitted from the output. I think this method may not be well enough defined to be depended on into the future, I don't entirely trust myself not to change it. (related #3362) So another option, though rather verbose, is to add the tags to every plugin other than [agent]
omit_hostname = true
[[inputs.cpu]]
[inputs.cpu.tags]
host = "$HOST"
atom = "howdy"
[[inputs.mem]]
[inputs.mem.tags]
host = "$HOST"
atom = "howdy" One more issue you might want to keep an eye on is #1778, I think a solution for that issue would also meet your requirements. I should say that I don't see that issue being implemented in the short term. Ultimately with the PR, we don't add new plugin level options unless we are very confident in the design, and I feel that the proposed pull request complicates things by adding tags as global and then removing them. I would rather have adding a tag be only an additive process. Also this is a rare enough requirement that I'm not inclined to add a new option specifically for it. |
Frankly, if we're looking at the "quirky" workaround you proposed, then there's also an option to create an explicit value to be dropped on the "modify" step.
This may (in fact) be the most concise option. Having said that, I still feel like these are "hacks" to get around the explicit setting the semi-global values per-plugin. I fully understand your point about this an unusual back-and-forth in the configuration, although I'm a bit surprised at this being a rare requirement. In our case this came up very naturally as we're trying to achieve a situation where a telegraf agent can be used as a transpartent proxy for the measurement. We must have the internal telemetry for the proxy, but we can't allow the proxy to change the measurements. Regardless, you have a valid point about wanting to keep the tags additive. I see an alternative that can get us there and that's creating "named arrays" of tags and allowing plugins to use these. That would be a compromise between having to set at each plugin, but also having a single point of change if necessary. What do you think? Example:
|
I believe the above idea will throw away the entire metric if it matches the
You setup is typical, but most are content to set the
Agreed these are hacks, but I'm beginning to think semi-global tags aren't something we really want. There are certainly new configuration constructs we could add, but I think something like this example is reasonable. I haven't tested it, but it is essentially your original workaround with 2 override processors to do the tag work, keeping the exclude out of the output: [[inputs.influxdb_listener]]
[inputs.influxdb_listener.tags]
ingress="influxdb_listener"
[[processors.override]]
order = 1
[processors.override.tagdrop]
ingress=["influxdb_listener"]
[processors.override.tags]
hostname="host.fqdn"
atom="distinct_atom"
[[processors.override]]
order = 2
tagexclude = ["ingress"] |
I think that we're managing without this feature and so it can be closed. |
Feature Request
When setting up multi-layer telegraf system, we often find ourselves in a situation where we expect global tags to be applied on all but one plugin. An example of such situation is a proxy-telegraf where we need to apply the "hostname" for all the internal and hardware plugins but we need it excluded from the http_listener input. If the measurement comes via HTTP without the "host" tag, we want to honor that. On top of that the capability to "tagexclude" doesn't work in this case as we expect some metrics to come in with the "host" present and we want to retain it.
Proposal:
Create a configuration on the running_input to exclude some global tags.
Current behavior:
All global tags are applied on every input.
Desired behavior:
It is possible to configure an input to not apply specific global tags.
Use case:
Telegraf agent accepts measurements submitted via http (used as influxdb proxy). It doesn't apply the hostname tag to the measurements that are proxied and don't have it, but it applies hostname tag to its internal telemetry.
The text was updated successfully, but these errors were encountered: