Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add "value" output data_format #3363

Closed
wants to merge 1 commit into from
Closed

Add "value" output data_format #3363

wants to merge 1 commit into from

Conversation

Paxa
Copy link

@Paxa Paxa commented Oct 19, 2017

This PR add feature to output data "as is"

For example:

[[outputs.file]]
  files = ["stdout", "/tmp/metrics.out"]
  data_format = "value"

Will just print as received (value field)

User cases:

  • Kafka mirror
  • Process logs and saving local copy (for debugging or archiving)
  • Forwarding data without modifications

It's just a prof of concept (i'm new to go lang), if you guys like this idea I will add test and documentation.

Btw will it be useful to send any fields other then value? (hard coded now)

@danielnelson
Copy link
Contributor

I'm not sure about this serializer because the value format is so basic, and so the output would be very lossy.

The value parser is useful for interop with services that cannot be changed, but I'm not sure it is good for debugging or archiving when there are better choices. Wouldn't you rather have output in a format at least with a timestamp?

@Paxa
Copy link
Author

Paxa commented Oct 19, 2017

I'm experimenting with data replication and data archiving. For example we have kafka with data points as json (event sourcing) and I wanna use telegraf to send this data somewhere else e.g. aws kinesis or other kafka server. So we can feed it to our application again later.

Standard serialiser will convert messages to something like:

kafka_consumer,host=MacBook-Pro.local value="{\"key\": \"value\"}" 1508428048076273000

But I wanna have just original message. Probably our situation is not a common use case, but may be it can be useful to forward data without extra formatting.

Wouldn't you rather have output in a format at least with a timestamp?

Most of apps already have timestamp in logs (e.g. rails, java, nodejs, telegraf)

@danielnelson
Copy link
Contributor

Makes sense, could be useful for passthrough without paying for any validation/parsing. I would just leave it hardcoded to the value field.

}

func (s *ValueSerializer) Serialize(metric telegraf.Metric) ([]byte, error) {
result := fmt.Sprintf("%v", metric.Fields()["value"])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this will print <nil>, if there is no value field I think it should return []byte("").

@danielnelson
Copy link
Contributor

@Paxa Are you still working on this?

@Paxa
Copy link
Author

Paxa commented Jul 27, 2018

No, I already don't need it

@danielnelson
Copy link
Contributor

Okay, I'm going to close, perhaps we will add something like this in the future, but for now there are plenty of other things more pressing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants