feat(serializers): Add a serializer based on go templates

[gazpachoking]

Required for all PRs

• Updated associated README.md.
• Wrote appropriate unit tests.
• Pull request title or commits are in conventional commit format

resolves #13637

Adds a new serializer that lets the user define a go template for the serialization format.

I haven't really used Go before, so this might need a bit of review.

[gazpachoking]

Hmmm. I was very cut and pasty with this PR. Now reading #13606, I'm thinking I shouldn't fully copy the way it's being done in the template processor plugin, as it looks like we are moving away from that method.

[srebhan]

Hmmm. I was very cut and pasty with this PR. Now reading #13606, I'm thinking I shouldn't fully copy the way it's being done in the template processor plugin, as it looks like we are moving away from that method.

@gazpachoking right. You can just use the metric directly and the TemplateMetric interface. You might want to take a look at the lookup processor...

[srebhan]

@gazpachoking this already looks very beautiful, thanks for your contribution. I do have some smaller comments and one request. The request is regarding the single vs. batch serialization. I think you should match the template input to what is intended by the user as the user cannot know which method is called.

[gazpachoking]

I don't believe the latest ci failure is to do with my changes.

[gazpachoking]

Hmm, working with the go templating engine a bit has shown that it's a bit inflexible (without some helpers.) I'm thinking we should include the sprig functions here. Thoughts?

[powersj]

Hmm, working with the go templating engine a bit has shown that it's a bit inflexible (without some helpers.) I'm thinking we should include the sprig functions here. Thoughts?

Hmm sprig doesn't appear to be maintained or updated in ~year. Is there a use-case that you could provide for using them versus not using the library?

[gazpachoking]

Hmm sprig doesn't appear to be maintained or updated in ~year. Is there a use-case that you could provide for using them versus not using the library?

I was having trouble trying to comma separate tags or metrics without putting a trailing comma. Really it's sorta convoluted even with sprig with tags, since their dict functions don't work directly on the tags map (they work fine on the fields map though.) Here's what I did using sprig, I couldn't even figure out how to do it at all without it.

template = "{{$tags := .Tags|toJson|fromJson}}{{range $tags|keys|initial}}{{.}}={{get $tags .}}, {{end}}{{$tags|keys|last}}={{$tags|values|last}}\n"

The toJson fromJson dance is the convoluted bit I mentioned. That converts the tag map from map[string]string to map[string]interface{} which is what's needed to use the sprig dict helpers.

I picked it because Helm uses it, and it's already used in the postgresql output plugin. I'm fine with removing it though, but I would like to figure out if doing commas between things is possible, as that seems like it might be a common use case.

[srebhan]

Nice update @gazpachoking! Have some more comments...

[srebhan]

I love the PR. @gazpachoking can you please show one config in the README where all available options are included to give users an overview of what is there!? You might then show more config-examples below for different use-cases....

[gazpachoking]

Added the batch_template option in the example. Also noticed that the TOML parser doesn't allow embedding double quotes inside a basic multi-line string, which the spec says should be fine. (it also doesn't seem to like single quotes within a multi-line literal string)

[srebhan]

Looks good to me. Thanks @gazpachoking for the cool serializer!

[telegraf-tiger]

Download PR build artifacts for linux_amd64.tar.gz, darwin_amd64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_arm64.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz

[ven0hm]

Thank you @gazpachoking! This serializer is exactly what I need!

When I try to use it with [[outputs.file]] it works just fine.
But when I try the same with [[outputs.socket_writer]], it throws an error
plugin outputs.socket_writer: line 48: configuration specified the fields ["use_batch_format"], but they weren't used

Full config:

[[outputs.socket_writer]]
  address = "tcp://localhost:80"
  keep_alive_period = "10m"
  content_encoding = "identity"
  data_format = "template"
  use_batch_format = true
  batch_template = '''
{{range $metric := . -}}
{{range $metric.Fields | keys | initial -}}
{{.}}={{get $metric.Fields .}}, {{end}}
{{- $metric.Tag "name"}}={{$metric.Fields|values|last}}
{{end -}}
'''

I'm using telegraf-1.28.0~ff54c603_linux_amd64.tar.gz from the link above.
Should I submit this as a bug, or I'm just configuring it wrong?

[powersj]

["use_batch_format"], but they weren't used

This is not a valid option for the socket_writer. The socket writer writes each metric as it comes, not in batches.

[gazpachoking]

@ven0hm Do you need the batch format for your purposes, or are individual metrics fine? Allowing batch format in the socket_writer plugin would be a valid feature request.