agent discards the whole batch of metrics in case of a serialization issue in a single metric

[ayansen]

Relevant telegraf.conf:

[global_tags]  
  interval = "30"
  m_application = "mt-relay"

[agent]
  interval = "30s"
  flush_interval = "30s"
  metric_buffer_limit = 10000 
  debug = true 
 
[[outputs.http]]
  url = "http://localhost:1337/influx/metric"
  data_format = "influx"

[[inputs.prometheus]]
  monitor_kubernetes_pods = true
  urls = ["http://prometheus-server/metrics"]

System info:

Telegraf 1.9.5
Ubuntu 16.04 LTS

Steps to reproduce:

1. ... expose a prometheus metric who which has gauge metric with an invalid value

Expected behavior:

The metric should be discarded by the agent

Actual behavior:

The whole batch of metrics get discarded due to a serialization failure in a single metric

Additional info:

2019-05-15T06:49:00Z E! [agent] Error writing to output [http]: "kafka_consumer_fetch_latency_max,address=10.1.206.168,app=enriched-order-events-producer,client_id=enriched-order-events-producer-84126f3b-093b-41d3-bece-5b1cf8deebe1-StreamThread-1-consumer,host=mt-relay-5b86dd7bcd-k9g6n,interval=30,m_application=enriched-order-events-producer,m_region=us-east-1,namespace=neurons,pod-template-hash=732570943,pod_name=enriched-order-events-producer-c769c4f87-x699z,prometheus.io/path=/metrics,prometheus.io/port=8080,prometheus.io/scrape=true,url=http://10.1.206.168:8080/metrics": no serializable fields
2019-05-15T06:49:00Z D! [serializers.influx] could not serialize field "gauge": is Inf; discarding field

Code line : https://github.com/influxdata/telegraf/blob/master/agent/agent.go#L543