vsphere Client.Timeout exceeded while awaiting headers

[ragonlan]

Relevant telegraf.conf:

file telegraf.d/manu010.vm.conf

[[inputs.vsphere]]
   ## List of vCenter URLs to be monitored. These three lines must be uncommented
   ## and edited for the plugin to work.
   vcenters = [ "https://manu010.vm/sdk"]
   username = "xxxx@DOM1.priv"
   password = "XXXXX"

   ## VMs
   ## Typical VM metrics (if omitted or empty, all metrics are collected)
   vm_metric_include = []
   vm_metric_exclude = ["*"] ## Nothing is excluded by default
   vm_instances = false ## true by default

   ## Hosts
   ## Typical host metrics (if omitted or empty, all metrics are collected)
   host_metric_include = []
   host_metric_exclude = ["*"] ## Nothing excluded by default
   host_instances = false ## true by default

   ## Clusters
   cluster_metric_include = [] ## if omitted or empty, all metrics are collected
   cluster_metric_exclude = [] ## Nothing excluded by default
   cluster_instances = true ## true by default

   ## Datastores
   datastore_metric_include = [] ## if omitted or empty, all metrics are collected
   datastore_metric_exclude = ["*"] ## Nothing excluded by default
   datastore_instances = false ## false by default for Datastores only

   ## Datacenters
   datacenter_metric_include = [] ## if omitted or empty, all metrics are collected
   datacenter_metric_exclude = [] ## Datacenters are not collected by default.
   datacenter_instances = true ## false by default for Datastores only

   max_query_objects = 100
   max_query_metrics = 100
   collect_concurrency = 30
   discover_concurrency = 10
   force_discover_on_init = false
   object_discovery_interval = "1600s"
   insecure_skip_verify = true

file telegraf.d/manu001.vm.conf

[[inputs.vsphere]]
   vcenters = [ "https://manu001.vm/sdk"]
   username = "XXXX@XXX.dom2.priv"
   password = "XXXXX"

   ## VMs
   ## Typical VM metrics (if omitted or empty, all metrics are collected)
   vm_metric_include = []
   vm_metric_exclude = ["*"] ## Nothing is excluded by default
   vm_instances = false ## true by default

   ## Hosts
   ## Typical host metrics (if omitted or empty, all metrics are collected)
   host_metric_include = []
   host_metric_exclude = ["*"] ## Nothing excluded by default
   host_instances = false ## true by default

   ## Clusters
   cluster_metric_include = [] ## if omitted or empty, all metrics are collected
   cluster_metric_exclude = [] ## Nothing excluded by default
   cluster_instances = true ## true by default

   ## Datastores
   datastore_metric_include = [] ## if omitted or empty, all metrics are collected
   datastore_metric_exclude = ["*"] ## Nothing excluded by default
   datastore_instances = false ## false by default for Datastores only

   ## Datacenters
   datacenter_metric_include = [] ## if omitted or empty, all metrics are collected
   datacenter_metric_exclude = [] ## Datacenters are not collected by default.
   datacenter_instances = true ## false by default for Datastores only

   max_query_objects = 100
   max_query_metrics = 1000
   collect_concurrency = 30
   discover_concurrency = 10
   force_discover_on_init = false
   object_discovery_interval = "1600s"
   insecure_skip_verify = true

### System info:
OS: Debian 8.11
ii  telegraf                                              1.9.0-1                         amd64                           Plugin-driven server agent for reporting metrics into InfluxDB.

Inmediately after telegraf stars manu100.vm outputs this error:

2018-12-11T11:25:14Z D! [input.vsphere]: Starting plugin
2018-12-11T11:25:14Z D! [input.vsphere]: Starting plugin
2018-12-11T11:25:14Z D! [input.vsphere]: Creating client: manu010.vm
2018-12-11T11:25:14Z D! [input.vsphere]: Creating client: manu001.vm
2018-12-11T11:25:14Z E! [input.vsphere]: Error in discovery for manu010.vm: ServerFaultCode: Permission to perform this operation was denied.
2018-12-11T11:25:14Z D! [input.vsphere] vCenter maxQueryMetrics is defined: -1
2018-12-11T11:25:14Z D! [input.vsphere] vCenter says max_query_metrics should be 10000
2018-12-11T11:25:14Z D! [input.vsphere]: Discover new objects for manu001.vm
2018-12-11T11:25:14Z D! [input.vsphere] Discovering resources for datacenter
2018-12-11T11:25:14Z D! [input.vsphere]: No parent found for Folder:group-d1 (ascending from Folder:group-d1)
2018-12-11T11:25:15Z D! [input.vsphere] Discovering resources for cluster
2018-12-11T11:25:15Z D! [input.vsphere] Discovering resources for host
2018-12-11T11:25:16Z D! [input.vsphere] Discovering resources for vm
2018-12-11T11:25:16Z D! [input.vsphere] Discovering resources for datastore

No metrics is stored in influxdb for manu010.vm.

From time to time appears errors for the other vcenter, manu001.vm:

2018-12-11T11:31:00Z E! [inputs.vsphere]: Error in plugin: Post https://manu001.vm/sdk: context deadline exceeded
2018-12-11T11:31:00Z E! [inputs.vsphere]: Error in plugin: Post https://manu001.vm/sdk: context deadline exceeded

There is metrics for manu001 but not continually. Fails for an hour and after that continues getting metrics again.

[prydin]

Add timeout = "60s" to the config!

[danielnelson]

@prydin Can you help me understand this recommendation better, isn't the default already timeout 60s in 1.9.0?

[prydin]

I sas assuming this was 1.8.x, but maybe not.

[ragonlan]

Yes, I run telegraf 1.9:
telegraf 1.9.0-1 amd64
in a Debian 9.6

I set the value 60 but the timeouts keep coming. Actually I putted it to 120s and nothing changed. A pity that debug mode does not show what is doing when timeouts occur or the SeverFaultCode.

[prydin]

Try this with version 1.10 of telegraf. There's been some improvement of the timeout handling.

[Venopsis]

Hi,
For info, got same issue and the problem was coming from configuration I copied-pasted from website. In timeout the guy forgot the "s" at the end of timeout ="1800s".

Good evening!

[rwaffen]

tried with 1.10.3 and timeout = "1800s" ... the messages are still comming ...

[sspaink]

There has been a lot of changes to the vsphere plugin since this bug report was opened, is this still an issue with the latest version of Telegraf? If it is please post config and logs and we can investigate this problem.

[telegraf-tiger]

Hello! I am closing this issue due to inactivity. I hope you were able to resolve your problem, if not please try posting this question in our Community Slack or Community Page. Thank you!