http: superfluous response.WriteHeader call from github.com/prometheus/client_golang/...

[cameronkerrnz]

After upgrading to v0.16.1 I am seeing the following log message being logged not infrequently (a few times per minute perhaps).

snmp_exporter[26979]: level=info ts=2019-12-18T11:14:56.238Z caller=main.go:141 msg="Starting snmp_exporter" version="(version=0.16.1, branch=HEAD, revision=4b2bb401378cf40c409aba91ab25189be5670183)"
snmp_exporter[26979]: level=info ts=2019-12-18T11:14:56.238Z caller=main.go:142 msg="Build context" (gogo1.13.5,userroot@89bd31baeb1f,date20191210-15:23:21)=(MISSING)
snmp_exporter[26979]: level=info ts=2019-12-18T11:14:56.245Z caller=main.go:235 msg="Listening on address" address=:9116

snmp_exporter[26979]: 2019/12/19 00:24:05 http: superfluous response.WriteHeader call from github.com/prometheus/client_golang/prometheus/promhttp.httpError (http.go:344)

(its just the last line that is often logged)

I have also commented on prometheus/client_golang#672, which regards client_golang

Host operating system: output of uname -a

Linux ... 3.10.0-1062.9.1.el7.x86_64 #1 SMP Mon Dec 2 08:31:54 EST 2019 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.7 (Maipo)

snmp_exporter version: output of snmp_exporter -version

v0.16.1 is the first release that exhibits the problem in a production environment

v0.15.0 does not exhibit the problem

What device/snmpwalk OID are you using?

For this, mostly Cisco Wireless Access Points over SNMP v2c. The configuration for this is fairly large, but here it is (its actually a Jinja2 template, but that's only to inject auth details)

  cisco_wireless_controllers:
    {{ snmp_credentials.cisco_wireless_controllers | to_nice_yaml(indent=2) | indent(4) }}
    walk:
      - 1.3.6.1.4.1.9.9.513.1.1.1 # CISCO-LWAPP-AP-MIB::cLApTable
      - 1.3.6.1.4.1.14179.2.1.1   # AIRESPACE-WIRELESS-MIB::bsnDot11EssTable
    lookups:
      - source_indexes: ['cLApSysMacAddress']
        lookup: cLApName
        drop_source_indexes: true
      - source_indexes: ['bsnDot11EssIndex']
        lookup: bsnDot11EssSsid
        drop_source_indexes: true

    overrides:

      # CISCO-LWAPP-AP-MIB::cLApTable, which is very wide and often useful. We omit many columns
      # It would be very helpful if we could specify a whitelist of table columns
      #
      cLApSysMacAddress:  # index for each row
        type: PhysAddress48
        ignore: false
      cLApIfMacAddress:
        type: PhysAddress48
        ignore: false
      cLApMaxNumberOfDot11Slots:
        ignore: true
      cLApEntPhysicalIndex:
        ignore: true
      cLApName:
        ignore: false
      cLApUpTime:
        ignore: false
      cLLwappUpTime:
        ignore: false
      cLLwappJoinTakenTime:
        ignore: false
      cLApMaxNumberOfEthernetSlots:
        ignore: true
      cLApPrimaryControllerAddressType:
        ignore: true
      cLApPrimaryControllerAddress:
        ignore: true
      cLApSecondaryControllerAddressType:
        ignore: true
      cLApSecondaryControllerAddress:
        ignore: true
      cLApTertiaryControllerAddressType:
        ignore: true
      cLApTertiaryControllerAddress:
        ignore: true
      cLApLastRebootReason:
        ignore: false
      cLApEncryptionEnable:
        ignore: true
      cLApFailoverPriority:
        ignore: true
      cLApPowerStatus:
        ignore: true
      cLApTelnetEnable:
        ignore: true
      cLApSshEnable:
        ignore: true
      cLApPreStdStateEnabled:
        ignore: true
      cLApPwrInjectorStateEnabled:
        ignore: true
      cLApPwrInjectorSelection:
        ignore: true
      cLApPwrInjectorSwMacAddr:
        ignore: true
      cLApWipsEnable:
        ignore: true
      cLApMonitorModeOptimization:
        ignore: true
      cLApDomainName:
        ignore: true
      cLApNameServerAddressType:
        ignore: true
      cLApNameServerAddress:
        ignore: true
      cLApAMSDUEnable:
        ignore: true
      cLApEncryptionSupported:
        ignore: true
      cLApRogueDetectionEnabled:
        ignore: true
      cLApTcpMss:
        ignore: true
      cLApDataEncryptionStatus:
        ignore: true
      cLApNsiKey:
        ignore: true
      cLApAdminStatus:
        ignore: true
      cLApPortNumber:
        ignore: true
      cLApRetransmitCount:
        ignore: true
      cLApRetransmitTimeout:
        ignore: true
      cLApVenueConfigVenueGroup:
        ignore: true
      cLApVenueConfigVenueType:
        ignore: true
      cLApVenueConfigVenueName:
        ignore: true
      cLApVenueConfigLanguage:
        ignore: true
      cLApLEDState:
        ignore: true
      cLApTrunkVlan:
        ignore: true
      cLApTrunkVlanStatus:
        ignore: true
      cLApLocation:
        ignore: true
      cLApSubMode:
        ignore: true
      cLApAssocCount:
        ignore: false
      cLApAssocFailResourceCount:
        ignore: false
      cLApRealTimeStatsModeEnabled:
        ignore: true
      cLApAssociatedClientCount:
        ignore: false
      cLApMemoryCurrentUsage:
        ignore: false
      cLApMemoryAverageUsage:
        ignore: false
      cLApCpuCurrentUsage:
        ignore: false
      cLApCpuAverageUsage:
        ignore: false
      cLApUpgradeFromVersion:
        ignore: true
      cLApUpgradeToVersion:
        ignore: true
      cLApUpgradeFailureCause:
        ignore: true
      cLApMaxClientLimitNumberTrap:
        ignore: true
      cLApMaxClientLimitCause:
        ignore: true
      cLApMaxClientLimitSet:
        ignore: true
      cLApFloorLabel:
        ignore: false   # just curious; suspect these may be all over the place
      cLApConnectCount:
        ignore: false
      cLApReassocSuccCount:
        ignore: false
      cLApReassocFailCount:
        ignore: false
      cLApAssocFailCountByRate:
        ignore: false
      cLApAbnormalOfflineCount:
        ignore: false
      cLApActiveClientCount:
        ignore: false
      cLApAssocFailCountForRssiLow:
        ignore: false
      cLApSysNetId:
        ignore: true
      cLApAssocFailTimes:
        ignore: false
      cLApAntennaBandMode:
        ignore: true
      cLApHeartBeatRspAvgTime:
        ignore: false
      cLApEchoRequestCount:
        ignore: false
      cLApEchoResponseLossCount:
        ignore: false
      cLApModuleInserted:
        ignore: true
      cLApEnableModule:
        ignore: true
      cLApIsUniversal:
        ignore: true
      cLApUniversalPrimeStatus:
        ignore: true
      cLApIsMaster:
        ignore: true
      cLApBleFWDownloadStatus:
        ignore: true
      cLApDot11XorDartConnectorStatus:
        ignore: true
      cLApCtsSxpDefaultPassword:
        ignore: true
      cLApCtsSxpState:
        ignore: true
      cLApCtsSxpListenerMinHoldtime:
        ignore: true
      cLApCtsSxpListenerMaxHoldtime:
        ignore: true
      cLApCtsSxpReconcilePeriod:
        ignore: true
      cLApCtsSxpRetryPeriod:
        ignore: true
      cLApCtsSxpSpeakerHoldTime:
        ignore: true
      cLApCtsSxpSpeakerKeepAlive:
        ignore: true
      cLApCtsInlineTagStatus:
        ignore: true
      cLApCtsSgaclStatus:
        ignore: true
      cLApCtsOverrideStatus:
        ignore: true

      # AIRESPACE-WIRELESS-MIB::bsnDot11EssTable contains various information, but
      # the most useful thing it provides is number of clients on each WLC/SSID
      #
      bsnDot11EssIndex:
        ignore: true
      bsnDot11EssSsid:
        ignore: false
      bsnDot11EssSessionTimeout:
        ignore: true
      bsnDot11EssMacFiltering:
        ignore: true
      bsnDot11EssAdminStatus:
        ignore: true
      bsnDot11EssSecurityAuthType:
        ignore: true
      bsnDot11EssStaticWEPSecurity:
        ignore: true
      bsnDot11EssStaticWEPEncryptionType:
        ignore: true
      bsnDot11EssStaticWEPDefaultKey:
        ignore: true
      bsnDot11EssStaticWEPKeyIndex:
        ignore: true
      bsnDot11EssStaticWEPKeyFormat:
        ignore: true
      bsnDot11Ess8021xSecurity:
        ignore: true
      bsnDot11Ess8021xEncryptionType:
        ignore: true
      bsnDot11EssWPASecurity:
        ignore: true
      bsnDot11EssWPAEncryptionType:
        ignore: true
      bsnDot11EssIpsecSecurity:
        ignore: true
      bsnDot11EssVpnEncrTransform:
        ignore: true
      bsnDot11EssVpnAuthTransform:
        ignore: true
      bsnDot11EssVpnIkeAuthMode:
        ignore: true
      bsnDot11EssVpnSharedKey:
        ignore: true
      bsnDot11EssVpnSharedKeySize:
        ignore: true
      bsnDot11EssVpnIkePhase1Mode:
        ignore: true
      bsnDot11EssVpnIkeLifetime:
        ignore: true
      bsnDot11EssVpnIkeDHGroup:
        ignore: true
      bsnDot11EssIpsecPassthruSecurity:
        ignore: true
      bsnDot11EssVpnPassthruGateway:
        ignore: true
      bsnDot11EssWebSecurity:
        ignore: true
      bsnDot11EssRadioPolicy:
        ignore: true
      bsnDot11EssQualityOfService:
        ignore: true
      bsnDot11EssDhcpRequired:
        ignore: true
      bsnDot11EssDhcpServerIpAddress:
        ignore: true
      bsnDot11EssVpnContivityMode:
        ignore: true
      bsnDot11EssVpnQotdServerAddress:
        ignore: true
      bsnDot11EssBlacklistTimeout:
        ignore: true
      bsnDot11EssNumberOfMobileStations:
        ignore: false
      bsnDot11EssWebPassthru:
        ignore: true
      bsnDot11EssCraniteSecurity:
        ignore: true
      bsnDot11EssBlacklistingCapability:
        ignore: true
      bsnDot11EssInterfaceName:
        ignore: true
      bsnDot11EssAclName:
        ignore: true
      bsnDot11EssAAAOverride:
        ignore: true
      bsnDot11EssWPAAuthKeyMgmtMode:
        ignore: true
      bsnDot11EssWPAAuthPresharedKey:
        ignore: true
      bsnDot11EssWepAllowSharedKeyAuth:
        ignore: true
      bsnDot11EssFortressSecurity:
        ignore: true
      bsnDot11EssL2tpSecurity:
        ignore: true
      bsnDot11EssWPAAuthPresharedKeyHex:
        ignore: true
      bsnDot11EssBroadcastSsid:
        ignore: true
      bsnDot11EssExternalPolicyValidation:
        ignore: true
      bsnDot11EssRSNSecurity:
        ignore: true
      bsnDot11EssRSNWPACompatibilityMode:
        ignore: true
      bsnDot11EssRSNAllowTKIPClients:
        ignore: true
      bsnDot11EssRSNAuthKeyMgmtMode:
        ignore: true
      bsnDot11EssRSNAuthPresharedKey:
        ignore: true
      bsnDot11EssRSNAuthPresharedKeyHex:
        ignore: true
      bsnDot11EssIPv6Bridging:
        ignore: true
      bsnDot11EssRowStatus:
        ignore: true
      bsnDot11EssWmePolicySetting:
        ignore: true
      bsnDot11Ess80211ePolicySetting:
        ignore: true
      bsnDot11EssWebPassthroughEmail:
        ignore: true
      bsnDot11Ess7920PhoneSupport:
        ignore: true
      bsnDot11EssDhcpScopeName:
        ignore: true
      bsnDot11EssCentralNatEnable:
        ignore: true
      bsnDot11EssP2PBlockingEnable:
        ignore: true
      bsnDot11EssRadiusAuthPrimaryServer:
        ignore: true
      bsnDot11EssRadiusAuthSecondaryServer:
        ignore: true
      bsnDot11EssRadiusAuthTertiaryServer:
        ignore: true
      bsnDot11EssRadiusAcctPrimaryServer:
        ignore: true
      bsnDot11EssRadiusAcctSecondaryServer:
        ignore: true
      bsnDot11EssRadiusAcctTertiaryServer:
        ignore: true

If this is a new device, please link to the MIB(s).

What did you do that produced an error?

Upgraded from v0.15.0 to v0.16.1

What did you expect to see?

Healthy target report in Prometheus

What did you see instead?

About 3 out of 11 (wireless access points) showed as having a 'context deadline exception'. However, it is possible that previously there was sporadic silent failures looking at the occassional voids in Grafana.

[brian-brazil]

This is going to be a client_golang issue, I'd suggest filing a new issue there.

[cameronkerrnz]

Closing this issue here because its rightfully a problem in client_golang