Too high Http Check response times

[cccaballero]

Bug report summary

I have two VPSs, on VPS1 I have a couple of websites, and on VPS2 I have netdata installed and Http Check configured to monitor the websites hosted on VPS1. Netdata is reporting response times of around 4000 ms (very long) for the web while, if I use curl from the VPS2 itself to measure response times, it returns around 350 ms which is expected.

OS / Environment

/etc/lsb-release:DISTRIB_ID=Ubuntu
/etc/lsb-release:DISTRIB_RELEASE=18.04
/etc/lsb-release:DISTRIB_CODENAME=bionic
/etc/lsb-release:DISTRIB_DESCRIPTION="Ubuntu 18.04.4 LTS"
/etc/os-release:NAME="Ubuntu"
/etc/os-release:VERSION="18.04.4 LTS (Bionic Beaver)"
/etc/os-release:ID=ubuntu
/etc/os-release:ID_LIKE=debian
/etc/os-release:PRETTY_NAME="Ubuntu 18.04.4 LTS"
/etc/os-release:VERSION_ID="18.04"
/etc/os-release:HOME_URL="https://www.ubuntu.com/"
/etc/os-release:SUPPORT_URL="https://help.ubuntu.com/"
/etc/os-release:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
/etc/os-release:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
/etc/os-release:VERSION_CODENAME=bionic
/etc/os-release:UBUNTU_CODENAME=bionic

Netdata version

netdata v1.28.0-47-nightly

Component Name

collectors

Steps To Reproduce

1. Install netdata
2. configure httpcheck using ./edit-config go.d/httpcheck.conf
3. Include in ./edit-config go.d/httpcheck.conf the lines:

jobs:
  - name: service name
    url: https://myurl.com
    status_accepted: [200]
    update_every: 300
    timeout: 5

4. Check HTTP Response Time in dashboard (around 4000 ms in my website)
5. Test web response time using curl:

curl 'https://myurl.com' -H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: en-US,en;q=0.8,ja;q=0.6' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.86 Safari/537.36' -H 'Connection: keep-alive' --compressed -s -o /dev/null -w  "%{time_starttransfer}\n"

(around 350 ms in my website)

Expected behavior

Netdata HTTP Response Time and curl response time are expected to be similar.

[ilyam8]

time_starttransfer seems is Time to first byte. Httpcheck module doesn't measure it.

https://github.com/netdata/go.d.plugin/blob/47505d59438c246a7f5e3f82823011e1d2d2cd77/modules/httpcheck/collect.go#L33-L35

I think there is what we need - https://github.com/davecheney/httpstat

• dns lookup time
• tcp connect time
• tls handshake time
• server processing time
• content transfer time

[cccaballero]

@ilyam8 thanks for your answer and help. I have just tested using httpstat and I'm still getting really high response times in comparison. For example, the site https://daxslab.com using httpstat is reporting ~400 ms, but netdata values are around 2000 ms, with peaks of 4000+ (all measurements taken from the same server)

[ilyam8]

@cccaballero 🤔 show debug output

[ilyam8]

Cleaning up issues. Closing due to no response.