Test for existing http response header

[kingfisher77]

Problem

We would like to test for the existence of a http response header.

Description

http response header like "x-robots-tag: noindex,nofollow" could be tested e.g. for websites which should not be indexed by search engines.

Importance

Important to have

Value proposition

The quality of an automated application deployment also depends on the existence of http response headers. With this function in Netdata, this quality level can be better measured.

Proposed implementation

Feature could be implemented as an addition to HTTP Endpoints.

[ilyam8]

Implemented in netdata/go.d.plugin#1451

Example configurations

# "value" is a pattern, syntax: https://github.com/netdata/go.d.plugin/tree/master/pkg/matcher#supported-format


jobs:
    # The "X-Robots-Tag" header must be present in the HTTP response header,
    # but the value of the header does not matter.
    # This config checks for the presence of the header regardless of its value.
  - name: local
    url: http://127.0.0.1:8080
    header_match:
      - key: X-Robots-Tag

    # The "X-Robots-Tag" header must be present in the HTTP response header
    # only if its value is equal to "noindex, nofollow".
    # This config checks both the presence of the header and its value.
  - name: local
    url: http://127.0.0.1:8080
    header_match:
      - key: X-Robots-Tag
        value: '= noindex,nofollow'

    # The "X-Robots-Tag" header must not be present in the HTTP response header
    # but the value of the header does not matter.
    # This config checks for the presence of the header regardless of its value.
  - name: local
    url: http://127.0.0.1:8080
    header_match:
      - key: X-Robots-Tag
        exclude: yes

    # The "X-Robots-Tag" header must not be present in the HTTP response header
    # only if its value is equal to "noindex, nofollow".
    # This config checks both the presence of the header and its value.
  - name: local
    url: http://127.0.0.1:8080
    header_match:
      - key: X-Robots-Tag
        exclude: yes
        value: '= noindex,nofollow'

If you think I need to adjust something, feel free to comment.

---

It will be in the nightly release later this week.

[kingfisher77]

Hey, this is really cool. Can't wait to test it! We will report.

[ilyam8]

Should be available in today's nightly release (tomorrow).

[kingfisher77]

Thank you - is the new feature part of httpcheck? Should the config go into httpcheck.conf?

[ilyam8]

Yes, see configuration examples in #16662 (comment).

[netdata-community-bot]

This issue has been mentioned on the Netdata Community Forums. There might be relevant details there:

https://community.netdata.cloud/t/how-to-test-if-a-response-header-exist-and-or-the-value-matches/4976/3

[kingfisher77]

@ilyam8 It works, but without alert notification.

www-example-com does not have a x-robot--tag header.

# TEST: www.example.com shall expose a x-robots-tag
  - name: www-example-com-x-robots-tag
    url: https://www.example.com
    header_match:
      - key: X-Robots-Tag
        value: '= noindex,nofollow'

netdata@hostname:/usr/libexec/netdata/plugins.d$ ./go.d.plugin -d -m httpcheck
BEGIN 'httpcheck_www-example-com-x-robots-tag.request_status' 4999913
SET 'success' = 0
SET 'no_connection' = 0
SET 'timeout' = 0
SET 'redirect' = 0
SET 'bad_content' = 0
SET 'bad_status' = 0
SET 'bad_header' = 1
END

The chart shows a line at the 0 level. Expectation is to receive an alter notification. What do you think?

[ilyam8]

@kingfisher77 you are right, I forgot to add the alert. Added in #16736.

[kingfisher77]

Hi @ilyam8 The change in health.d/httpcheck.conf did not yet arrived. We are on v1.44.0-148-nightly. health.d/httpcheck.conf is still from 28.11. Do we miss something?

[ilyam8]

Are you using custom health.d/httpcheck.conf? Do you have it in /etc/netdadta/? Netdata prefers user configuration over stock: it ignores /usr/lib/netdata/conf.d/health.d/httpcheck.conf (updated version) if /etc/netdadta/health.d/httpcheck.conf is available. If so, you need to modify your /etc/netdadta/health.d/httpcheck.conf.

[kingfisher77]

I understand. Sorry for not thinking twice. I did indeed have a health.d/httpcheck.conf with no changes of my own. Maybe to try something out. Will keep an eye on the new feature now!

[kingfisher77]

The space between the "=" and the value "noindex,nofollow" was too much in our case. If we remove it, it works:

  - name: example-com-x-robots-tag
    url: https://example.com
    username: user
    password: password
    header_match:
      - key: X-Robots-Tag
        value: '=noindex,nofollow'

And: in our case there is no space between noindex and nofollow.

[kingfisher77]

Hi @ilyam8
we believe we have some "ghost warnings" triggered by the new header_match function.

The job runs every 60 seconds. There is another job for the same URL. Is there possibly a connection with two jobs on the same URL?

  - name: www-example-com
    url: https://www.example.com
    headers:
      Server: Apache
    status_accepted:
      - 200
    update_every: 60
    timeout: 3

# www.example.com shall not expose a x-robots-tag
  - name: www-example-com-x-robots-tag
    url: https://www.example-com
    update_every: 60
    header_match:
      - key: X-Robots-Tag
        exclude: yes

Debug:

DBG module/job.go:251 skip the tick due to previous run hasn't been finished collector=httpcheck job=www-example-com
DBG module/job.go:251 skip the tick due to previous run hasn't been finished collector=httpcheck job=www-example-com-x-robots-tag
DBG httpcheck/collect.go:48 Get "https://www.exemple.com": context deadline exceeded (Client.Timeout exceeded while awaiting headers) collector=httpcheck job=www-example-com-x-robots-
tag

Alert email:

Alert: httpcheck_web_service_timeouts
Chart: httpcheck_www-example-com-x-robots-tag.request_status
Raised to warning, for 0 second

[kingfisher77]

We run a curl on the domain and the header in a loop every 2 seconds and get no error message.

[ilyam8]

httpcheck_web_service_timeouts

DBG httpcheck/collect.go:48 Get "https://www.exemple.com/": context deadline exceeded (Client.Timeout exceeded while awaiting headers) collector=httpcheck job=www-example-com-x-robots-

It is this alert - timeout on request, no unexpected headers.

netdata/health/health.d/httpcheck.conf

Lines 60 to 73 in a3d5e6c

	template: httpcheck_web_service_timeouts
	on: httpcheck.status
	class: Latency
	type: Web Server
	component: HTTP endpoint
	lookup: average -5m unaligned percentage of timeout
	every: 10s
	units: %
	warn: $this >= 10 AND $this < 40
	crit: $this >= 40
	delay: down 5m multiplier 1.5 max 1h
	summary: HTTP check for ${label:url} timeouts
	info: Percentage of timed-out HTTP requests to ${label:url} in the last 5 minutes
	to: webmaster

The default timeout is 1 second, you can increase it using the timeout setting.

[kingfisher77]

Ok. Thank you.