New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Heartbeat event format #4091

Merged
merged 1 commit into from Apr 28, 2017

Conversation

Projects
None yet
3 participants
@urso
Collaborator

urso commented Apr 23, 2017

PR addresses #3406 and #3838.

Additional work on extracting meta-data from connection layers should be done in follow-up PR.

Work done:

  • add look.Status
  • update event format per monitor
  • Rewrite heartbeat dialchain package:
    • simplify package
    • have connection layers add standardized connection data to the final event
    • add some more helpers
    • add some godocs

Event format changes:

Old Field Changes
monitor -> monitor.id
type
  • -> monitor.name
  • defaults to monitor.type if not configured
  • introduce new type fields with default 'monitor' until type is not required anymore
duration -> monitor.duration
scheme -> monitor.scheme
up
  • -> monitor.status
  • - replace boolean with keywords 'up', 'down'
host
  • -> monitor.host
  • -> resolve.host
ip
  • -> monitor.ip
  • -> resolve.ip
resolve_rtt -> resolve.rtt
icmp_rtt -> icmp.rtt
port -> tcp.port
tcp_connect_rtt -> tcp.rtt.connect
validate_rtt -> tcp.rtt.validate
socks5_connect_rtt -> socks5.rtt.connect
tls_handshake_rtt -> tls.rtt.handshake
url -> http.url
response.status -> http.response.status
rtt -> http.rtt.total
http_rtt -> http.rtt.response_header

New fields:

  • type: always set to "monitor". Can be removed when not required by libbeat anymore
  • monitor.type: the monitor type, e.g. "tcp", "icmp", "http".
  • http.rtt.validate: Duration between first byte of HTTP request being written and response being processed by validator. Duration based on already available network connection.
  • http.rtt.validate_body: Duration of validator required to read and validate the response
  • http.rtt.write_request: Duration of writing the complete HTTP request. Duration based on already available network connection.

Sample Events:

ICMP Event:

{
  "@timestamp": "2017-04-24T11:36:53.639Z",
  "beat": {
    "hostname": "mbp",
    "name": "mbp",
    "version": "6.0.0-alpha1"
  },
  "icmp": {
    "requests": 1,
    "rtt": {
      "us": 87
    }
  },
  "monitor": {
    "duration": {
      "us": 2557
    },
    "host": "localhost",
    "ip": "127.0.0.1",
    "name": "myname",
    "status": "up",
    "type": "icmp"
  },
  "resolve": {
    "host": "localhost",
    "ip": "127.0.0.1",
    "rtt": {
      "us": 2316
    }
  }
}

ICMP event for unknown host:

{
  "@timestamp": "2017-04-24T11:36:53.639Z",
  "beat": {
    "hostname": "mbp",
    "name": "mbp",
    "version": "6.0.0-alpha1"
  },
  "error": {
    "message": "lookup unknown-host: no such host",
    "type": "io"
  },
  "monitor": {
    "duration": {
      "us": 7628
    },
    "host": "unknown-host",
    "name": "myname",
    "status": "down",
    "type": "icmp"
  },
  "resolve": {
    "host": "unknown-host"
  }
}

TCP Event:

{
  "@timestamp": "2017-04-24T12:29:04.649Z",
  "beat": {
    "hostname": "mbp",
    "name": "mbp",
    "version": "6.0.0-alpha1"
  },
  "monitor": {
    "duration": {
      "us": 4461
    },
    "host": "localhost",
    "ip": "127.0.0.1",
    "name": "tcp",
    "scheme": "tcp", // Note: can be 'tcp' or 'tls' for TCP monitor. HTTP monitor sets 'http' or 'https'
    "status": "up",
    "type": "tcp"
  },
  "resolve": {
    "host": "localhost",
    "ip": "127.0.0.1",
    "rtt": {
      "us": 4136
    }
  },
  "tcp": {
    "port": "6060",
    "rtt": {
      "connect": {
        "us": 183
      }
    }
  }
}

TCP + TLS Event:

{
  "@timestamp": "2017-04-24T12:29:04.649Z",
  "beat": {
    "hostname": "mbp",
    "name": "mbp",
    "version": "6.0.0-alpha1"
  },
  "monitor": {
    "duration": {
      "us": 234591
    },
    "host": "play.golang.org",
    "ip": "216.58.213.241",
    "name": "tcp",
    "scheme": "tls",
    "status": "up",
    "type": "tcp"
  },
  "resolve": {
    "host": "play.golang.org",
    "ip": "216.58.213.241",
    "rtt": {
      "us": 51663
    }
  },
  "tcp": {
    "port": "443",
    "rtt": {
      "connect": {
        "us": 46140
      }
    }
  },
  "tls": {
    "rtt": {
      "handshake": {
        "us": 136605
      }
    }
  }
}

HTTP Event with 404:

{
  "@timestamp": "2017-04-24T13:44:44.718Z",
  "beat": {
    "hostname": "mbp",
    "name": "mbp",
    "version": "6.0.0-alpha1"
  },
  "error": {
    "message": "404 Not Found",
    "type": "validate"
  },
  "http": {
    "response": {
      "status": 404
    },
    "rtt": {
      "content": {
        "us": 244
      },
      "response_header": {
        "us": 225
      },
      "total": {
        "us": 537
      },
      "write_request": {
        "us": 63
      }
    },
    "url": "http://localhost:6060"
  },
  "monitor": {
    "duration": {
      "us": 4935
    },
    "host": "localhost",
    "ip": "127.0.0.1",
    "name": "http",
    "scheme": "http",
    "status": "down",
    "type": "http"
  },
  "resolve": {
    "host": "localhost",
    "ip": "127.0.0.1",
    "rtt": {
      "us": 4271
    }
  },
  "tcp": {
    "port": "6060",
    "rtt": {
      "connect": {
        "us": 247
      }
    }
  }
}

HTTPS event:

{
  "@timestamp": "2017-04-24T13:44:44.718Z",
  "beat": {
    "hostname": "mbp",
    "name": "mbp",
    "version": "6.0.0-alpha1"
  },
  "http": {
    "response": {
      "status": 200
    },
    "rtt": {
      "content": {
        "us": 167523
      },
      "response_header": {
        "us": 167433
      },
      "total": {
        "us": 339594
      },
      "write_request": {
        "us": 102
      }
    },
    "url": "https://play.golang.org"
  },
  "monitor": {
    "duration": {
      "us": 344043
    },
    "host": "play.golang.org",
    "ip": "216.58.213.241",
    "name": "http",
    "scheme": "https",
    "status": "up",
    "type": "http"
  },
  "resolve": {
    "host": "play.golang.org",
    "ip": "216.58.213.241",
    "rtt": {
      "us": 4272
    }
  },
  "tcp": {
    "port": "443",
    "rtt": {
      "connect": {
        "us": 38350
      }
    }
  },
  "tls": {
    "rtt": {
      "handshake": {
        "us": 133618
      }
    }
  }
}
func UDPDialer(name string, to time.Duration) NetDialer {
return NetDialer{name, transport.NetDialer(to)}
func (c *DialerChain) TestBuild() error {

This comment has been minimized.

@elasticmachine

elasticmachine Apr 23, 2017

[golint] reported by reviewdog 🐶
exported method DialerChain.TestBuild should have comment or be unexported

@@ -53,16 +52,16 @@ func (s IPSettings) Network() string {
return ""
}
func MakeSimpleJob(name, typ string, f func() (common.MapStr, error)) Job {
return MakeJob(name, typ, func() (common.MapStr, []TaskRunner, error) {
func MakeSimpleJob(jobName string, f func() (common.MapStr, error)) Job {

This comment has been minimized.

@elasticmachine

elasticmachine Apr 24, 2017

[golint] reported by reviewdog 🐶
exported function MakeSimpleJob should have comment or be unexported

func MakeJob(name, typ string, f func() (common.MapStr, []TaskRunner, error)) Job {
return &funcJob{name, typ, funcTask{func() (common.MapStr, []TaskRunner, error) {
return annotated(time.Now(), typ, f).Run()
func MakeJob(jobName string, f func() (common.MapStr, []TaskRunner, error)) Job {

This comment has been minimized.

@elasticmachine

elasticmachine Apr 24, 2017

[golint] reported by reviewdog 🐶
exported function MakeJob should have comment or be unexported

type IPSettings struct {
IPv4 bool `config:"ipv4"`
IPv6 bool `config:"ipv6"`
Mode PingMode `config:"mode"`
}
type JobSettings struct {

This comment has been minimized.

@elasticmachine

elasticmachine Apr 26, 2017

[golint] reported by reviewdog 🐶
exported type JobSettings should have comment or be unexported

Fields common.MapStr
}
type HostJobSettings struct {

This comment has been minimized.

@elasticmachine

elasticmachine Apr 26, 2017

[golint] reported by reviewdog 🐶
exported type HostJobSettings should have comment or be unexported

@@ -342,3 +444,42 @@ func filterIPs(ips []net.IP, filt func(net.IP) bool) []net.IP {
}
return out
}
func MakeJobSetting(name string) JobSettings {

This comment has been minimized.

@elasticmachine

elasticmachine Apr 26, 2017

[golint] reported by reviewdog 🐶
exported function MakeJobSetting should have comment or be unexported

return JobSettings{Name: name}
}
func (s JobSettings) WithFields(m common.MapStr) JobSettings {

This comment has been minimized.

@elasticmachine

elasticmachine Apr 26, 2017

[golint] reported by reviewdog 🐶
exported method JobSettings.WithFields should have comment or be unexported

return s
}
func (s *JobSettings) AddFields(m common.MapStr) { addFields(&s.Fields, m) }

This comment has been minimized.

@elasticmachine

elasticmachine Apr 26, 2017

[golint] reported by reviewdog 🐶
exported method JobSettings.AddFields should have comment or be unexported

func (s *JobSettings) AddFields(m common.MapStr) { addFields(&s.Fields, m) }
func MakeHostJobSettings(name, host string, ip IPSettings) HostJobSettings {

This comment has been minimized.

@elasticmachine

elasticmachine Apr 26, 2017

[golint] reported by reviewdog 🐶
exported function MakeHostJobSettings should have comment or be unexported

return HostJobSettings{Name: name, Host: host, IP: ip}
}
func (s HostJobSettings) WithFields(m common.MapStr) HostJobSettings {

This comment has been minimized.

@elasticmachine

elasticmachine Apr 26, 2017

[golint] reported by reviewdog 🐶
exported method HostJobSettings.WithFields should have comment or be unexported

return s
}
func (s *HostJobSettings) AddFields(m common.MapStr) { addFields(&s.Fields, m) }

This comment has been minimized.

@elasticmachine

elasticmachine Apr 26, 2017

[golint] reported by reviewdog 🐶
exported method HostJobSettings.AddFields should have comment or be unexported

@urso urso changed the title from [WIP] Heartbeat event format to Heartbeat event format Apr 26, 2017

@urso urso added review and removed in progress labels Apr 26, 2017

@ruflin

Nice cleanup. It makes the dialer part much easier to follow. As this is a major change on how it works I'm happy to merge it after the minor comments are adressed and then to further iterations from there.

- name: validate_body
type: group
description: |

This comment has been minimized.

@ruflin

ruflin Apr 26, 2017

Collaborator

Normally we use a >. I assume | also works?

This comment has been minimized.

@urso

urso Apr 26, 2017

Collaborator

oh, oversight from my side. in YAML one can use >, >- or |.

>: folded-style. Replace newlines with space. Multiple blank lines are replaced with newline (or space in some implementations)
|: literal block. Keeps newlines and blank lines.

I wanted to enforce the newline (or better blank line?) before "Note:" in the hopes of the asciidoc putting the note in a separate new line. That's why I used |.

If I remember correctly >- replaces a blank line (\n\n) by one newline (\n).

@@ -9,6 +9,7 @@ import (
"net/http"
"strconv"
"strings"

This comment has been minimized.

@ruflin

ruflin Apr 26, 2017

Collaborator

empty line too much?

This comment has been minimized.

@urso

urso Apr 26, 2017

Collaborator

interesting. so gimport woes here? Will fix.

@@ -41,25 +42,25 @@ func newHTTPMonitorHostJob(
return nil, err
}
hostname, port, err := splitHostnamePort(request)
hostname, _, err := splitHostnamePort(request)

This comment has been minimized.

@ruflin

ruflin Apr 26, 2017

Collaborator

port is excluded because it is already in the tcp layer, right?

This comment has been minimized.

@urso

urso Apr 26, 2017

Collaborator

Port is missing here. This constructor is used when a proxy is configured + name lookup happens by proxy itself. I did undo the tcp dialer to report the port number due to potential use of proxies.

case 1:
return makeTask(cont[0])
case
1:

This comment has been minimized.

@ruflin

ruflin Apr 26, 2017

Collaborator

is the 1: intentional on a new line?

This comment has been minimized.

@urso

urso Apr 26, 2017

Collaborator

ups.

return MakeCont(func() (common.MapStr, []TaskRunner, error) {
fmt.Println("run task")

This comment has been minimized.

@ruflin

ruflin Apr 26, 2017

Collaborator

leftover?

This comment has been minimized.

@urso

urso Apr 26, 2017

Collaborator

Thanks!

event, cont, err := fn()
fmt.Println("task returns: ", event, cont, err)

This comment has been minimized.

@ruflin

ruflin Apr 26, 2017

Collaborator

left over?

if err != nil {
if event == nil {
event = common.MapStr{}
}
fmt.Println("set error: ", err)

This comment has been minimized.

@ruflin

ruflin Apr 26, 2017

Collaborator

left over?

event["error"] = look.Reason(err)
}
if event != nil {
event.Update(common.MapStr{
status := look.Status(err)
fmt.Println("set status: ", status)

This comment has been minimized.

@ruflin

ruflin Apr 26, 2017

Collaborator

left over?

@ruflin

ruflin approved these changes Apr 27, 2017

Could you rebase and squash?

urso added a commit to urso/beats that referenced this pull request Apr 27, 2017

Heartbeat event format
See PR elastic#4091 for detailed list of event format changes and sample events.

- export transport.DialWith
- update heartbeat look:
  - add look.Status
  - add some missing godoc
- rewrite heartbeat dialchain package
  - simplify package
  - have connection layers add standardized connection data to the final event
  - add some more helpers
  - add some godocs
- update event format. See PR elastic#4091
- add more detailed duration measure to HTTP module
- update job settings
  - introduce explicit structs for job settings
  - job settings can contain static monitor fields to be added to every event
  - helpers can add more static monitor fields to settings
- update fields.yml structure
- update kibana dashboard
Heartbeat event format
See PR #4091 for detailed list of event format changes and sample events.

- export transport.DialWith
- update heartbeat look:
  - add look.Status
  - add some missing godoc
- rewrite heartbeat dialchain package
  - simplify package
  - have connection layers add standardized connection data to the final event
  - add some more helpers
  - add some godocs
- update event format. See PR #4091
- add more detailed duration measure to HTTP module
- update job settings
  - introduce explicit structs for job settings
  - job settings can contain static monitor fields to be added to every event
  - helpers can add more static monitor fields to settings
- update fields.yml structure
- update kibana dashboard

@ruflin ruflin merged commit 6b01fa9 into elastic:master Apr 28, 2017

6 checks passed

CLA Commit author is a member of Elasticsearch
Details
codecov/patch 0% of diff hit (within 100% threshold of 60.27%)
Details
codecov/project 64.64% (+4.36%) compared to 1344e4f
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
default Build finished.
Details

ramon-garcia added a commit to ramon-garcia/beats that referenced this pull request Dec 5, 2017

Heartbeat event format (elastic#4091)
See PR elastic#4091 for detailed list of event format changes and sample events.

- export transport.DialWith
- update heartbeat look:
  - add look.Status
  - add some missing godoc
- rewrite heartbeat dialchain package
  - simplify package
  - have connection layers add standardized connection data to the final event
  - add some more helpers
  - add some godocs
- update event format. See PR elastic#4091
- add more detailed duration measure to HTTP module
- update job settings
  - introduce explicit structs for job settings
  - job settings can contain static monitor fields to be added to every event
  - helpers can add more static monitor fields to settings
- update fields.yml structure
- update kibana dashboard

athom added a commit to athom/beats that referenced this pull request Jan 25, 2018

Heartbeat event format (elastic#4091)
See PR elastic#4091 for detailed list of event format changes and sample events.

- export transport.DialWith
- update heartbeat look:
  - add look.Status
  - add some missing godoc
- rewrite heartbeat dialchain package
  - simplify package
  - have connection layers add standardized connection data to the final event
  - add some more helpers
  - add some godocs
- update event format. See PR elastic#4091
- add more detailed duration measure to HTTP module
- update job settings
  - introduce explicit structs for job settings
  - job settings can contain static monitor fields to be added to every event
  - helpers can add more static monitor fields to settings
- update fields.yml structure
- update kibana dashboard
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment