Introduce "cilium-health", a new tool for investigating cluster connectivity issues. #2052

joestringer · 2017-11-16T07:56:35Z

This series extends the Cilium API to provide access to known nodes in the cluster and their IPs via /healthz (The same as #2139), and adds a new program to the tree: cilium-health.

cilium-health can be run as a daemon, which will periodically probe other nodes in the cluster for connectivity, starting with basic ping (ICMP) access and HTTP probes to other nodes. By default, when cilium is run, it will launch cilium-health locally. The results of the probes will be cached and exposed over a REST API. cilium-health can also be run as a commandline client for the daemon, to fetch the connectivity status and format it. Currently, cilium-health exposes:

GET /hello
- Returns success code with empty body, used for probing
GET /healthz
- Daemon uptime
- Node load (1min/5min/15min)
- Cilium /healthz
GET /status
- Timestamp of probe
- Local node name
- List of nodes
  - Name
  - IP
  - Latency to IPs
  - Errors, if any, attempting to reach IPs
PUT /status/probe
- Similar to GET /status - same result, but it triggers a probe and only returns when the probe completes.

Sample output

$ cilium-health status
Probe time:   2017-11-27T13:08:40-08:00
Nodes:
  cilium-master (localhost):
    Host connectivity to 10.0.2.15:
      ICMP:          OK, RTT=167.551µs
      HTTP via L3:   OK, RTT=780.628µs

Commandline options

$ ./cilium-health
Agent for hosting and querying the Cilium health status API

Usage:
  cilium-health [flags]
  cilium-health [command]

Available Commands:
  get         Display local cilium-health status
  ping        Check whether the cilium-health API is up
  status      Display cilium connectivity to other nodes

Flags:
      --admin string    Expose resources over 'unix' socket, 'any' socket (default "any")
  -c, --cilium string   URI to Cilium server API
  -d, --daemon          Run as a daemon
  -D, --debug           Enable debug messages
  -H, --host string     URI to cilum-health server API
  -i, --interval int    Interval (in seconds) for periodic connectivity probes (default 60)
  -p, --passive         Only respond to HTTP health checks

Use "cilium-health [command] --help" for more information about a command.

Next Tasks:

The scope of this PR is primarily to establish groundwork of determining connectivity to other nodes. Endpoint connectivity and policy connectivity is likely to be addressed in a followup PR.

For more future tasks, see Issue #1947.

houndci-bot · 2017-11-16T07:56:49Z

pkg/health/server/server.go

+	if err := pinger.Run(); err != nil {
+		log.WithError(err).Info("Failed to run ping")
+		return nil, err
+	} else {


if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary)

houndci-bot · 2017-11-16T07:56:49Z

pkg/health/server/server.go

+	}
+}
+
+// FetchStatusResponse() updates the cluster with the latest set of nodes,


comment on exported method Server.FetchStatusResponse should be of the form "FetchStatusResponse ..."

aanm

Looks good so far!

houndci-bot · 2017-11-17T03:01:12Z

pkg/health/client/client.go

+	"github.com/go-openapi/strfmt"
+)
+
+type Client struct {


exported type Client should have comment or be unexported

houndci-bot · 2017-11-17T03:01:12Z

pkg/client/client.go

@@ -101,3 +103,54 @@ func Hint(err error) error {
 	}
 	return fmt.Errorf("%s", e)
 }
+
+func FormatStatusResponse(w io.Writer, sr *models.StatusResponse) {


exported function FormatStatusResponse should have comment or be unexported

tgraf · 2017-11-29T22:15:56Z

api/v1/health/openapi.yaml

+            "$ref": "../openapi.yaml#/definitions/Error"
+  "/status/probe":
+    put:
+      summary: Get connectivity status of the Cilium cluster


The summary should already indicate that it is synchronous

Run synchronous connectivity probe to determine status of Cilium cluster

Ack, will fix.

tgraf · 2017-11-29T22:25:52Z

api/v1/health/openapi.yaml

+    properties:
+      cilium:
+        description: Status of Cilium daemon
+        "$ref": "../openapi.yaml#/definitions/StatusResponse"


So we have a StatusResponse in both yaml files? Can we rename one of them?

Sure, renamed the health status response.

ianvernon

Looks great overall - the size made it hard to fully grok some parts, but that is the nature of bootstrapping new components :)

Some small nits. You don't have to request another review from me.

ianvernon · 2017-11-30T00:02:19Z

pkg/health/server/prober.go

+	}
+}
+
+// Run sends a single probes out to all of the other cilium nodes to gather


... a single probe...

cilium --> Cilium

ianvernon · 2017-11-30T00:08:32Z

pkg/launcher/launcher.go

+	launcher.setStdout(stdout)
+}
+
+// Restart stops the daemon whilauncher will trigger a rerun.


fix typo "whilauncher"

Thanks for looking it over. I tried to split it up into reasonable separate patches for better review, but it is still quite large. I'll fix this up.

houndci-bot · 2017-11-30T01:36:43Z

api/v1/health/models/health_status_response.go

+// HealthStatusResponse Connectivity status to other daemons
+// swagger:model HealthStatusResponse
+
+type HealthStatusResponse struct {


exported type HealthStatusResponse should have comment or be unexported

This should live in pkg/node. Signed-off-by: Joe Stringer <joe@covalent.io>

This makes it reusable from cilium-health client. Signed-off-by: Joe Stringer <joe@covalent.io>

Export a list of known cluster nodes, including names, IPs and CIDR allocations in the health status requested on a local node. Signed-off-by: Joe Stringer <joe@covalent.io>

This is only used by some commands right now, so leave it hidden (since it's a bit inconsistent). When each command implements json-formatting the responses, we can make this not hidden. Signed-off-by: Joe Stringer <joe@covalent.io>

Previously, the client package created the client by passing the entire "unix://foo" or "http://foo" host string into the hostname field of the client, however when creating a client using an "http://" host string this would result in HTTP requests attempting to reach "http://http://hostname". Signed-off-by: Joe Stringer <joe@covalent.io>

goprocinfo provides a wrapper around /proc on linux systems, which will be used to collect system load in an upcoming commit. Signed-off-by: Joe Stringer <joe@covalent.io>

GET /status: Returns the connectivity status to all other cilium-health instances using interval-based probing. GET /status/probe: Runs a synchronous probe to all other cilium-health instances and returns the connectivity status. GET /healthz: Returns current health: * uptime * system load * local agent status See also Issue cilium#1947 Signed-off-by: Joe Stringer <joe@covalent.io>

Signed-off-by: Joe Stringer <joe@covalent.io>

cilium-health is a new client and daemon which allows users to probe the status of the local node, and also of the connectivity between the local node and remote nodes in the cluster. Signed-off-by: Joe Stringer <joe@covalent.io>

Also add the transitive dependencies on golang.org ICMP/IPv4/IPv6. Signed-off-by: Joe Stringer <joe@covalent.io>

Signed-off-by: Joe Stringer <joe@covalent.io>

Add support to the cilium-health client and server to host a new /hello API call over TCP socket 4240, and be able to make requests to it. Pinging this port on all known cluster nodes is done periodically, just like how regular ICMP ping polling is done. Signed-off-by: Joe Stringer <joe@covalent.io>

Signed-off-by: Joe Stringer <joe@covalent.io>

A generic launcher will be useful for an upcoming commit where the Cilium daemon also launches the cilium-health daemon. Signed-off-by: Joe Stringer <joe@covalent.io>

Signed-off-by: Joe Stringer <joe@covalent.io>

The new '--admin' option restricts which protocols that the /healthz and /status resources are served over. If specified as 'unix', it will only serve these resources over the unix socket (if not otherwise disabled by --passive); if specified as 'any' (default), these resources will also be served over the HTTP sockets. Signed-off-by: Joe Stringer <joe@covalent.io>

joestringer added kind/feature This introduces new functionality. wip labels Nov 16, 2017

joestringer requested review from aanm and ianvernon as code owners November 16, 2017 07:56

joestringer requested a review from a team November 16, 2017 07:56

joestringer requested review from a team as code owners November 16, 2017 07:56

houndci-bot reviewed Nov 16, 2017

View reviewed changes

joestringer added the release-note/major This PR introduces major new functionality to Cilium. label Nov 16, 2017

joestringer changed the title ~~RFC: Add cilium-health client/daemon~~ RFC: Introduce "cilium-health", a new tool for investigating cluster connectivity issues. Nov 16, 2017

aanm reviewed Nov 16, 2017

View reviewed changes

joestringer force-pushed the submit/cilium-health branch from 7194ec9 to cb3b955 Compare November 17, 2017 03:00

joestringer requested a review from a team as a code owner November 17, 2017 03:00

houndci-bot reviewed Nov 17, 2017

View reviewed changes

cilium deleted a comment from houndci-bot Nov 17, 2017

joestringer force-pushed the submit/cilium-health branch 4 times, most recently from cfd9d94 to 8858645 Compare November 29, 2017 22:18

tgraf requested changes Nov 29, 2017

View reviewed changes

ianvernon approved these changes Nov 30, 2017

View reviewed changes

joestringer force-pushed the submit/cilium-health branch 2 times, most recently from 302d36b to a7b512c Compare November 30, 2017 01:36

houndci-bot reviewed Nov 30, 2017

View reviewed changes

joestringer added 16 commits November 29, 2017 17:46

daemon: Refactor getting local node addressing

2212f04

This should live in pkg/node. Signed-off-by: Joe Stringer <joe@covalent.io>

client: Refactor formatting Cilium StatusResponse

67560cd

This makes it reusable from cilium-health client. Signed-off-by: Joe Stringer <joe@covalent.io>

api: Export cluster nodes in /status

402d5ec

Export a list of known cluster nodes, including names, IPs and CIDR allocations in the health status requested on a local node. Signed-off-by: Joe Stringer <joe@covalent.io>

cmd: Add "--json" option.

b8edf10

This is only used by some commands right now, so leave it hidden (since it's a bit inconsistent). When each command implements json-formatting the responses, we can make this not hidden. Signed-off-by: Joe Stringer <joe@covalent.io>

vendor: Add github.com/c9s/goprocinfo/linux

fe17e1a

goprocinfo provides a wrapper around /proc on linux systems, which will be used to collect system load in an upcoming commit. Signed-off-by: Joe Stringer <joe@covalent.io>

api: Add swagger-generated health API

284ca68

Signed-off-by: Joe Stringer <joe@covalent.io>

cilium-health: Add new health status agent

4fb43db

cilium-health is a new client and daemon which allows users to probe the status of the local node, and also of the connectivity between the local node and remote nodes in the cluster. Signed-off-by: Joe Stringer <joe@covalent.io>

vendor: Add github.com/servak/go-fastping

9561eab

Also add the transitive dependencies on golang.org ICMP/IPv4/IPv6. Signed-off-by: Joe Stringer <joe@covalent.io>

cilium-health: Report ping status to other nodes

1fc16a0

Signed-off-by: Joe Stringer <joe@covalent.io>

cilium-health: Add --passive mode for HTTP server

cb692b9

Signed-off-by: Joe Stringer <joe@covalent.io>

pkg: Refactor monitor launcher common code

1022893

A generic launcher will be useful for an upcoming commit where the Cilium daemon also launches the cilium-health daemon. Signed-off-by: Joe Stringer <joe@covalent.io>

daemon: Launch cilium-health on startup

9bbad70

Signed-off-by: Joe Stringer <joe@covalent.io>

joestringer force-pushed the submit/cilium-health branch from a7b512c to 92bb2ba Compare November 30, 2017 01:47

tgraf approved these changes Nov 30, 2017

View reviewed changes

tgraf merged commit 62b9d61 into cilium:master Nov 30, 2017

joestringer mentioned this pull request Nov 30, 2017

Proposal: Network fabric connectivity debugging tool #1947

Closed

23 tasks

joestringer deleted the submit/cilium-health branch December 1, 2017 18:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce "cilium-health", a new tool for investigating cluster connectivity issues. #2052

Introduce "cilium-health", a new tool for investigating cluster connectivity issues. #2052

joestringer commented Nov 16, 2017 •

edited

houndci-bot Nov 16, 2017

houndci-bot Nov 16, 2017

aanm left a comment

houndci-bot Nov 17, 2017

houndci-bot Nov 17, 2017

tgraf Nov 29, 2017

joestringer Nov 30, 2017

tgraf Nov 29, 2017

joestringer Nov 30, 2017 •

edited

ianvernon left a comment

ianvernon Nov 30, 2017

ianvernon Nov 30, 2017

joestringer Nov 30, 2017

houndci-bot Nov 30, 2017

Introduce "cilium-health", a new tool for investigating cluster connectivity issues. #2052

Introduce "cilium-health", a new tool for investigating cluster connectivity issues. #2052

Conversation

joestringer commented Nov 16, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aanm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joestringer Nov 30, 2017 • edited

Choose a reason for hiding this comment

ianvernon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joestringer commented Nov 16, 2017 •

edited

joestringer Nov 30, 2017 •

edited