Skip to content

Commit

Permalink
Add hc parent health
Browse files Browse the repository at this point in the history
  • Loading branch information
rob05c committed Oct 24, 2022
1 parent 8362b86 commit e0b7d46
Show file tree
Hide file tree
Showing 24 changed files with 4,002 additions and 739 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
- [#7023](https://github.com/apache/trafficcontrol/pull/7023) *Traffic Ops* Added the `ASN` field in TO Server struct, which provides the ability to query servers by `ASN`.
- [#2101](https://github.com/apache/trafficcontrol/issues/2101) *Traffic Portal* Added the ability to tell if a Delivery Service is the target of another steering DS.
- [#6033](https://github.com/apache/trafficcontrol/issues/6033) *Traffic Ops, Traffic Portal* Added ability to assign multiple server capabilities to a server.
- [#7096](https://github.com/apache/trafficcontrol/issues/7096) [Health Client] Added health client parent health
- [#7032](https://github.com/apache/trafficcontrol/issues/7032) *Cache Config* Add t3c-apply flag to use local ATS version for config generation rather than Server package Parameter, to allow managing the ATS OS package via external tools. See 'man t3c-apply' and 'man t3c-generate' for details.

### Changed
Expand Down
173 changes: 173 additions & 0 deletions lib/go-llog/llog.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
// Package llog provides logging utilities for library packages.
//
// This allows libraries to log if desired, while still allowing
// the library functions to have no side effects and not use
// global loggers, which may be different than the application's
// primary log library.
//
// This also allows users of the library to decide their logging
// level for this particular library. For example, an application
// may wish to generally log at the debug level, but not log
// debug messages for some particular library.
//
// Or, a user may wish to log warning messages from a library
// as errors. Setting two log levels to the same io.Writer
// is permissible.
//
// This is not itself a logging library. Rather, it allows
// applications using any log library to interface with libraries
// using llog, by constructing a Loggers object from their own
// logger.
package llog

/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

import (
"fmt"
"io"
)

// Log is an interface which library functions may accept
// in order to log without side effects, if the caller desires.
//
// Applications using a library which uses Log may use
// NewLog, or may themselves implement the interface.
//
// Library functions should immediately call DefaultIfNil to allow callers to
// pass a nil Log. Importantly, this allows applications using the library
// to avoid importing llog, if they don't want the library to log.
type Log interface {
Errorf(format string, v ...interface{})
Errorln(v ...interface{})
Warnf(format string, v ...interface{})
Warnln(v ...interface{})
Infof(format string, v ...interface{})
Infoln(v ...interface{})
Debugf(format string, v ...interface{})
Debugln(v ...interface{})
}

// LibInit initializes the Log for libraries.
//
// All public functions in Libraries using llog should immediately call LibInit.
// The return value should be assigned, like `log = llog.LibInit(log)`
//
// Applications creating a Log to pass to a library func should never call LibInit.
//
// This creates a Nop Log if the passed lg is nil, which allows applications
// to pass a nil Log.
//
// It may do other things in the future.
func LibInit(lg Log) Log {
if lg == nil {
return Nop()
}
return lg
}

// Nop returns a Log that never logs.
// Applications which don't want a library to log can pass Nop
// to libraries that take a Log.
func Nop() Log { return &loggers{} }

// New creates a new Log.
//
// Applications can use New to create a Log from their own internal log
// libraries and writers, to pass to libraries using liblog.
//
// Standard log example:
//
// errLog := log.New(os.Stdout, "ERROR: ", log.Ldate|log.Ltime|log.Lshortfile)
// mylib.MyFunc(liblog.New(errLog, nil, nil, nil), myArg)
//
// github.com/apache/trafficcontrol/lib/go-log example:
//
// import("github.com/apache/trafficcontrol/lib/go-log")
//
// log.Init(nil, os.Stderr, os.Stderr, nil, nil)
//
// lLog := log.LLog() // lib/go-tc has a built-in llog helper
//
// // alternatively, what the lib/go-log helper is doing internally:
// ltow := func(lg *log.Logger) io.Writer {
// return llog.WriterFunc(func(p []byte) (n int, err error) {
// Logln(lg, string(p))
// })
// lLog = llog.New(ltow(Error), ltow(Warning), ltow(Info), ltow(Debug))
//
// mylib.MyFunc(lLog, myArg)
//
// zap example:
//
// import("go.uber.org/zap")
//
// func main() {
// logger, _ := zap.NewProduction()
// sugar := logger.Sugar()
// zapErrLog := liblog.WriterFunc(func(p []byte) (n int, err error){
// logger.Sugar().Error(string(p))
// })
// mylib.MyFunc(liblog.New(zapErrLog, nil, nil, nil), myArg)
// }
func New(err io.Writer, warn io.Writer, info io.Writer, debug io.Writer) Log {
return &loggers{
err: err,
warn: warn,
inf: info,
dbg: debug,
}
}

// WriterFunc is an adapter to allow the use of ordinary functions as io.Writers.
// This behaves similar to http.HandlerFunc for http.Handler.
type WriterFunc func(p []byte) (n int, err error)

// Write implements io.Writer.
func (wf WriterFunc) Write(p []byte) (n int, err error) { return wf(p) }

type loggers struct {
err io.Writer
warn io.Writer
inf io.Writer
dbg io.Writer
}

func (ls *loggers) Errorf(format string, v ...interface{}) { logf(ls.err, format, v...) }
func (ls *loggers) Errorln(v ...interface{}) { logln(ls.err, v...) }
func (ls *loggers) Warnf(format string, v ...interface{}) { logf(ls.warn, format, v...) }
func (ls *loggers) Warnln(v ...interface{}) { logln(ls.warn, v...) }
func (ls *loggers) Infof(format string, v ...interface{}) { logf(ls.inf, format, v...) }
func (ls *loggers) Infoln(v ...interface{}) { logln(ls.inf, v...) }
func (ls *loggers) Debugf(format string, v ...interface{}) { logf(ls.dbg, format, v...) }
func (ls *loggers) Debugln(v ...interface{}) { logln(ls.dbg, v...) }

func logf(wr io.Writer, format string, v ...interface{}) {
if wr == nil {
return
}
fmt.Fprintf(wr, format, v...)
}

func logln(wr io.Writer, v ...interface{}) {
if wr == nil {
return
}
fmt.Fprintln(wr, v...)
}
22 changes: 22 additions & 0 deletions lib/go-log/log.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ import (
"log"
"os"
"time"

"github.com/apache/trafficcontrol/lib/go-llog"
)

var (
Expand Down Expand Up @@ -278,3 +280,23 @@ func InitCfg(cfg Config) error {
Init(eventW, errW, warnW, infoW, debugW)
return nil
}

// LLog returns an llog.Log, for passing to libraries using llog.
//
// Note the returned Log will have writers tied to loggers at its time of creation.
// Thus, it's safe to reuse LLog if an application never re-initializes the loggers,
// such as when reloading config on a HUP signal.
// If the application re-initializes loggers, LLog should be called again to get
// a new llog.Log associated with the new log locations.
func LLog() llog.Log {
// ltow converts a log.Logger into an io.Writer
// This is relatively inefficient. If performance is necessary, this package could be made to
// keep track of the original io.Writer, to avoid an extra function call and string copy.
ltow := func(lg *log.Logger) io.Writer {
return llog.WriterFunc(func(p []byte) (n int, err error) {
Logln(lg, string(p))
return len(p), nil
})
}
return llog.New(ltow(Error), ltow(Warning), ltow(Info), ltow(Debug))
}
96 changes: 76 additions & 20 deletions tc-health-client/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,26 +69,28 @@ Requires Apache TrafficServer 8.1.0 or later.

# OPTIONS

-f, -\-config-file=config-file

Specify the config file to use.
Defaults to /etc/trafficcontro-health-client/tc-health-client.json
-f, -\-config-file=config-file

-h, -\-help
Specify the config file to use.
Defaults to /etc/trafficcontro-health-client/tc-health-client.json

Prints command line usage and exits
-h, -\-help

Prints command line usage and exits

-l, -\-logging-dir=logging-directory

Specify the directory where log files are kept. The default location
is **/var/log/trafficcontrol/**
Specify the directory where log files are kept. The default location
is **/var/log/trafficcontrol/**

-l, -\-logging-dir=logging-directory

-v, -\-verbose

Logging verbosity. Errors are logged to the default log file
**/var/log/trafficcontrol/tc-health-client.log**
To add Warnings, use -v. To add Warnings and Informational
logging, use -vv. Finally you may add Debug logging using -vvv.
Logging verbosity. Errors are logged to the default log file
**/var/log/trafficcontrol/tc-health-client.log**
To add Warnings, use -v. To add Warnings and Informational
logging, use -vv. Finally you may add Debug logging using -vvv.

# CONFIGURATION

Expand All @@ -103,21 +105,28 @@ Sample configuarion file:
"enable-active-markdowns": false,
"reason-code": "active",
"to-credential-file": "/etc/credentials",
"to-url": "https://tp.cdn.com:443",
"to-url": "https://tp.cdn.com:443",
"to-request-timeout-seconds": "5s",
"tm-poll-interval-seconds": "60s",
"tm-proxy-url", "http://sample-http-proxy.cdn.net:80",
"tm-proxy-url": "http://sample-http-proxy.cdn.net:80",
"to-login-dispersion-factor": 90,
"unavailable-poll-threshold": 2,
"markup-poll-threshold": 1,
"trafficserver-config-dir": "/opt/trafficserver/etc/trafficserver",
"trafficserver-bin-dir": "/opt/trafficserver/bin",
"poll-state-json-log": "/var/log/trafficcontrol/poll-state.json",
"enable-poll-state-log": false
"enable-poll-state-log": false,
"parent-health-poll-ms": 10000,
"serve-parent-health": true,
"parent-health-service-port": 31337,
"parent-health-log-location": "/var/log/trafficcontrol/tc-health-client_parent-health.log",
"health-methods": ["traffic-monitor", "parent-l4", "parent-l7", "parent-service"],
"markdown-methods": ["traffic-monitor", "parent-l4", "parent-l7", "parent-service"],
"hostname": ""
}
```

### cdn-name
### cdn-name

The name of the CDN that the Traffic Server host is a member of.

Expand All @@ -135,7 +144,7 @@ hosts in the Traffic Server **HostStatus** subsystem.

### to-credential-file

The file where **Traffic Ops** credentials are read. The file should define the
The file where **Traffic Ops** credentials are read. The file should define the
following variables:

* TO_URL="https://trafficops.cdn.com"
Expand Down Expand Up @@ -185,7 +194,7 @@ with the parent reported as healthy. The default threshold is 1.

### trafficserver-config-dir

The location on the host where **Traffic Server** configuration files are
The location on the host where **Traffic Server** configuration files are
located.

### trafficserver-bin-dir
Expand All @@ -195,7 +204,7 @@ be found.

### poll-state-json-log ###

The full path to the polling state file which contains information
The full path to the polling state file which contains information
about the current status of parents and the health client configuration.
Polling state data is written to this file after each polling cycle when
enabled, see **enable-poll-state-log**
Expand All @@ -205,6 +214,54 @@ enabled, see **enable-poll-state-log**
Enable writing the Polling state to the **poll-state-json-log** after
eache polling cycle. Default **false**, disabled

### markdown-min-interval-ms ###

Minimum interval between markdown processing in milliseconds. When health polls finish, they automatically signal the markdown service to mark down accordingly. This is the minimum time to wait between processing, to avoid too much processing. To always process markdowns as soon as every health poll finishes, set to 0. Default is 5 seconds.

### parent-health-l4-poll-ms ###

Interval to poll for parent health via L4 in milliseconds.

### parent-health-l7-poll-ms ###

Interval to poll for parent health via L7 in milliseconds.

### parent-health-service-poll-ms ###

Interval to poll for parent health from parents' health service in milliseconds.

### parent-health-service-port ###

The port to serve the JSON parent health data over HTTP. To disable serving, set to < 1. Default is 0, disabled.

### parent-health-log-location ###

The location to log parent health changes. May be stdout, stderr, null, or a file path.

### health-methods ###

The health methods to poll. Options are 'traffic-monitor', 'parent-l4', 'parent-l7', and 'parent-service'.

Traffic Monitor requests Traffic Monitor and uses its boolean CRStates API for health.

Parent L4 polls all parents via a HTTP request. Any valid response, including HTTP error codes, is considered a success. Only a failed HTTP request is considered unhealthy.

Parent L7 polls all parents via a TCP SYN. Any valid TCP ACK response within the timeout is considered healthy. Failure to receive an ACK before the timeout is considered unhealthy. Note this also sends a TCP Reset to aid the host in quickly releasing resources.

Parent Service polls the tc-health-client parent service, on the same port as this host's parent service, and uses a heuristic of the parent's own available parents to determine health. The heuristic is currently 50%, but that may change or be made configurable in the future. That is, if more than 50% of a parent cache's own parents are unavailable, the parent is unhealthy.

### markdown-methods ###

Markdown methods are the health methods to consider when marking down parents. See health-methods. Hence, a method may be polled via health-methods and logged and served for informational purposes, without using that method to mark down parents.

### num-health-workers ###

The number of worker microthreads (goroutines) per health poll method.

Note this only applies to Parent L4, Parent L7, and Parent Service health; the Traffic Monitor health poll is a single HTTP request and thus doesn't need workers.

### hostname ###

# Files

* /etc/trafficcontrol/tc-health-client.json
Expand All @@ -215,4 +272,3 @@ eache polling cycle. Default **false**, disabled
* Traffic Server **parent.config**
* Traffic Server **strategies.yaml**
* Traffic Server **traffic_ctl** command

0 comments on commit e0b7d46

Please sign in to comment.