Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add no-file and keep-file log option for docker driver. #1906

Merged
merged 2 commits into from
Apr 13, 2020
Merged

Add no-file and keep-file log option for docker driver. #1906

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
533a0d1
Add no-file and keep-file option.
cyriltovena Apr 7, 2020
ff47e83
Review feedback !
cyriltovena Apr 8, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 26 additions & 23 deletions cmd/docker-driver/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ docker run --log-driver=loki \

> **Note**: The Loki logging driver still uses the json-log driver in combination with sending logs to Loki, this is mainly useful to keep the `docker logs` command working.
> You can adjust file size and rotation using the respective log option `max-size` and `max-file`.
> You can deactivate this behavior by setting the log option `no-file` to true.

### Configure the default logging driver

Expand Down Expand Up @@ -115,29 +116,31 @@ You can add more labels by using `loki-external-labels`,`loki-pipeline-stage-fil

To specify additional logging driver options, you can use the --log-opt NAME=VALUE flag.

| Option | Required? | Default Value | Description
| ------------------------- | :-------: | :------------------: | -------------------------------------- |
| `loki-url` | Yes | | Loki HTTP push endpoint.
| `loki-external-labels` | No | `container_name={{.Name}}` | Additional label value pair separated by `,` to send with logs. The value is expanded with the [Docker tag template format](https://docs.docker.com/config/containers/logging/log_tags/). (eg: `container_name={{.ID}}.{{.Name}},cluster=prod`)
| `loki-timeout` | No | `10s` | The timeout to use when sending logs to the Loki instance. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
| `loki-batch-wait` | No | `1s` | The amount of time to wait before sending a log batch complete or not. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
| `loki-batch-size` | No | `102400` | The maximum size of a log batch to send.
| `loki-min-backoff` | No | `100ms` | The minimum amount of time to wait before retrying a batch. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
| `loki-max-backoff` | No | `10s` | The maximum amount of time to wait before retrying a batch. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
| `loki-retries` | No | `10` | The maximum amount of retries for a log batch.
| `loki-pipeline-stage-file` | No | | The location of a pipeline stage configuration file ([example](./pipeline-example.yaml)). Pipeline stages allows to parse log lines to extract more labels. [see documentation](../../docs/logentry/processing-log-lines.md)
| `loki-tenant-id` | No | | Set the tenant id (http header`X-Scope-OrgID`) when sending logs to Loki. It can be overrides by a pipeline stage.
| `loki-tls-ca-file` | No | | Set the path to a custom certificate authority.
| `loki-tls-cert-file` | No | | Set the path to a client certificate file.
| `loki-tls-key-file` | No | | Set the path to a client key.
| `loki-tls-server-name` | No | | Name used to validate the server certificate.
| `loki-tls-insecure-skip-verify` | No | `false` | Allow to skip tls verification.
| `loki-proxy-url` | No | | Proxy URL use to connect to Loki.
| `max-size` | No | -1 | The maximum size of the log before it is rolled. A positive integer plus a modifier representing the unit of measure (k, m, or g). Defaults to -1 (unlimited). This is used by json-log required to keep the `docker log` command working.
| `max-file` | No | 1 | The maximum number of log files that can be present. If rolling the logs creates excess files, the oldest file is removed. Only effective when max-size is also set. A positive integer. Defaults to 1.
| `labels` | No | | Comma-separated list of keys of labels, which should be included in message, if these labels are specified for container.
| `env` | No | | Comma-separated list of keys of environment variables to be included in message if they specified for a container.
| `env-regex` | No | | A regular expression to match logging-related environment variables. Used for advanced log label options. If there is collision between the label and env keys, the value of the env takes precedence. Both options add additional fields to the labels of a logging message.
| Option | Required? | Default Value | Description |
|---------------------------------|:---------:|:--------------------------:|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `loki-url` | Yes | | Loki HTTP push endpoint. |
| `loki-external-labels` | No | `container_name={{.Name}}` | Additional label value pair separated by `,` to send with logs. The value is expanded with the [Docker tag template format](https://docs.docker.com/config/containers/logging/log_tags/). (eg: `container_name={{.ID}}.{{.Name}},cluster=prod`) |
| `loki-timeout` | No | `10s` | The timeout to use when sending logs to the Loki instance. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". |
| `loki-batch-wait` | No | `1s` | The amount of time to wait before sending a log batch complete or not. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". |
| `loki-batch-size` | No | `102400` | The maximum size of a log batch to send. |
| `loki-min-backoff` | No | `100ms` | The minimum amount of time to wait before retrying a batch. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". |
| `loki-max-backoff` | No | `10s` | The maximum amount of time to wait before retrying a batch. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". |
| `loki-retries` | No | `10` | The maximum amount of retries for a log batch. |
| `loki-pipeline-stage-file` | No | | The location of a pipeline stage configuration file ([example](./pipeline-example.yaml)). Pipeline stages allows to parse log lines to extract more labels. [see documentation](../../docs/logentry/processing-log-lines.md) |
| `loki-tenant-id` | No | | Set the tenant id (http header`X-Scope-OrgID`) when sending logs to Loki. It can be overrides by a pipeline stage. |
| `loki-tls-ca-file` | No | | Set the path to a custom certificate authority. |
| `loki-tls-cert-file` | No | | Set the path to a client certificate file. |
| `loki-tls-key-file` | No | | Set the path to a client key. |
| `loki-tls-server-name` | No | | Name used to validate the server certificate. |
| `loki-tls-insecure-skip-verify` | No | `false` | Allow to skip tls verification. |
| `loki-proxy-url` | No | | Proxy URL use to connect to Loki. |
| `no-file` | No | `false` | This indicates the driver to not create log files on disk, however this means you won't be able to use `docker logs` on the container anymore. You can use this if you don't need to use `docker logs` and you run with limited disk space. (By default files are created) |
| `keep-file` | No | `false` | This indicates the driver to keep json log files once the container is stopped. By default files are removed, this means you won't be able to use `docker logs` once the container is stopped. |
| `max-size` | No | -1 | The maximum size of the log before it is rolled. A positive integer plus a modifier representing the unit of measure (k, m, or g). Defaults to -1 (unlimited). This is used by json-log required to keep the `docker log` command working. |
| `max-file` | No | 1 | The maximum number of log files that can be present. If rolling the logs creates excess files, the oldest file is removed. Only effective when max-size is also set. A positive integer. Defaults to 1. |
| `labels` | No | | Comma-separated list of keys of labels, which should be included in message, if these labels are specified for container. |
| `env` | No | | Comma-separated list of keys of environment variables to be included in message if they specified for a container. |
| `env-regex` | No | | A regular expression to match logging-related environment variables. Used for advanced log label options. If there is collision between the label and env keys, the value of the env takes precedence. Both options add additional fields to the labels of a logging message. |

## Uninstall the plugin

Expand Down
16 changes: 16 additions & 0 deletions cmd/docker-driver/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,8 @@ const (
cfgMaxRetriesKey = "loki-retries"
cfgPipelineStagesKey = "loki-pipeline-stage-file"
cfgTenantIDKey = "loki-tenant-id"
cfgNofile = "no-file"
cfgKeepFile = "keep-file"

swarmServiceLabelKey = "com.docker.swarm.service.name"
swarmStackLabelKey = "com.docker.stack.namespace"
Expand Down Expand Up @@ -101,6 +103,8 @@ func validateDriverOpt(loggerInfo logger.Info) error {
case cfgMaxRetriesKey:
case cfgPipelineStagesKey:
case cfgTenantIDKey:
case cfgNofile:
case cfgKeepFile:
case "labels":
case "env":
case "env-regex":
Expand Down Expand Up @@ -321,3 +325,15 @@ func parseInt(key string, logCtx logger.Info, set func(i int)) error {
}
return nil
}

func parseBoolean(key string, logCtx logger.Info, defaultValue bool) (bool, error) {
value, ok := logCtx.Config[key]
if !ok || value == "" {
return defaultValue, nil
}
b, err := strconv.ParseBool(value)
if err != nil {
return false, err
}
return b, nil
}
66 changes: 49 additions & 17 deletions cmd/docker-driver/driver.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,10 @@ type logPair struct {
stream io.ReadCloser
info logger.Info
logger log.Logger
// folder where json log files will be created.
folder string
// keep created files after stopping the container.
keepFile bool
}

func (l *logPair) Close() {
Expand All @@ -45,6 +49,9 @@ func (l *logPair) Close() {
if err := l.lokil.Close(); err != nil {
level.Error(l.logger).Log("msg", "error while closing loki logger", "err", err)
}
if l.jsonl == nil {
return
}
if err := l.jsonl.Close(); err != nil {
level.Error(l.logger).Log("msg", "error while closing json logger", "err", err)
}
Expand All @@ -65,16 +72,29 @@ func (d *driver) StartLogging(file string, logCtx logger.Info) error {
return fmt.Errorf("logger for %q already exists", file)
}
d.mu.Unlock()
folder := fmt.Sprintf("/var/log/docker/%s/", logCtx.ContainerID)
logCtx.LogPath = filepath.Join(folder, "json.log")

if logCtx.LogPath == "" {
logCtx.LogPath = filepath.Join("/var/log/docker", logCtx.ContainerID)
}
if err := os.MkdirAll(filepath.Dir(logCtx.LogPath), 0755); err != nil {
return errors.Wrap(err, "error setting up logger dir")
noFile, err := parseBoolean(cfgNofile, logCtx, false)
if err != nil {
return err
}
jsonl, err := jsonfilelog.New(logCtx)

keepFile, err := parseBoolean(cfgKeepFile, logCtx, false)
if err != nil {
return errors.Wrap(err, "error creating jsonfile logger")
return err
}

var jsonl logger.Logger
if !noFile {
if err := os.MkdirAll(folder, 0755); err != nil {
return errors.Wrap(err, "error setting up logger dir")
}

jsonl, err = jsonfilelog.New(logCtx)
if err != nil {
return errors.Wrap(err, "error creating jsonfile logger")
}
}

lokil, err := New(logCtx, d.logger)
Expand All @@ -88,7 +108,7 @@ func (d *driver) StartLogging(file string, logCtx logger.Info) error {
}

d.mu.Lock()
lf := &logPair{jsonl, lokil, f, logCtx, d.logger}
lf := &logPair{jsonl, lokil, f, logCtx, d.logger, folder, keepFile}
d.logs[file] = lf
d.idx[logCtx.ContainerID] = lf
d.mu.Unlock()
Expand All @@ -100,12 +120,19 @@ func (d *driver) StartLogging(file string, logCtx logger.Info) error {
func (d *driver) StopLogging(file string) {
level.Debug(d.logger).Log("msg", "Stop logging", "file", file)
d.mu.Lock()
defer d.mu.Unlock()
lf, ok := d.logs[file]
if ok {
lf.Close()
delete(d.logs, file)
if !ok {
return
cyriltovena marked this conversation as resolved.
Show resolved Hide resolved
}
lf.Close()
delete(d.logs, file)
if !lf.keepFile && lf.jsonl != nil {
// delete the folder where all log files were created.
if err := os.RemoveAll(lf.folder); err != nil {
level.Debug(d.logger).Log("msg", "error deleting folder", "folder", lf.folder)
}
}
d.mu.Unlock()
}

func consumeLog(lf *logPair) {
Expand Down Expand Up @@ -138,10 +165,11 @@ func consumeLog(lf *logPair) {
if err := lf.lokil.Log(&msg); err != nil {
level.Error(lf.logger).Log("msg", "error pushing message to loki", "id", lf.info.ContainerID, "err", err, "message", msg)
}

if err := lf.jsonl.Log(&msg); err != nil {
level.Error(lf.logger).Log("msg", "error writing log message", "id", lf.info.ContainerID, "err", err, "message", msg)
continue
if lf.jsonl != nil {
if err := lf.jsonl.Log(&msg); err != nil {
level.Error(lf.logger).Log("msg", "error writing log message", "id", lf.info.ContainerID, "err", err, "message", msg)
continue
}
}

buf.Reset()
Expand All @@ -156,10 +184,14 @@ func (d *driver) ReadLogs(info logger.Info, config logger.ReadConfig) (io.ReadCl
return nil, fmt.Errorf("logger does not exist for %s", info.ContainerID)
}

if lf.jsonl == nil {
return nil, fmt.Errorf("%s option set to true, no reading capability", cfgNofile)
}

r, w := io.Pipe()
lr, ok := lf.jsonl.(logger.LogReader)
if !ok {
return nil, fmt.Errorf("logger does not support reading")
return nil, errors.New("logger does not support reading")
}

go func() {
Expand Down