Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker metrics (read metrics from cgroups for specified container) #8886

Closed
wants to merge 1 commit into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
40 changes: 40 additions & 0 deletions api/client/commands.go
Original file line number Diff line number Diff line change
Expand Up @@ -812,6 +812,46 @@ func (cli *DockerCli) CmdInspect(args ...string) error {
return nil
}

func (cli *DockerCli) CmdMetrics(args ...string) error {
cmd := cli.Subcmd("metrics", "CONTAINER", "Return runtime information on a container")
if err := cmd.Parse(args); err != nil {
return nil
}
if cmd.NArg() < 1 {
cmd.Usage()
return nil
}

indented := new(bytes.Buffer)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use json.Unmarshal and a struct or map for this, please.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's copy-paste from CmdInspect. I've marked both functions with TODO (will
extract common things and make their code less ugly).

Thanks for the tip, just a first week with Go :)
On Fri 31 Oct 2014 at 10:22 pm Erik Hollensbe notifications@github.com
wrote:

In api/client/commands.go:

@@ -812,6 +812,46 @@ func (cli *DockerCli) CmdInspect(args ...string) error {
return nil
}

+func (cli *DockerCli) CmdMetrics(args ...string) error {

  • cmd := cli.Subcmd("metrics", "CONTAINER", "Return runtime information on a container")
  • if err := cmd.Parse(args); err != nil {
  •   return nil
    
  • }
  • if cmd.NArg() < 1 {
  •   cmd.Usage()
    
  •   return nil
    
  • }
  • indented := new(bytes.Buffer)

use json.Unmarshal and a struct or map for this, please.


Reply to this email directly or view it on GitHub
https://github.com/docker/docker/pull/8886/files#r19688050.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool. thanks. just let us know when it's ready for review again.

indented.WriteByte('[')
status := 0

for _, name := range cmd.Args() {
obj, _, err := readBody(cli.call("GET", "/containers/"+name+"/metrics", nil, false))
if err = json.Indent(indented, obj, "", " "); err != nil {
fmt.Fprintf(cli.err, "%s\n", err)
status = 1
continue
}
indented.WriteString(",")
}

if indented.Len() > 1 {
// Remove trailing ','
indented.Truncate(indented.Len() - 1)
}
indented.WriteByte(']')

if _, err := io.Copy(cli.out, indented); err != nil {
return err
}

if status != 0 {
return &utils.StatusError{StatusCode: status}
}
return nil
}

func (cli *DockerCli) CmdTop(args ...string) error {
cmd := cli.Subcmd("top", "CONTAINER [ps OPTIONS]", "Display the running processes of a container")
if err := cmd.Parse(args); err != nil {
Expand Down
14 changes: 14 additions & 0 deletions api/server/server.go
Original file line number Diff line number Diff line change
Expand Up @@ -343,6 +343,19 @@ func getContainersTop(eng *engine.Engine, version version.Version, w http.Respon
return job.Run()
}

func getContainersMetrics(eng *engine.Engine, version version.Version, w http.ResponseWriter, r *http.Request, vars map[string]string) error {
if err := parseForm(r); err != nil {
return err
}
if vars == nil {
return fmt.Errorf("Missing parameter")
}

job := eng.Job("container_metrics", vars["name"])
streamJSON(job, w, false)
return job.Run()
}

func getContainersJSON(eng *engine.Engine, version version.Version, w http.ResponseWriter, r *http.Request, vars map[string]string) error {
if err := parseForm(r); err != nil {
return err
Expand Down Expand Up @@ -1262,6 +1275,7 @@ func createRouter(eng *engine.Engine, logging, enableCors bool, dockerVersion st
"/containers/{name:.*}/top": getContainersTop,
"/containers/{name:.*}/logs": getContainersLogs,
"/containers/{name:.*}/attach/ws": wsContainersAttach,
"/containers/{name:.*}/metrics": getContainersMetrics,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I recall correctly, the agreement was to have a push/watch based interface for metrics in the docker daemon. Is that not a requirement anymore?
cc @crosbymichael @vmarmol

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are correct. We want a push approach where a client will subscribe to an docker endpoint and metrics will be pushed to the consumer with a requested internal

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I understand we need to implement websocket endpoint. I think I can make
it in my branch.

Also we need to add customizable interval option with reasonable default
value (1s for example).

Do we need to have template formatter to receive partial data?

Maybe we also need to have special predefined formatters to output metrics
(pretty output for humans by default and, machine readable JSON for
scripts)?

Are there any public spec for this feature?
On Sat 1 Nov 2014 at 1:05 am Michael Crosby notifications@github.com
wrote:

In api/server/server.go:

@@ -1262,6 +1275,7 @@ func createRouter(eng engine.Engine, logging, enableCors bool, dockerVersion st
"/containers/{name:.
}/top": getContainersTop,
"/containers/{name:.}/logs": getContainersLogs,
"/containers/{name:.
}/attach/ws": wsContainersAttach,

  •       "/containers/{name:.*}/metrics":   getContainersMetrics,
    

You are correct. We want a push approach where a client will subscribe to
an docker endpoint and metrics will be pushed to the consumer with a
requested internal


Reply to this email directly or view it on GitHub
https://github.com/docker/docker/pull/8886/files#r19696346.

},
"POST": {
"/auth": postAuth,
Expand Down
1 change: 1 addition & 0 deletions daemon/daemon.go
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,7 @@ func (daemon *Daemon) Install(eng *engine.Engine) error {
"container_changes": daemon.ContainerChanges,
"container_copy": daemon.ContainerCopy,
"container_inspect": daemon.ContainerInspect,
"container_metrics": daemon.ContainerMetrics,
"containers": daemon.Containers,
"create": daemon.ContainerCreate,
"rm": daemon.ContainerRm,
Expand Down
97 changes: 97 additions & 0 deletions daemon/metrics.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
package daemon

import (
"fmt"
"os"
"strings"

"io/ioutil"
"path/filepath"

"github.com/docker/docker/engine"
"github.com/docker/libcontainer/cgroups"
)

var metricInfoTable map[string][]string

func init() {
metricInfoTable = map[string][]string{
"CpuUsage": {"cpuacct", "usage"},
"MemoryUsage": {"memory", "usage_in_bytes"},
"MemoryMaxUsage": {"memory", "max_usage_in_bytes"},
"MemoryLimit": {"memory", "limit_in_bytes"},
}
}

func (daemon *Daemon) ContainerMetrics(job *engine.Job) engine.Status {
if len(job.Args) != 1 {
return job.Errorf("usage: %s NAME", job.Name)
}
name := job.Args[0]
if container := daemon.Get(name); container != nil {
container.Lock()
defer container.Unlock()

out := &engine.Env{}
out.Set("Id", container.ID)
out.Set("Image", container.Image)
out.Set("Name", container.Name)
out.SetJson("State", container.State)

metrics := map[string]string{}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use the stats that libcontainer already exports?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

libcontainer's stats is limited only to native driver. Fetching metrics directly from cgroups is more portable. I think I need to move this code to drivers.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is quite a bit of code in libcontainer to handle stats. I don't see a
point in replicating it. Wouldn't 'lxc-info' work for the lxc driver?

On Mon, Nov 3, 2014 at 2:03 PM, Gleb M Borisov notifications@github.com
wrote:

In daemon/metrics.go:

+func (daemon *Daemon) ContainerMetrics(job *engine.Job) engine.Status {

  • if len(job.Args) != 1 {
  •   return job.Errorf("usage: %s NAME", job.Name)
    
  • }
  • name := job.Args[0]
  • if container := daemon.Get(name); container != nil {
  •   container.Lock()
    
  •   defer container.Unlock()
    
  •   out := &engine.Env{}
    
  •   out.Set("Id", container.ID)
    
  •   out.Set("Image", container.Image)
    
  •   out.Set("Name", container.Name)
    
  •   out.SetJson("State", container.State)
    
  •   metrics := map[string]string{}
    

libcontainer's stats is limited only to native driver. Fetching metrics
directly from cgroups is more portable. I think I need to move this code to
drivers.


Reply to this email directly or view it on GitHub
https://github.com/docker/docker/pull/8886/files#r19770078.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see any metrics in lxc-info output:

# lxc-info --name 6942bf93dd69dc47b31a7236c92c2e40afc554bd9de73efbd4a2bb2e366e2e77
state:   RUNNING
pid:     30634

We always used to run custom shell scripts to fetch lxc metrics from cgroups

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need lxc-cgroup:

lxc-cgroup -n foo devices.list
          display the allowed devices to be used.

for metricKey, cgroupInfo := range metricInfoTable {
cgroupSystem, cgroupKey := cgroupInfo[0], cgroupInfo[1]
value, err := cgSubsystemValue(container.ID, cgroupSystem, cgroupKey)
if err != nil {
metrics[metricKey] = err.Error()
} else {
metrics[metricKey] = value
}
}
out.SetJson("Metrics", metrics)

if _, err := out.WriteTo(job.Stdout); err != nil {
return job.Error(err)
}
return engine.StatusOK
}
return job.Errorf("No such container: %s", name)
}

func cgSubsystemValue(id, subsystem, key string) (string, error) {
cgDir, err := cgSubsystemDir(id, subsystem)
if err != nil {
return "", err
}
data, err := ioutil.ReadFile(filepath.Join(cgDir, fmt.Sprintf("%s.%s", subsystem, key)))
if err != nil {
return "", err
}
return strings.Trim(string(data), "\r\n"+string(0)), nil
}

func cgSubsystemDir(id, subsystem string) (string, error) {
cgroupRoot, err := cgroups.FindCgroupMountpoint(subsystem)
if err != nil {
return "", err
}

cgroupDir, err := cgroups.GetThisCgroupDir(subsystem)
if err != nil {
return "", err
}

// With more recent lxc versions use, cgroup will be in lxc/, we'll search in bot
dirnames := []string{
filepath.Join(cgroupRoot, cgroupDir, id),
filepath.Join(cgroupRoot, cgroupDir, "docker", id),
filepath.Join(cgroupRoot, cgroupDir, "lxc", id),
}
for i := range dirnames {
if _, err := os.Stat(dirnames[i]); err == nil {
return dirnames[i], nil
}
}

return "", fmt.Errorf("Error: cgroup subsystem '%s' directory not found for container '%s'", subsystem, id)
}
1 change: 1 addition & 0 deletions docker/flags.go
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ func init() {
{"login", "Register or log in to a Docker registry server"},
{"logout", "Log out from a Docker registry server"},
{"logs", "Fetch the logs of a container"},
{"metrics", "Fetch metrics of a container"},
{"port", "Lookup the public-facing port that is NAT-ed to PRIVATE_PORT"},
{"pause", "Pause all processes within a container"},
{"ps", "List containers"},
Expand Down