Add a monitor that monitors the CPU usage of a container

[VincenzoFerme]

The required functionalities are:

1. it must work from inside a container;
2. it must monitor the Stats of a list of containers identified by names and provided through an Environment variables. The containers's name are separated by ":";
3. it must store the average CPU usage for the following time intervals: last 5sec, last 30sec, last 60sec;
4. it must define an API to access to the stored information about the CPU averages. It should be possible to query these data for all the containers (average of the averages CPU), or by individual container identified by its name;
5. it must work with the less impact as possible on the monitored container performance.

Notes about CPU usage:

• Compute the percentage usage given the number of core assigned to a container, not according to the host
• cpushares: can be a relative weight to other containers
• total_usage: CPU percentage is not feasible because Docker enables many options to share the CPU with other containers. We use the total_usage instead, and we keep a delta in different intervals

Some useful references:

1. Powerful go-dockerclient and Stats API: https://godoc.org/github.com/fsouza/go-dockerclient#Client.Stats
2. A test case that show how to use the API using the client at point 1: https://github.com/fsouza/go-dockerclient/blob/34eaaf52874d8ce5d57be011a4852eb83d950125/container_test.go#L1630
3. Docker Stats APIs: https://docs.docker.com/reference/api/docker_remote_api_v1.20/#get-container-stats-based-on-resource-usage

[VincenzoFerme]

@Cerfoglg what is still missing in your #7 pull request?

[Cerfoglg]

Stil doesn't connect properly to the docker socket directly, as we have seen a few weeks ago

[VincenzoFerme]

After some debugging, I managed to have a code that works. If you put the following code the the main method of the cpu monitor, you get the statistics:

id := "db"
endpoint := "unix:///var/run/docker.sock"
client, err := docker.NewClient(endpoint)
   if err != nil {
   log.Fatal(err)
}

errC := make(chan error, 1)
statsC := make(chan *docker.Stats)
done := make(chan bool)
go func() {
   errC <- client.Stats(docker.StatsOptions{id, statsC, true, done, 0})
   close(errC)
}()

for {
   stats, ok := <-statsC
   if !ok {
      break
   }
   //done <- true
   fmt.Println(stats.CPUStats.CPUUsage.TotalUsage)
}
err = <-errC
if err != nil {
   fmt.Println(err)
}

The main difference with respect to you code is that I set 0 as timeout. During the tests I noticed that you get on the socket if you set a timeout different with 0. We don't need to change it, the default behaviour is fine for us.

You need to share the docker socket with the container, to make it work:

docker run -v /var/run/docker.sock:/var/run/docker.sock -t --rm <IMAGE_NAME>

With the approach provided before, also the done channel seems to work (ref to: benchflow/collectors#23).

@Cerfoglg let me know if you manage to solve this issue.

[Cerfoglg]

@VincenzoFerme For me it didn't solve the issue of the done channel by just setting the timeout to 0. However, I was only able to test with TSLClient instead of the direct socket connection (having a few issues with using a VM and docker), so it is possible that this only works with the direct socket connection. Either way, if it works like that for you, my code should work as well.

[Cerfoglg]

@VincenzoFerme I was able to get it to work. Turns out the Stats function will return an error when signalling on the Done channel, but the function does indeed stop. Docker socket works as well by sharing the socket with the container upon running it.

I did comment on the repo of the dockerclient API about the Stats function giving an error when stopping it via the Done channel. I believe it would be best if it didn't do that, but rather exited more gracefully.

[VincenzoFerme]

@Cerfoglg ok, lets wait for an answer and see if it can be improved. It seems enough for now, as solved in #10