Add image stats and integration test #257

Random-Liu · 2017-09-18T08:03:03Z

Fixes #195.
Part of #121.
Depends on kubernetes/kubernetes#52635.

This PR:

Update Kubernetes to include CRI stats fix. Fix CRI container/imagefs stats. kubernetes/kubernetes#52635
Add ImageFsInfo support.
Add integration test framework and integration test for ImageFsInfo.

@kubernetes-incubator/maintainers-cri-containerd

Random-Liu · 2017-09-18T08:07:26Z

@miaoyq This PR is using mount.Lookup function in containerd, which may be useful for your PR.

miaoyq · 2017-09-18T11:48:41Z

@Random-Liu Yes, I have used mount.GetMount function in docker, but I think mount.Lookup is better.
I will use this function after merged.

Random-Liu · 2017-09-19T00:22:06Z

@abhinandanpb @mikebrow Ready for review.

Random-Liu · 2017-09-20T01:33:38Z

Will rebase after #264 is merged.

miaoyq · 2017-09-20T14:07:19Z

integration/test_utils.go

+
+// Eventually waits for f to return nil, it checks every period, and
+// returns error if timeout exceeds.
+func Eventually(f func() (bool, error), period, timeout time.Duration) error {


It will be better if func() (bool, error) is defined as a type, and given an explanation.

Random-Liu · 2017-09-20T17:38:19Z

cmd/cri-containerd/options/options.go

@@ -31,6 +31,8 @@ const configFilePathArgName = "config"

 // ContainerdConfig contains config related to containerd
 type ContainerdConfig struct {
+	// ContainerdRootDir is the root directory path for containerd.
+	ContainerdRootDir string


As is suggested by @yanxuean, add toml here.

Signed-off-by: Lantao Liu <lantaol@google.com>

miaoyq · 2017-09-23T02:33:10Z

pkg/store/snapshot/snapshot.go

+
+// Get returns the snapshot with specified id. Returns store.ErrNotExist if the
+// snapshot doesn't exist.
+func (s *Store) Get(id string) (Snapshot, error) {


nit: id to key

miaoyq · 2017-09-23T02:37:03Z

pkg/store/snapshot/snapshot.go

+}
+
+// Delete deletes the snapshot with specified id.
+func (s *Store) Delete(id string) {


miaoyq · 2017-09-23T02:43:14Z

pkg/store/snapshot/snapshot_test.go

+	sns = s.List()
+	assert.Len(sns, 2)
+
+	t.Logf("get should return nil after deletion")


get should return empty struct and ErrNotExist after deletion

abhi

Sorry for the delay. Just did a first pass

abhi · 2017-09-25T13:38:22Z

hack/test-integration.sh

+REPORT_DIR=${REPORT_DIR:-"/tmp/test-integration"}
+
+mkdir -p ${REPORT_DIR}
+start_cri_containerd ${REPORT_DIR}


Just a thought. Should we call this testSetup and testTeardown(for kill_cri_containerd) ? Because I remember I was trying to look at why containerd was not being started and realized its started as part of this :)

Will do. :)

abhi · 2017-09-25T13:39:05Z

hack/test-utils.sh


  # Start cri-containerd
  sudo ${ROOT}/_output/cri-containerd --alsologtostderr --v 4 ${CRI_CONTAINERD_FLAGS} \
 	  &> ${report_dir}/cri-containerd.log &
+  readiness_check "sudo ${GOPATH}/bin/crictl --runtime-endpoint=${CRICONTAINERD_SOCK} info"


looks cleaner :D

abhi · 2017-09-25T13:44:33Z

pkg/os/os.go

+}
+
+// DeviceUUID gets device uuid of a device. The passed in device should be
+// a aboslute path of the device.


nit: %s/a/an

nit /aboslute/absolute/

abhi · 2017-09-25T13:48:44Z

pkg/server/imagefs_info.go

+	var usedBytes, inodesUsed uint64
+	for _, sn := range snapshots {
+		// Use the oldest timestamp as the timestamp of imagefs info.
+		if sn.Timestamp < timestamp {


just for my knowledge. This is being done because we are caching the info stats ?

We need a timestamp for the imagefs stats. And the imagefs stats is got from all snapshot stat.
Given so, it seems that we should use the earliest timestamp as the timestamp of imagefs stats?

abhi · 2017-09-25T13:57:39Z

pkg/server/imagefs_info_test.go

+	}
+	resp, err := c.ImageFsInfo(context.Background(), &runtime.ImageFsInfoRequest{})
+	require.NoError(t, err)
+	assert.Equal(t, expected, resp.GetImageFilesystems())


Is the order returned by resp.GetImageFilesystems() gauranteed to be same here ? Because if we return from cache which is in the store (map) . I am sure you must have tested this.

There will be only one filesystem stat in the list. :)

Agreed :) May be just be explicit in this case? Sorry just in case somebody reuses this test to build on top they might just expect it to be in order since its done here.

abhi · 2017-09-25T14:10:05Z

pkg/store/snapshot/snapshot.go

+
+// Store stores all snapshots.
+type Store struct {
+	lock      sync.RWMutex


nit: I think we can remove the name ? and just keep the type ?

Removing the name means exposing the Lock/Unlock function to the user, which we may want to avoid.

abhi · 2017-09-25T14:15:38Z

pkg/server/snapshots.go

+	tick := time.NewTicker(s.syncPeriod)
+	go func() {
+		defer tick.Stop()
+		for {


so how is this go routine going to exit ? should we pass a context and cancel it ?
Note: I did see the comment above :)

this seems expensive to be running on a timer even if there are no changes.. maybe a TODO to come up with better solution.

so how is this go routine going to exit ? should we pass a context and cancel it ?
Note: I did see the comment above :)

The goroutine will only stop when the process stops. Mentioned in the comment No stop function is needed because the syncer doesn't update any persistent states, it's fine to let it exit with the process.

this seems expensive to be running on a timer even if there are no changes.. maybe a TODO to come up with better solution.

Yes, this is expensive, but we have to do it. :( This is also how cadvisor works. Kubernetes retrieves stats every <10 second, and expect the stats should be returned immediately. So we have to cache the stats periodically for Kubernetes to use.

right.. I remember that.. just pointing out we could use a todo to come up with a better way.

@mikebrow OK. Will do.

mikebrow · 2017-09-25T15:13:15Z

pkg/os/os.go

+}
+
+// DeviceUUID gets device uuid of a device. The passed in device should be
+// a aboslute path of the device.


nit /aboslute/absolute/

mikebrow · 2017-09-25T15:33:42Z

pkg/server/imagefs_info.go

+		if sn.Timestamp < timestamp {
+			timestamp = sn.Timestamp
+		}
+		usedBytes += sn.Size


was the response supposed to be an aggregate response? If so why did they create an array of filesystemusage? Or is that what the below todo is for?

Because there may be multiple filesystems used for image management.

mikebrow · 2017-09-25T15:46:08Z

pkg/server/snapshots.go

+	tick := time.NewTicker(s.syncPeriod)
+	go func() {
+		defer tick.Stop()
+		for {


this seems expensive to be running on a timer even if there are no changes.. maybe a TODO to come up with better solution.

Random-Liu · 2017-09-25T17:29:51Z

@miaoyq @abhinandanpb @mikebrow Addressed/replied comments.

abhi

LGTM. One minor comment.

Random-Liu · 2017-09-25T20:59:30Z

@abhinandanpb @mikebrow Addressed comments.

Will squash the latest 2 commits after LGTM.

Signed-off-by: Lantao Liu <lantaol@google.com>

Random-Liu · 2017-09-25T21:03:06Z

Apply LGTM based on #257 (review).

Will merge it after test passes.

mikebrow

/LGTM

Random-Liu assigned abhi and mikebrow Sep 18, 2017

k8s-ci-robot added cncf-cla: yes size/XXL labels Sep 18, 2017

Random-Liu added this to the v1.0.0-alpha.0 milestone Sep 18, 2017

Random-Liu force-pushed the add-image-stats branch from ac63683 to a31f07f Compare September 18, 2017 08:06

Random-Liu changed the title ~~Add image stats~~ Add image stats and integration test Sep 18, 2017

Random-Liu force-pushed the add-image-stats branch from a31f07f to 4e80f19 Compare September 18, 2017 08:09

Random-Liu force-pushed the add-image-stats branch 3 times, most recently from bbce1d3 to 3eb8ebf Compare September 19, 2017 00:21

Random-Liu force-pushed the add-image-stats branch 3 times, most recently from e2d861f to 2e7a61a Compare September 19, 2017 01:26

Random-Liu mentioned this pull request Sep 19, 2017

Ensure the mount point is propagated #230

Merged

Random-Liu force-pushed the add-image-stats branch 3 times, most recently from 3b7ee00 to f695f06 Compare September 19, 2017 21:51

Random-Liu force-pushed the add-image-stats branch from f695f06 to 5015efa Compare September 20, 2017 04:32

abhi mentioned this pull request Sep 20, 2017

Adding container metrics support #265

Merged

miaoyq reviewed Sep 20, 2017

View reviewed changes

Random-Liu commented Sep 20, 2017

View reviewed changes

Random-Liu force-pushed the add-image-stats branch 2 times, most recently from b0a4b2f to 051307a Compare September 21, 2017 06:24

Random-Liu mentioned this pull request Sep 21, 2017

CRI-Containerd Missing Pieces #62

Closed

42 tasks

Random-Liu force-pushed the add-image-stats branch from 051307a to f8c0b1a Compare September 21, 2017 21:31

Random-Liu force-pushed the add-image-stats branch 2 times, most recently from efadb8f to e6ab872 Compare September 22, 2017 20:27

Import CRI remote client package.

b85be3d

Signed-off-by: Lantao Liu <lantaol@google.com>

Random-Liu force-pushed the add-image-stats branch from e6ab872 to 9ab6b48 Compare September 22, 2017 23:39

miaoyq reviewed Sep 23, 2017

View reviewed changes

abhi reviewed Sep 25, 2017

View reviewed changes

mikebrow reviewed Sep 25, 2017

View reviewed changes

abhi reviewed Sep 25, 2017

View reviewed changes

abhi approved these changes Sep 25, 2017

View reviewed changes

Random-Liu added 2 commits September 25, 2017 21:02

Add ImageFsInfo support

491400c

Signed-off-by: Lantao Liu <lantaol@google.com>

Add integration test framework

8fb57da

Signed-off-by: Lantao Liu <lantaol@google.com>

Random-Liu force-pushed the add-image-stats branch from fa5c285 to 8fb57da Compare September 25, 2017 21:02

Random-Liu added the lgtm label Sep 25, 2017

mikebrow approved these changes Sep 25, 2017

View reviewed changes

Random-Liu merged commit b9200ac into containerd:master Sep 25, 2017

Random-Liu deleted the add-image-stats branch September 25, 2017 22:35

miaoyq mentioned this pull request Sep 27, 2017

Get the mountInfo by 'LookupMount' in containerd #295

Merged

Add image stats and integration test #257

Add image stats and integration test #257

Conversation

Random-Liu commented Sep 18, 2017

Random-Liu commented Sep 18, 2017

miaoyq commented Sep 18, 2017

Random-Liu commented Sep 19, 2017

Random-Liu commented Sep 20, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abhi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikebrow Sep 25, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikebrow Sep 25, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Random-Liu Sep 25, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Random-Liu commented Sep 25, 2017

abhi left a comment

Choose a reason for hiding this comment

Random-Liu commented Sep 25, 2017

Random-Liu commented Sep 25, 2017

mikebrow left a comment

Choose a reason for hiding this comment

mikebrow Sep 25, 2017 •

edited

Loading

mikebrow Sep 25, 2017 •

edited

Loading

Random-Liu Sep 25, 2017 •

edited

Loading