Add node e2e test CI. #145

Random-Liu · 2017-08-20T07:44:09Z

This PR:

Added node e2e test CI. I've enabled the daily build cron job, and node e2e test usually takes 20min, currently let's only run it in the cron job to accelerate our development.
Change travis to only build with golang 1.8.x and tip, but only test with 1.8.x. @mikebrow
Clearly define the dependencies between different Make targets.

Please note that:

I've commented the TRAVIS_EVENT_TYPE related code for now, so that node e2e test will still be triggered in this PR. I'll uncomment that after node e2e passes.
I'm also working on setup a node e2e test framework in our test infra, which will test against different node image.

/cc @kubernetes-incubator/maintainers-cri-containerd
Signed-off-by: Lantao Liu lantaol@google.com

Random-Liu · 2017-08-20T07:45:49Z

I've run all the non-slow, non-serial and non-flaky test locally, here is the test result:

Summarizing 10 Failures:

[Fail] [k8s.io] Security Context When creating a container with runAsUser [It] should run the container with uid 65534 
/home/lantaol/workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:202

[Fail] [k8s.io] Kubelet Cgroup Manager Pod containers On scheduling a Guaranteed Pod [It] Pod containers should have been created under the cgroup-root 
/home/lantaol/workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/pods_container_manager_test.go:205

[Fail] [k8s.io] Kubelet Cgroup Manager Pod containers On scheduling a Burstable Pod [It] Pod containers should have been created under the Burstable cgroup 
/home/lantaol/workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/pods_container_manager_test.go:293

[Fail] [k8s.io] Security Context when creating containers with AllowPrivilegeEscalation [It] should not allow privilege escalation when false 
/home/lantaol/workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/pods.go:202

[Fail] [k8s.io] Kubelet Cgroup Manager Pod containers On scheduling a BestEffort Pod [It] Pod containers should have been created under the BestEffort cgroup 
/home/lantaol/workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/pods_container_manager_test.go:249

[Fail] [k8s.io] Summary API when querying /stats/summary [It] should report resource usage through the stats api 
/home/lantaol/workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/summary_test.go:263

[Fail] [k8s.io] ImageID [It] should be set to the manifest digest (from RepoDigests) when available 
/home/lantaol/workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/image_id_test.go:64

[Fail] [k8s.io] AppArmor [Feature:AppArmor] when running with AppArmor [It] should enforce a permissive profile 
/home/lantaol/workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/apparmor_test.go:149

[Fail] [k8s.io] Kubelet Cgroup Manager QOS containers On enabling QOS cgroup hierarchy [It] Top level QoS containers should have been created 
/home/lantaol/workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/pods_container_manager_test.go:167

[Fail] [k8s.io] AppArmor [Feature:AppArmor] when running with AppArmor [It] should enforce a profile blocking writes 
/home/lantaol/workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/apparmor_test.go:149

Ran 183 of 246 Specs in 1372.085 seconds
FAIL! -- 173 Passed | 10 Failed | 0 Pending | 63 Skipped 

Ginkgo ran 1 suite in 22m52.377204165s
Test Suite Failed

Failures are mainly caused by several missing features:

RunAsUser
AppArmor
Metrics
Cgroup hierarchy

mikebrow · 2017-08-20T13:35:30Z

.travis.yml

      script:
        - make install.deps
        - make test
-        - make binaries


make test-cri will build anyway so removing make binaries just postpones the step..

Suggest keeping the name Verify.. and removing the make binaries from the 1.8.x since that will be done anyway in the test step for 1.8.3. But keep the make binaries in tip as an extra step since build and test isn't being run for it. Suggest keeping build and test since it actually does build then test...

I feel like Build and Test seems better, which is more clear. :)

Build for tip but not build for 1.8.x makes people confusing. Build doesn't actually take too long, so an extra build seems acceptable?

mostly just pointing out that deleting the -make binaries line did not remove the call to make binaries. Yeah an extra build is acceptable since it's so fast. Just pointing out it's also unnecessary on the 1.8 build because you'll be doing it twice... not sure why it's confusing to have a build in a verify step but not verify in a build step :-)

mikebrow

See comments. Overall great new feature!

mikebrow · 2017-08-20T13:40:20Z

.travis.yml

      go: 1.8.x
-    - script:


ok sure, not doing the long test twice is ok, the verify & build step for tip should be enough to make sure most of the golang issues are addressed. Will only miss runtime issues.

mikebrow · 2017-08-20T14:00:58Z

Makefile

 verify: lint gofmt boiler

 version:
 	@echo $(VERSION)

-lint: check-gopath


was only necessary to do this test prior to go 1.8 and we require 1.8 so.. yeah ok to remove :-) Maybe to a go version check?

The problem is that check-gopath is a PHONY,, target depends on it will run every time.
I want to avoid rebuild, if the binaries are built and source code is not changed, so I have to remove this.

mikebrow · 2017-08-20T14:03:37Z

Makefile

 clean:
-	rm -f $(BUILD_DIR)/cri-containerd
+	rm -rf $(BUILD_DIR)/*


d seems not required if r is specified.

d is even if the directory is empty..

IIUC, r is a superset of d. :)

interesting.. from quick test I see you are correct. Thx

mikebrow · 2017-08-20T14:11:20Z

Makefile

 	@./hack/test-cri.sh

+test-e2e-node: binaries


need to add test-e2e-node to the above help section...

mikebrow · 2017-08-20T14:27:39Z

hack/test-utils.sh

@@ -0,0 +1,53 @@
+#!/bin/bash


mikebrow · 2017-08-20T14:32:31Z

hack/test-e2e-node.sh

+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+set -o nounset


# prevent unset variables

All script has this, don't think we want to comment all of them. :)

mikebrow · 2017-08-20T14:35:34Z

hack/test-e2e-node.sh

+# See the License for the specific language governing permissions and
+# limitations under the License.
+set -o nounset
+set -o pipefail


# force script to exit if any command in a pipeline errors

mikebrow · 2017-08-20T14:37:55Z

hack/test-e2e-node.sh

+export SKIP=${SKIP:-${DEFAULT_SKIP}}
+REPORT_DIR=${REPORT_DIR:-"/tmp/test-e2e-node"}
+
+if [[ -z "${GOPATH}" ]]; then


That's ironic :-)

See comment above.

mikebrow

/LGTM your call on the nits.

Cheers!

Random-Liu · 2017-08-21T01:09:00Z

@mikebrow Let's wait for the test to finish, till know everything looks good. :)
With this PR, we'll have node e2e CI test running. We should be more confident with cri-containerd then. :)

@kubernetes-incubator/maintainers-cri-containerd

mikebrow · 2017-08-21T01:35:08Z

Travis is in a bad state. The last test seems to have over run the log max of 4m and CI was not successful in stopping the jobs.

Random-Liu · 2017-08-21T17:19:57Z

@mikebrow Yeah, we are printing log into travis test output directly. It seems that we have to upload them somewhere else.

Random-Liu · 2017-08-21T18:04:42Z

@mikebrow Separated the e2e test into another stage, and only print out cri-containerd log for now.

mikebrow · 2017-08-21T18:35:21Z

.travis.yml

-    - script:
+    - stage: E2E Test
+      script:
+        # TODO(random-liu): Uncomment after test passes.


What's the cron job schedule?

Daily for now. There are only 3 options: daily, weekly and monthly.

mikebrow · 2017-08-21T18:36:43Z

.travis.yml

+        - make test-e2e-node
+      after_script:
+        # TODO(random-liu): Upload log to GCS.
+        - test -f /tmp/test-e2e-node/cri-containerd.log && cat /tmp/test-e2e-node/cri-containerd.log


probably a todo or issue here to figure out how to filter the logs a bit more so there's room in our 4m limit to get the containerd log...

@mikebrow I've changed the log level, :)
I think the best solution is to upload the log onto GCS. Or run the test in kubernetes test_infra, which I'm also working on recently.

mikebrow

/LGTM just a nit for a todo comment and a question

mikebrow · 2017-08-21T19:00:24Z

All pass so good to remove comments from if not cron job skip for the e2e bucket :-)

Random-Liu · 2017-08-21T19:03:14Z

seems that some network related test fails. We need to figure it out. :)

…

On Aug 21, 2017 12:00 PM, "Mike Brown" ***@***.***> wrote: All pass so good to remove comments from if not cron job skip for the e2e bucket :-) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#145 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFjVuzO9H8Fa8Qi6sk9nS8n226swzR7Dks5sadPKgaJpZM4O8i3l> .

Random-Liu · 2017-08-21T20:46:32Z

@mikebrow Actually, there are still test failing which succeed in my environment:

Summarizing 6 Failures:
[Fail] [k8s.io] Container Lifecycle Hook when create a pod with lifecycle hook [It] should execute prestop exec hook properly [Conformance] 
/home/travis/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/lifecycle_hook_test.go:83
[Fail] [k8s.io] Networking [k8s.io] Granular Checks: Pods [It] should function for intra-pod communication: http [Conformance] 
/home/travis/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:217
[Fail] [k8s.io] Projected [It] should project all components that make up the projection API [Conformance] [Volume] [Projection] 
/home/travis/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/util.go:2224
[Fail] [k8s.io] Container Lifecycle Hook when create a pod with lifecycle hook [It] should execute poststart exec hook properly [Conformance] 
/home/travis/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/lifecycle_hook_test.go:74
[Fail] [k8s.io] Networking [k8s.io] Granular Checks: Pods [It] should function for intra-pod communication: udp [Conformance] 
/home/travis/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e/framework/networking_utils.go:217
[Fail] [k8s.io] Container Runtime Conformance Test container runtime conformance blackbox test when running a container with a new image [It] should be able to pull from private registry with secret [Conformance] 
/home/travis/gopath/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:385
Ran 163 of 223 Specs in 1305.905 seconds
FAIL! -- 157 Passed | 6 Failed | 0 Pending | 60 Skipped

Because I didn't return that error code, the test passes.

However, I thought it was OK. Let's get the test running first, and then fix these test failures. @mikebrow

Signed-off-by: Lantao Liu <lantaol@google.com>

Add node e2e test CI.

k8s-ci-robot added the cncf-cla: yes label Aug 20, 2017

Random-Liu assigned mikebrow Aug 20, 2017

mikebrow reviewed Aug 20, 2017

View reviewed changes

Random-Liu force-pushed the add-node-e2e-ci branch from 21855e6 to c3ebd78 Compare August 21, 2017 00:39

mikebrow approved these changes Aug 21, 2017

View reviewed changes

Random-Liu force-pushed the add-node-e2e-ci branch 2 times, most recently from 56199a0 to c8814d2 Compare August 21, 2017 18:04

mikebrow reviewed Aug 21, 2017

View reviewed changes

mikebrow approved these changes Aug 21, 2017

View reviewed changes

mikebrow added the lgtm label Aug 21, 2017

Random-Liu force-pushed the add-node-e2e-ci branch from c8814d2 to 6694e4b Compare August 21, 2017 20:49

Random-Liu mentioned this pull request Aug 21, 2017

Add container attach #128

Merged

Random-Liu force-pushed the add-node-e2e-ci branch from 6694e4b to b860d56 Compare August 21, 2017 21:15

Add node e2e test CI.

c05a7e7

Signed-off-by: Lantao Liu <lantaol@google.com>

Random-Liu force-pushed the add-node-e2e-ci branch from b860d56 to c05a7e7 Compare August 21, 2017 21:23

Random-Liu merged commit 010a562 into containerd:master Aug 21, 2017

Random-Liu deleted the add-node-e2e-ci branch August 21, 2017 21:41

Random-Liu mentioned this pull request Aug 24, 2017

Update Test Document for Node E2E #162

Closed

lanchongyizu pushed a commit to lanchongyizu/cri-containerd that referenced this pull request Sep 3, 2017

Merge pull request containerd#145 from Random-Liu/add-node-e2e-ci

2c574d1

Add node e2e test CI.

Add node e2e test CI. #145

Add node e2e test CI. #145

Conversation

Random-Liu commented Aug 20, 2017 • edited

Random-Liu commented Aug 20, 2017

mikebrow Aug 20, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikebrow left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Random-Liu Aug 21, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikebrow left a comment • edited

Choose a reason for hiding this comment

Random-Liu commented Aug 21, 2017

mikebrow commented Aug 21, 2017

Random-Liu commented Aug 21, 2017 • edited

Random-Liu commented Aug 21, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikebrow Aug 21, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikebrow left a comment

Choose a reason for hiding this comment

mikebrow commented Aug 21, 2017

Random-Liu commented Aug 21, 2017 via email

Random-Liu commented Aug 21, 2017

Random-Liu commented Aug 20, 2017 •

edited

mikebrow Aug 20, 2017 •

edited

Random-Liu Aug 21, 2017 •

edited

mikebrow left a comment •

edited

Random-Liu commented Aug 21, 2017 •

edited

mikebrow Aug 21, 2017 •

edited