test: Improve `gke/{select,release}-cluster.sh` scripts #11173

errordeveloper · 2020-04-27T14:47:54Z

use explicit paths
make better use of gcloud features, reduce external dependencies
replace get-cluster-version.sh with a version file
use region instead of zone to enable regional spread and avoid availibity issue due to a zonal outages; namely we can create clusters in any region/zone and the Jenkins job is not tied into any particular one
list node pools instead of assuming default pool

coveralls · 2020-04-27T15:17:13Z

Coverage decreased (-0.01%) to 44.635% when pulling c18f5f9 on pr/errordeveloper/improve-gke-scripts into 15f1a12 on master.

errordeveloper · 2020-04-27T19:19:21Z

test-me-please

errordeveloper · 2020-04-27T19:26:30Z

@nebril I'm happy with this one now, please take a look when you get a chance. Once it's in, I'll start new clusters rolling in.

nebril

Good changes overall, there is one issue though - get-cluster-version.sh was used in gke jenkinsfile, please change https://github.com/cilium/cilium/blob/master/jenkinsfiles/ginkgo-gke.Jenkinsfile#L141 to use new scheme of getting k8s version.

nebril

OK, I totally missed the first file in review xD

nebril · 2020-04-28T09:06:53Z

test-gke

nebril · 2020-04-28T09:18:25Z

retest-ginkgo

jenkinsfiles/ginkgo-gke.Jenkinsfile

errordeveloper · 2020-04-28T12:49:47Z

test-gke

nebril · 2020-04-28T14:21:47Z

test-gke

nebril · 2020-04-28T14:49:31Z

test-gke

errordeveloper · 2020-04-28T16:43:36Z

@nebril did it fail before you re-ran it? I was hoping to have a look at the logs... do you have a link handy, or was it an unrelated flake?

nebril · 2020-04-28T17:02:30Z

@errordeveloper it was an unrelated flake while building docker image: https://jenkins.cilium.io/job/Cilium-PR-K8s-GKE/745/execution/node/78/log/

errordeveloper · 2020-04-28T17:50:35Z

@nebril I see. I'm seeing a different thing now:

20:34:18  Archiving artifacts
20:34:19  java.lang.InterruptedException: no matches found within 10000
20:34:19  	at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2803)
20:34:19  	at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2682)
20:34:19  	at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2663)
20:34:19  	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3050)
20:34:19  Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to jenkins-node-gke-fixed-0
20:34:19  		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
20:34:19  		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
20:34:19  		at hudson.remoting.Channel.call(Channel.java:957)
20:34:19  		at hudson.FilePath.act(FilePath.java:1070)
20:34:19  		at hudson.FilePath.act(FilePath.java:1059)
20:34:19  		at hudson.FilePath.validateAntFileMask(FilePath.java:2661)
20:34:19  		at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
20:34:19  		at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
20:34:19  		at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
20:34:19  		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
20:34:19  		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
20:34:19  Caused: hudson.FilePath$TunneledInterruptedException
20:34:19  	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3052)
20:34:19  	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
20:34:19  	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
20:34:19  	at hudson.remoting.Request$2.run(Request.java:369)
20:34:19  	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
20:34:19  Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
20:34:19  	at hudson.FilePath.act(FilePath.java:1072)
20:34:19  	at hudson.FilePath.act(FilePath.java:1059)
20:34:19  	at hudson.FilePath.validateAntFileMask(FilePath.java:2661)
20:34:19  	at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
20:34:19  	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
20:34:19  	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
20:34:19  	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
20:34:19  	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
20:34:19  	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
20:34:19  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
20:34:19  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
20:34:19  	at java.lang.Thread.run(Thread.java:748)

Should we just ignore that or it could be not completely random?

aanm

Small nit, can't we merge both scripts and use an bash argument instead to pick which operation we want to do in the clusters? like test/gke/cluster.sh release and test/gke/cluster.sh select

errordeveloper · 2020-04-29T09:17:10Z

@aanm thanks for the review! To be honest, I am not sure what cluster.sh would buy us now, the idea is to make a few short-term improvements, but long-term the aim is to replace these scripts with something that relies on an API (CRD-based) and there would be an operator. The GKE version of the operator would manage a dynamic pool of clusters and could just create new clusters instead of attempting to recycle them, there is a fairly simple way of implementing this for GKE actually. That's the current plan!

Signed-off-by: Ilya Dmitrichenko <errordeveloper@gmail.com>

- use explicit paths - make better use of `gcloud` features, reduce external dependencies - replace `get-cluster-version.sh` with a version file - use region instead of zone to enable regional spread and avoid availibity issue due to a zonal outage; namely we can create clusters in any region/zone and the Jenkins job is not tied into any particular one - list node pools instead of assuming default pool Signed-off-by: Ilya Dmitrichenko <errordeveloper@gmail.com>

errordeveloper · 2020-04-29T09:23:05Z

Here's what failed last time in the GKE suite:

01:19:59  Summarizing 54 Failures:
01:19:59  
01:19:59  [Fail] K8sKafkaPolicyTest Kafka Policy Tests [BeforeEach] KafkaPolicies 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/k8sT/assertionHelpers.go:98
01:19:59  
01:19:59  [Fail] K8sChaosTest Restart with long lived connections [It] L3/L4 policies still work while Cilium is restarted 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/k8sT/Chaos.go:261
01:19:59  
01:19:59  [Fail] K8sHubbleTest [BeforeEach] Hubble Observe Test L3/L4 Flow 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/k8sT/hubble.go:130
01:19:59  
01:19:59  [Fail] K8sHubbleTest [BeforeEach] Hubble Observe Test L7 Flow 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sFQDNTest [BeforeEach] Restart Cilium validate that FQDN is still working 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/k8sT/fqdn.go:80
01:19:59  
01:19:59  [Fail] K8sFQDNTest [BeforeEach] Validate that multiple specs are working correctly 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sPolicyTest Basic Test [BeforeEach] checks all kind of Kubernetes policies 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/k8sT/Policies.go:196
01:19:59  
01:19:59  [Fail] K8sPolicyTest Basic Test [BeforeEach] Invalid Policy report status correctly 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sPolicyTest Basic Test [BeforeEach] ServiceAccount Based Enforcement 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sPolicyTest Basic Test [BeforeEach] CNP test MatchExpressions key 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sPolicyTest Basic Test [BeforeEach] Denies traffic with k8s default-deny ingress policy 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sPolicyTest Basic Test [BeforeEach] Denies traffic with k8s default-deny egress policy 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sPolicyTest Basic Test [BeforeEach] Denies traffic with k8s default-deny ingress-egress policy 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sPolicyTest Basic Test [BeforeEach] Denies traffic with cnp default-deny ingress policy 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sPolicyTest Basic Test [BeforeEach] Denies traffic with cnp default-deny egress policy 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sPolicyTest Basic Test [BeforeEach] Allows traffic with k8s default-allow ingress policy 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sPolicyTest Basic Test [BeforeEach] Allows traffic with k8s default-allow egress policy 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sPolicyTest Basic Test [BeforeEach] Validate to-entities policies Validate toEntities All 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sPolicyTest Basic Test [BeforeEach] Validate to-entities policies Validate toEntities World 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sPolicyTest Basic Test [BeforeEach] Validate to-entities policies Validate toEntities Cluster 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sPolicyTest Basic Test [BeforeEach] Validate to-entities policies Validate toEntities Host 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sPolicyTest Basic Test [BeforeEach] Validate CNP update Enforces connectivity correctly when the same L3/L4 CNP is updated 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sPolicyTest Basic Test [BeforeEach] Validate CNP update Verifies that a CNP with L7 HTTP rules can be replaced with L7 Kafka rules 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sPolicyTest Basic Test [BeforeEach] Traffic redirections to proxy Tests HTTP proxy visibility without policy 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sPolicyTest Basic Test [BeforeEach] Traffic redirections to proxy Tests DNS proxy visibility without policy 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sPolicyTest Basic Test [BeforeEach] Traffic redirections to proxy Tests proxy visibility interactions with policy lifecycle operations 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sPolicyTest Multi-node policy test validates fromEntities policies [BeforeEach] with remote-node identity disabled Allows from all hosts with cnp fromEntities host policy 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/k8sT/Policies.go:1017
01:19:59  
01:19:59  [Fail] K8sPolicyTest Multi-node policy test validates fromEntities policies [BeforeEach] with remote-node identity enabled Validates fromEntities remote-node policy 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sPolicyTest GuestBook Examples [It] checks policy example 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/k8sT/Policies.go:1208
01:19:59  
01:19:59  [Fail] K8sPolicyTest Namespaces policies [BeforeEach] Tests the same Policy in different namespaces 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/k8sT/Policies.go:1312
01:19:59  
01:19:59  [Fail] K8sPolicyTest Namespaces policies [BeforeEach] Kubernetes Network Policy by namespace selector 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sPolicyTest Namespaces policies [BeforeEach] Cilium Network policy using namespace label and L7 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sPolicyTest Clusterwide policies [BeforeEach] Test clusterwide connectivity with policies 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/k8sT/Policies.go:1527
01:19:59  
01:19:59  [Fail] K8sDatapathConfig MonitorAggregation [It] Checks that monitor aggregation restricts notifications 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/k8sT/DatapathConfiguration.go:138
01:19:59  
01:19:59  [Fail] K8sDatapathConfig MonitorAggregation [It] Checks that monitor aggregation flags send notifications 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/k8sT/DatapathConfiguration.go:194
01:19:59  
01:19:59  [Fail] K8sDatapathConfig Encapsulation [It] Check connectivity with sockops and VXLAN encapsulation 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/k8sT/DatapathConfiguration.go:261
01:19:59  
01:19:59  [Fail] K8sDatapathConfig Encapsulation [It] Check connectivity with VXLAN encapsulation 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/k8sT/DatapathConfiguration.go:270
01:19:59  
01:19:59  [Fail] K8sServicesTest [BeforeEach] Checks ClusterIP Connectivity Checks service on same node 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/k8sT/Services.go:65
01:19:59  
01:19:59  [Fail] K8sServicesTest [BeforeEach] Checks ClusterIP Connectivity Checks service accessing itself (hairpin flow) 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sServicesTest [BeforeEach] Checks ClusterIP Connectivity IPv6 Connectivity 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sServicesTest [BeforeEach] Checks service across nodes Checks ClusterIP Connectivity 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sServicesTest [BeforeEach] Checks service across nodes IPv6 Connectivity 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sServicesTest [BeforeEach] Checks service across nodes Tests NodePort (kube-proxy) 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sServicesTest [BeforeEach] Checks service across nodes Tests NodePort (kube-proxy) with externalTrafficPolicy=Local 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sServicesTest [BeforeEach] Checks service across nodes with L7 policy Tests NodePort with L7 Policy 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sServicesTest [BeforeEach] Checks service across nodes Tests NodePort BPF 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sServicesTest [BeforeEach] Checks service across nodes Supports IPv4 fragments 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sServicesTest [BeforeEach] External services To Services first endpoint creation 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sServicesTest [BeforeEach] External services To Services first policy 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sServicesTest [BeforeEach] External services To Services first endpoint creation match service by labels 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sServicesTest [BeforeEach] External services To Services first policy, match service by labels 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sServicesTest [BeforeEach] Bookinfo Demo Tests bookinfo demo 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:539
01:19:59  
01:19:59  [Fail] K8sUpdates [It] Tests upgrade and downgrade from a Cilium stable image to master 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/k8sT/Updates.go:335
01:19:59  
01:19:59  [Fail] K8sHealthTest [It] checks cilium-health status between nodes 
01:19:59  /home/jenkins/workspace/Cilium-PR-K8s-GKE@6/src/github.com/cilium/cilium/test/k8sT/Health.go:61
01:19:59  
01:19:59  Ran 61 of 387 Specs in 12207.009 seconds
01:19:59  FAIL! -- 7 Passed | 54 Failed | 1 Pending | 325 Skipped
01:19:59  --- FAIL: TestTest (12207.02s)

I am not sure what's going in, will rebase and re-run.

errordeveloper · 2020-04-29T09:23:48Z

test-me-please

errordeveloper · 2020-04-29T13:54:59Z

GKE:

12:44:23  [Fail] K8sPolicyTest Multi-node policy test validates fromEntities policies with remote-node identity enabled [It] Validates fromEntities remote-node policy 
12:44:23  /home/jenkins/workspace/Cilium-PR-K8s-GKE@8/src/github.com/cilium/cilium/test/k8sT/Policies.go:1108
12:44:23  
12:44:23  [Fail] K8sServicesTest Checks service across nodes with L7 policy [It] Tests NodePort with L7 Policy 
12:44:23  /home/jenkins/workspace/Cilium-PR-K8s-GKE@8/src/github.com/cilium/cilium/test/k8sT/Services.go:573
12:44:23  
12:44:23  Ran 57 of 389 Specs in 7762.361 seconds
12:44:23  FAIL! -- 55 Passed | 2 Failed | 1 Pending | 331 Skipped
12:44:23  --- FAIL: TestTest (7762.36s)

Others have different failures:

11:47:27  • Failure in Spec Setup (BeforeEach) [264.820 seconds]
11:47:27  K8sServicesTest
11:47:27  /home/jenkins/workspace/Cilium-PR-Ginkgo-Tests-Kernel/k8s-1.17-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:396
11:47:27    External services
11:47:27    /home/jenkins/workspace/Cilium-PR-Ginkgo-Tests-Kernel/k8s-1.17-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:396
11:47:27      To Services first endpoint creation [BeforeEach]
11:47:27      /home/jenkins/workspace/Cilium-PR-Ginkgo-Tests-Kernel/k8s-1.17-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:431
11:47:27  
11:47:27      Endpoints are not ready after timeout
[2020-04-29T10:47:27.994Z]     Expected
[2020-04-29T10:47:27.994Z]         <*helpers.SSHMetaError | 0xc0019b11c0>: Error: context deadline exceeded
[2020-04-29T10:47:27.994Z]     Extended info: 	Cilium Pod: cilium-77rvp 	Endpoint: 929 	Identity: 5	 State: ready
[2020-04-29T10:47:27.994Z]     	Cilium Pod: cilium-77rvp 	Endpoint: 2704 	Identity: 4	 State: ready
[2020-04-29T10:47:27.994Z]     	Cilium Pod: cilium-7z2v8 	Endpoint: 223 	Identity: 11388	 State: ready
[2020-04-29T10:47:27.994Z]     	Cilium Pod: cilium-7z2v8 	Endpoint: 675 	Identity: 27860	 State: ready
[2020-04-29T10:47:27.994Z]     	Cilium Pod: cilium-7z2v8 	Endpoint: 1516 	Identity: 4	 State: ready
[2020-04-29T10:47:27.994Z]     
[2020-04-29T10:47:27.995Z]     
[2020-04-29T10:47:27.995Z]     to be nil
11:47:27  
11:47:27      /home/jenkins/workspace/Cilium-PR-Ginkgo-Tests-Kernel/k8s-1.17-gopath/src/github.com/cilium/cilium/test/k8sT/Services.go:1258

and

11:18:54  • Failure [79.193 seconds]
11:18:54  K8sServicesTest
11:18:54  /home/jenkins/workspace/Cilium-PR-Ginkgo-Tests-Validated/k8s-1.18-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:396
11:18:54    Checks service across nodes
11:18:54    /home/jenkins/workspace/Cilium-PR-Ginkgo-Tests-Validated/k8s-1.18-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:396
11:18:54      Tests NodePort (kube-proxy) [It]
11:18:54      /home/jenkins/workspace/Cilium-PR-Ginkgo-Tests-Validated/k8s-1.18-gopath/src/github.com/cilium/cilium/test/ginkgo-ext/scopes.go:431
11:18:54  
11:18:54      Pod "testclient-r47vc" can not connect to service "tftp://192.168.36.12:32524/hello" (failed in request 3/10)
[2020-04-29T10:18:54.283Z]     Expected command: kubectl exec -n default testclient-r47vc -- curl --path-as-is -s -D /dev/stderr --fail --connect-timeout 5 --max-time 8 tftp://192.168.36.12:32524/hello -w "time-> DNS: '%{time_namelookup}(%{remote_ip})', Connect: '%{time_connect}',Transfer '%{time_starttransfer}', total '%{time_total}'" 
[2020-04-29T10:18:54.283Z]     To succeed, but it failed:
[2020-04-29T10:18:54.283Z]     Exitcode: 28 
[2020-04-29T10:18:54.283Z]     Stdout:
[2020-04-29T10:18:54.283Z]      	 time-> DNS: '0.000020()', Connect: '0.000040',Transfer '0.000000', total '5.004996'
[2020-04-29T10:18:54.283Z]     Stderr:
[2020-04-29T10:18:54.283Z]      	 command terminated with exit code 28
[2020-04-29T10:18:54.283Z]     	 
[2020-04-29T10:18:54.283Z]     
11:18:54  
11:18:54      /home/jenkins/workspace/Cilium-PR-Ginkgo-Tests-Validated/k8s-1.18-gopath/src/github.com/cilium/cilium/test/k8sT/Services.go:622

nebril · 2020-04-29T14:00:46Z

All failures were unrelated to changes, we need this to go forward with stabilising GKE ci, merging it as is.

errordeveloper · 2020-04-29T14:58:11Z

@nebril thanks for merging! 🚀

errordeveloper requested a review from nebril April 27, 2020 14:47

errordeveloper requested a review from a team as a code owner April 27, 2020 14:47

This comment has been minimized.

Sign in to view

maintainer-s-little-helper bot added the dont-merge/needs-release-note label Apr 27, 2020

maintainer-s-little-helper bot added this to In progress in 1.8.0 Apr 27, 2020

errordeveloper added the release-note/ci This PR makes changes to the CI. label Apr 27, 2020

maintainer-s-little-helper bot removed the dont-merge/needs-release-note label Apr 27, 2020

This comment has been minimized.

Sign in to view

errordeveloper force-pushed the pr/errordeveloper/improve-gke-scripts branch from a029658 to 69eb72b Compare April 27, 2020 15:02

This comment has been minimized.

Sign in to view

errordeveloper force-pushed the pr/errordeveloper/improve-gke-scripts branch from 69eb72b to 8040be5 Compare April 27, 2020 18:06

This comment has been minimized.

Sign in to view

errordeveloper force-pushed the pr/errordeveloper/improve-gke-scripts branch 2 times, most recently from 1ec2e5f to d434acf Compare April 27, 2020 19:06

This comment has been minimized.

Sign in to view

errordeveloper force-pushed the pr/errordeveloper/improve-gke-scripts branch from d434acf to 3d7e249 Compare April 27, 2020 19:19

nebril requested changes Apr 28, 2020

View reviewed changes

nebril approved these changes Apr 28, 2020

View reviewed changes

nebril requested changes Apr 28, 2020

View reviewed changes

jenkinsfiles/ginkgo-gke.Jenkinsfile Show resolved Hide resolved

errordeveloper force-pushed the pr/errordeveloper/improve-gke-scripts branch from 3d7e249 to 87fab3c Compare April 28, 2020 12:46

nebril added the ci/fail-fast This label makes a CI build fail immediately if any test across all test suites fails. label Apr 28, 2020

This comment has been minimized.

Sign in to view

aanm reviewed Apr 29, 2020

View reviewed changes

errordeveloper added 2 commits April 29, 2020 10:20

test: Remove trailing while space

5444b66

Signed-off-by: Ilya Dmitrichenko <errordeveloper@gmail.com>

errordeveloper force-pushed the pr/errordeveloper/improve-gke-scripts branch from 87fab3c to c18f5f9 Compare April 29, 2020 09:21

nebril merged commit 2e957af into master Apr 29, 2020

1.8.0 automation moved this from In progress to Merged Apr 29, 2020

nebril deleted the pr/errordeveloper/improve-gke-scripts branch April 29, 2020 14:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: Improve `gke/{select,release}-cluster.sh` scripts #11173

test: Improve `gke/{select,release}-cluster.sh` scripts #11173

errordeveloper commented Apr 27, 2020 •

edited

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

coveralls commented Apr 27, 2020 •

edited

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

errordeveloper commented Apr 27, 2020

errordeveloper commented Apr 27, 2020

nebril left a comment

nebril left a comment

nebril commented Apr 28, 2020

nebril commented Apr 28, 2020

errordeveloper commented Apr 28, 2020

nebril commented Apr 28, 2020

nebril commented Apr 28, 2020

errordeveloper commented Apr 28, 2020 •

edited

nebril commented Apr 28, 2020

errordeveloper commented Apr 28, 2020

This comment has been minimized.

aanm left a comment •

edited

errordeveloper commented Apr 29, 2020 •

edited

errordeveloper commented Apr 29, 2020 •

edited

errordeveloper commented Apr 29, 2020

errordeveloper commented Apr 29, 2020

nebril commented Apr 29, 2020

errordeveloper commented Apr 29, 2020

test: Improve gke/{select,release}-cluster.sh scripts #11173

test: Improve gke/{select,release}-cluster.sh scripts #11173

Conversation

errordeveloper commented Apr 27, 2020 • edited

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

coveralls commented Apr 27, 2020 • edited

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

errordeveloper commented Apr 27, 2020

errordeveloper commented Apr 27, 2020

nebril left a comment

Choose a reason for hiding this comment

nebril left a comment

Choose a reason for hiding this comment

nebril commented Apr 28, 2020

nebril commented Apr 28, 2020

errordeveloper commented Apr 28, 2020

nebril commented Apr 28, 2020

nebril commented Apr 28, 2020

errordeveloper commented Apr 28, 2020 • edited

nebril commented Apr 28, 2020

errordeveloper commented Apr 28, 2020

This comment has been minimized.

aanm left a comment • edited

Choose a reason for hiding this comment

errordeveloper commented Apr 29, 2020 • edited

errordeveloper commented Apr 29, 2020 • edited

errordeveloper commented Apr 29, 2020

errordeveloper commented Apr 29, 2020

nebril commented Apr 29, 2020

errordeveloper commented Apr 29, 2020

test: Improve `gke/{select,release}-cluster.sh` scripts #11173

test: Improve `gke/{select,release}-cluster.sh` scripts #11173

errordeveloper commented Apr 27, 2020 •

edited

coveralls commented Apr 27, 2020 •

edited

errordeveloper commented Apr 28, 2020 •

edited

aanm left a comment •

edited

errordeveloper commented Apr 29, 2020 •

edited

errordeveloper commented Apr 29, 2020 •

edited