Test finalization for CRs #46439

ash2k · 2017-05-25T14:48:00Z

What this PR does / why we need it:
Updates #45511 with a test for finalizers for CRDs.

Release note:

NONE

@deads2k

k8s-ci-robot · 2017-05-25T14:48:08Z

Hi @ash2k. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with @k8s-bot ok to test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

ash2k · 2017-05-25T14:48:17Z

/sig api-machinery

ash2k · 2017-05-26T05:14:31Z

@deads2k @sttts this is ready for review.

deads2k · 2017-05-26T11:48:14Z

staging/src/k8s.io/kube-apiextensions-server/test/integration/finalization_test.go

+
+	uid := createdNoxuInstance.GetUID()
+	err = noxuResourceClient.Delete(name, &metav1.DeleteOptions{
+		Preconditions: &metav1.Preconditions{


Made me think. Do we have a test anywhere that makes sure that this precondition is actually respected? 95% sure it works, but I don't think I've explicitly checked.

I don't know :)

deads2k · 2017-05-26T11:49:47Z

staging/src/k8s.io/kube-apiextensions-server/test/integration/finalization_test.go

+	})
+	require.NoError(t, err)
+
+	gottenNoxuInstance, err := noxuResourceClient.Get(name)


I understand this test, but its an esoteric flow. Can you add comments explaining that deleting something with a finalizer sets deletiontimestamp, but leaves the item until the finalizer list is empty and someone issues another delete?

deads2k · 2017-05-26T11:50:29Z

staging/src/k8s.io/kube-apiextensions-server/test/integration/finalization_test.go

+
+	require.NotNil(t, gottenNoxuInstance.GetDeletionTimestamp())
+
+	gottenNoxuInstance.SetFinalizers(nil)


I think you should be able to issue another delete and have the item remain since the finalizer isn't finished. Can you confirm with a normal resource and add this check if so?

ok, will do.

deads2k · 2017-05-26T11:51:14Z

staging/src/k8s.io/kube-apiextensions-server/test/integration/finalization_test.go

+	})
+	require.NoError(t, err)
+
+	err = wait.PollImmediate(300*time.Millisecond, 1*time.Second, func() (bool, error) {


I'm surprised the final delete wasn't instantaneous. You confirmed that you had to wait? Any idea why?

I haven't confirmed whether it was or was not instantaneous - I'm following my past experience with deletes where a delete of an object does not remove it from api immediately and you can GET it and observe the deletion timestamp set. And then it disappears.

Works without Polling as well. 👍

deads2k · 2017-05-26T11:51:52Z

A few comments, but this looks very good. Glad it works for you :)

ash2k · 2017-05-26T11:56:09Z

Well, it works 50% of the time. It's extremely flaky. Here are some failures:

make test-integration WHAT=./staging/src/k8s.io/kube-apiextensions-server/test/integration/ KUBE_TEST_ARGS="-run TestFinalization"
+++ [0526 21:27:22] Checking etcd is on PATH
/Users/ash2k/gopath/src/k8s.io/kubernetes/third_party/etcd/etcd
+++ [0526 21:27:22] Starting etcd instance
etcd --advertise-client-urls http://127.0.0.1:2379 --data-dir /var/folders/wk/_45zctx51y57kvc6sytpfmm40000gn/T/tmp.Ea7v7ceH --listen-client-urls http://127.0.0.1:2379 --debug > "/dev/null" 2>/dev/null
Waiting for etcd to come up.
+++ [0526 21:27:22] On try 1, etcd: : http://127.0.0.1:2379
{"action":"set","node":{"key":"/_test","value":"","modifiedIndex":4,"createdIndex":4}}
+++ [0526 21:27:22] Running integration test cases
Running tests for APIVersion: v1
+++ [0526 21:27:28] Running tests without code coverage
W0526 21:27:31.997487    6573 authentication.go:50] Authentication is disabled
I0526 21:27:32.001158    6573 serve.go:85] Serving securely on 127.0.0.1:50070
I0526 21:27:32.001222    6573 crd_finalizer.go:249] Starting CRDFinalizer
I0526 21:27:32.001237    6573 customresource_discovery_controller.go:152] Starting DiscoveryController
I0526 21:27:32.001244    6573 naming_controller.go:284] Starting NamingConditionController
2017-05-26T21:27:32.005928336+10:00 AUDIT: id="309d0bef-0f49-4917-9baa-5011ca7497b2" ip="127.0.0.1" method="list" user="<none>" groups="<none>" as="<self>" asgroups="<lookup>" namespace="<none>" uri="/apis/apiextensions.k8s.io/v1alpha1/customresourcedefinitions?resourceVersion=0" response="200"
2017-05-26T21:27:32.006912838+10:00 AUDIT: id="762c4e22-b84f-4861-9db5-a156d9a061c3" ip="127.0.0.1" method="watch" user="<none>" groups="<none>" as="<self>" asgroups="<lookup>" namespace="<none>" uri="/apis/apiextensions.k8s.io/v1alpha1/customresourcedefinitions?resourceVersion=1&timeoutSeconds=582&watch=true" response="<deferred>"
2017-05-26T21:27:32.036872032+10:00 AUDIT: id="ab0703e5-30d5-48f0-b114-1a4147e38bc2" ip="127.0.0.1" method="get" user="<none>" groups="<none>" as="<self>" asgroups="<lookup>" namespace="<none>" uri="/healthz" response="200"
2017-05-26T21:27:32.037569565+10:00 AUDIT: id="5744cc23-af68-4261-9cb1-dc2a4d4e9f4f" ip="127.0.0.1" method="get" user="<none>" groups="<none>" as="<self>" asgroups="<lookup>" namespace="<none>" uri="/apis/apiextensions.k8s.io/v1alpha1" response="200"
{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"apiextensions.k8s.io/v1alpha1","resources":[{"name":"customresourcedefinitions","singularName":"","namespaced":false,"kind":"CustomResourceDefinition","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"customresourcedefinitions/status","singularName":"","namespaced":false,"kind":"CustomResourceDefinition","verbs":["update"]}]}
2017-05-26T21:27:32.039788023+10:00 AUDIT: id="6d17461a-db5d-4303-823b-7ef5c6220092" ip="127.0.0.1" method="create" user="<none>" groups="<none>" as="<self>" asgroups="<lookup>" namespace="<none>" uri="/apis/apiextensions.k8s.io/v1alpha1/customresourcedefinitions" response="201"
2017-05-26T21:27:32.041934955+10:00 AUDIT: id="5f89c562-5344-4443-8215-835ae71488b3" ip="127.0.0.1" method="get" user="<none>" groups="<none>" as="<self>" asgroups="<lookup>" namespace="<none>" uri="/apis/mygroup.example.com/v1alpha1" response="404"
2017-05-26T21:27:32.076868996+10:00 AUDIT: id="121a954c-0d6f-468a-b28d-d0140991fb32" ip="127.0.0.1" method="get" user="<none>" groups="<none>" as="<self>" asgroups="<lookup>" namespace="<none>" uri="/apis/mygroup.example.com/v1alpha1" response="404"
2017-05-26T21:27:32.106796844+10:00 AUDIT: id="a696b059-fade-4bf2-a942-e6364b7ed744" ip="127.0.0.1" method="get" user="<none>" groups="<none>" as="<self>" asgroups="<lookup>" namespace="<none>" uri="/apis/mygroup.example.com/v1alpha1" response="404"
2017-05-26T21:27:32.107245067+10:00 AUDIT: id="7ac9d6ca-8993-41b8-8f4f-f15b2d5363b9" ip="127.0.0.1" method="update" user="<none>" groups="<none>" as="<self>" asgroups="<lookup>" namespace="<none>" uri="/apis/apiextensions.k8s.io/v1alpha1/customresourcedefinitions/noxus.mygroup.example.com/status" response="200"
2017-05-26T21:27:32.132624516+10:00 AUDIT: id="082f87ec-7a26-400a-b705-3041f341e5e0" ip="127.0.0.1" method="get" user="<none>" groups="<none>" as="<self>" asgroups="<lookup>" namespace="<none>" uri="/apis/mygroup.example.com/v1alpha1" response="200"
2017-05-26T21:27:32.133292298+10:00 AUDIT: id="98b3f276-bdb6-47fd-91da-1861db2be094" ip="127.0.0.1" method="create" user="<none>" groups="<none>" as="<self>" asgroups="<lookup>" namespace="<none>" uri="/apis/mygroup.example.com/v1alpha1/noxus" response="201"
2017-05-26T21:27:32.134818471+10:00 AUDIT: id="fa6feea7-1b1a-4c6e-92e4-39aa3c54e903" ip="127.0.0.1" method="delete" user="<none>" groups="<none>" as="<self>" asgroups="<lookup>" namespace="<none>" uri="/apis/mygroup.example.com/v1alpha1/noxus/foo123" response="200"
2017-05-26T21:27:32.139720222+10:00 AUDIT: id="9961b9a9-e86c-4f24-8a38-dfc095e060de" ip="127.0.0.1" method="get" user="<none>" groups="<none>" as="<self>" asgroups="<lookup>" namespace="<none>" uri="/apis/mygroup.example.com/v1alpha1/noxus/foo123" response="200"
2017-05-26T21:27:32.14065171+10:00 AUDIT: id="d408cd4d-e6ea-402f-a8a4-dfa999ac9a06" ip="127.0.0.1" method="update" user="<none>" groups="<none>" as="<self>" asgroups="<lookup>" namespace="<none>" uri="/apis/mygroup.example.com/v1alpha1/noxus/foo123" response="409"
2017-05-26T21:27:32.141324285+10:00 AUDIT: id="e9577d53-6830-4045-8d41-f34bcabfed9d" ip="127.0.0.1" method="get" user="<none>" groups="<none>" as="<self>" asgroups="<lookup>" namespace="<none>" uri="/apis/mygroup.example.com/v1alpha1/noxus/foo123" response="200"
I0526 21:27:32.142014    6573 crd_finalizer.go:261] Shutting down CRDFinalizer
--- FAIL: TestFinalization (0.60s)
        finalization_test.go:45: get object map[string]interface {}{"metadata":map[string]interface {}{"resourceVersion":"5", "uid":"4f094c89-4206-11e7-bb84-80e65026e166", "creationTimestamp":"2017-05-26T11:27:32Z", "deletionTimestamp":"2017-05-26T11:27:32Z", "finalizers":[]interface {}{"noxu.example.com/finalizer"}, "namespace":"", "clusterName":"", "deletionGracePeriodSeconds":0, "name":"foo123", "selfLink":"/apis/mygroup.example.com/v1alpha1/foo123"}, "num":map[string]interface {}{"num1":9223372036854775807, "num2":1000000}, "apiVersion":"mygroup.example.com/v1alpha1", "content":map[string]interface {}{"key":"value"}, "kind":"WishIHadChosenNoxu"}
        Error Trace:    finalization_test.go:51
        Error:          Received unexpected error "the server reported a conflict (put noxus.mygroup.example.com foo123)"
                
        finalization_test.go:54: get object map[string]interface {}{"apiVersion":"mygroup.example.com/v1alpha1", "content":map[string]interface {}{"key":"value"}, "kind":"WishIHadChosenNoxu", "metadata":map[string]interface {}{"finalizers":[]interface {}{"noxu.example.com/finalizer"}, "namespace":"", "resourceVersion":"5", "selfLink":"/apis/mygroup.example.com/v1alpha1/foo123", "name":"foo123", "uid":"4f094c89-4206-11e7-bb84-80e65026e166", "clusterName":"", "creationTimestamp":"2017-05-26T11:27:32Z", "deletionGracePeriodSeconds":0, "deletionTimestamp":"2017-05-26T11:27:32Z"}, "num":map[string]interface {}{"num1":9223372036854775807, "num2":1000000}}
I0526 21:27:32.142029    6573 customresource_discovery_controller.go:163] Shutting down DiscoveryController
I0526 21:27:32.142055    6573 naming_controller.go:295] Shutting down NamingConditionController
FAIL
FAIL    k8s.io/kubernetes/staging/src/k8s.io/kube-apiextensions-server/test/integration 0.678s

make test-integration WHAT=./staging/src/k8s.io/kube-apiextensions-server/test/integration/ KUBE_TEST_ARGS="-run TestFinalization"
+++ [0526 21:43:28] Checking etcd is on PATH
/Users/ash2k/gopath/src/k8s.io/kubernetes/third_party/etcd/etcd
+++ [0526 21:43:28] Starting etcd instance
etcd --advertise-client-urls http://127.0.0.1:2379 --data-dir /var/folders/wk/_45zctx51y57kvc6sytpfmm40000gn/T/tmp.KCVNUY8D --listen-client-urls http://127.0.0.1:2379 --debug > "/dev/null" 2>/dev/null
Waiting for etcd to come up.
+++ [0526 21:43:28] On try 1, etcd: : http://127.0.0.1:2379
{"action":"set","node":{"key":"/_test","value":"","modifiedIndex":4,"createdIndex":4}}
+++ [0526 21:43:28] Running integration test cases
Running tests for APIVersion: v1
+++ [0526 21:43:34] Running tests without code coverage
W0526 21:43:38.181188   14865 authentication.go:50] Authentication is disabled
I0526 21:43:38.183867   14865 serve.go:85] Serving securely on 127.0.0.1:51007
I0526 21:43:38.183931   14865 crd_finalizer.go:249] Starting CRDFinalizer
I0526 21:43:38.183958   14865 naming_controller.go:284] Starting NamingConditionController
I0526 21:43:38.183970   14865 customresource_discovery_controller.go:152] Starting DiscoveryController
2017-05-26T21:43:38.217871492+10:00 AUDIT: id="4307d333-8d3c-4138-8d06-8ae25c94e7aa" ip="127.0.0.1" method="get" user="<none>" groups="<none>" as="<self>" asgroups="<lookup>" namespace="<none>" uri="/healthz" response="200"
2017-05-26T21:43:38.218455383+10:00 AUDIT: id="63023714-83ef-4fee-a4c7-70ebeb2adce8" ip="127.0.0.1" method="get" user="<none>" groups="<none>" as="<self>" asgroups="<lookup>" namespace="<none>" uri="/apis/apiextensions.k8s.io/v1alpha1" response="200"
{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"apiextensions.k8s.io/v1alpha1","resources":[{"name":"customresourcedefinitions","singularName":"","namespaced":false,"kind":"CustomResourceDefinition","verbs":["create","delete","deletecollection","get","list","patch","update","watch"]},{"name":"customresourcedefinitions/status","singularName":"","namespaced":false,"kind":"CustomResourceDefinition","verbs":["update"]}]}
E0526 21:43:38.263184   14865 status.go:62] apiserver received an error that is not an metav1.Status: rpc error: code = 13 desc = transport is closing
2017-05-26T21:43:38.219080519+10:00 AUDIT: id="5235bb05-f440-4b3a-bd93-56efb3b1d75b" ip="127.0.0.1" method="create" user="<none>" groups="<none>" as="<self>" asgroups="<lookup>" namespace="<none>" uri="/apis/apiextensions.k8s.io/v1alpha1/customresourcedefinitions" response="500"
--- FAIL: TestFinalization (0.33s)
        Error Trace:    finalization_test.go:24
        Error:          Received unexpected error "an error on the server (\"unknown\") has prevented the request from succeeding (post customresourcedefinitions.apiextensions.k8s.io)"
                
FAIL
FAIL    k8s.io/kubernetes/staging/src/k8s.io/kube-apiextensions-server/test/integration 0.410s

make test-integration WHAT=./staging/src/k8s.io/kube-apiextensions-server/test/integration/ KUBE_TEST_ARGS="-run TestFinalization"
+++ [0526 21:47:21] Checking etcd is on PATH
/Users/ash2k/gopath/src/k8s.io/kubernetes/third_party/etcd/etcd
+++ [0526 21:47:21] Starting etcd instance
etcd --advertise-client-urls http://127.0.0.1:2379 --data-dir /var/folders/wk/_45zctx51y57kvc6sytpfmm40000gn/T/tmp.H0T4XrEG --listen-client-urls http://127.0.0.1:2379 --debug > "/dev/null" 2>/dev/null
Waiting for etcd to come up.
/Users/ash2k/gopath/src/k8s.io/kubernetes/hack/lib/util.sh: line 33: 15573 Segmentation fault: 11  etcd --advertise-client-urls http://${ETCD_HOST}:${ETCD_PORT} --data-dir ${ETCD_DIR} --listen-client-urls http://${ETCD_HOST}:${ETCD_PORT} --debug 2> "${ETCD_LOGFILE}" > /dev/null
!!! [0526 21:47:43] Timed out waiting for etcd:  to answer at http://127.0.0.1:2379/v2/machines; tried 80 waiting 0.25 between each
!!! [0526 21:47:43] Call tree:
!!! [0526 21:47:43]  1: hack/make-rules/test-integration.sh:67 kube::etcd::start(...)
!!! [0526 21:47:43]  2: hack/make-rules/test-integration.sh:98 runTests(...)
+++ [0526 21:47:43] Cleaning up etcd
+++ [0526 21:47:43] Integration test cleanup complete
make: *** [test-integration] Error 1

This is obviously not acceptable to merge. I don't think it is a test though. Looks like some server-side issues.

ash2k · 2017-05-26T11:58:29Z

The first one is really weird - it is an UPDATE conflict but resource version is the same "5" before the update and after an update failure.

ash2k · 2017-05-26T12:58:09Z

Two new types of failures.
This one is the situation where PollImmediate times out after 3 seconds and the object is still in the api after delete.

2017-05-26T22:45:35.962834704+10:00 AUDIT: id="aca4d8d3-81e7-4d0d-8736-9a61455771ea" ip="127.0.0.1" method="get" user="<none>" groups="<none>" as="<self>" asgroups="<lookup>" namespace="<none>" uri="/apis/mygroup.example.com/v1alpha1" response="404"
--- FAIL: TestFinalization (30.22s)
        Error Trace:    finalization_test.go:24
        Error:          Received unexpected error "timed out waiting for the condition"

This one is especially alarming:

make test-integration WHAT=./staging/src/k8s.io/kube-apiextensions-server/test/integration/ KUBE_TEST_ARGS="-run TestFinalization"
+++ [0526 22:46:34] Checking etcd is on PATH
/Users/ash2k/gopath/src/k8s.io/kubernetes/third_party/etcd/etcd
+++ [0526 22:46:34] Starting etcd instance
panic: sync: inconsistent mutex state

goroutine 1 [running]:
panic(0x950640, 0xc8201da850)
        /usr/local/go/src/runtime/panic.go:481 +0x3e6
sync.(*Mutex).Lock(0xc82018fa00)
        /usr/local/go/src/sync/mutex.go:75 +0x16d
github.com/coreos/etcd/cmd/vendor/github.com/coreos/pkg/capnslog.NewPackageLogger(0xe08240, 0x1c, 0xd351e0, 0x8, 0x0)
        /home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/pkg/capnslog/logmap.go:221 +0x4a
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/proxy/tcpproxy.init()
        /home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/proxy/tcpproxy/userspace.go:27 +0x89
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.init()
        /home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/util.go:30 +0x106
main.init()
        /home/gyuho/go/src/github.com/coreos/etcd/cmd/main.go:29 +0x3b
!!! [0526 22:46:34] Call tree:
!!! [0526 22:46:34]  1: /Users/ash2k/gopath/src/k8s.io/kubernetes/hack/lib/etcd.sh:58 kube::etcd::validate(...)
!!! [0526 22:46:34]  2: hack/make-rules/test-integration.sh:67 kube::etcd::start(...)
!!! [0526 22:46:34]  3: hack/make-rules/test-integration.sh:98 runTests(...)
+++ [0526 22:46:34] Cleaning up etcd
+++ [0526 22:46:34] Integration test cleanup complete
make: *** [test-integration] Error 1

And this is the UPDATE conflict with mode details logged:

--- FAIL: TestFinalization (0.32s)
        finalization_test.go:47: get1 object map[string]interface {}{"num":map[string]interface {}{"num1":9223372036854775807, "num2":1000000}, "apiVersion":"mygroup.example.com/v1alpha1", "content":map[string]interface {}{"key":"value"}, "kind":"WishIHadChosenNoxu", "metadata":map[string]interface {}{"deletionGracePeriodSeconds":0, "namespace":"", "resourceVersion":"5", "selfLink":"/apis/mygroup.example.com/v1alpha1/foo123", "creationTimestamp":"2017-05-26T12:48:14Z", "deletionTimestamp":"2017-05-26T12:48:14Z", "finalizers":[]interface {}{"noxu.example.com/finalizer"}, "name":"foo123", "uid":"95114634-4211-11e7-bec5-80e65026e166", "clusterName":""}}
        Error Trace:    finalization_test.go:63
        Error:          Received unexpected error "the server reported a conflict (put noxus.mygroup.example.com foo123)"
        Messages:       &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:""}, Status:"Failure", Message:"the server reported a conflict (put noxus.mygroup.example.com foo123)", Reason:"Conflict", Details:(*v1.StatusDetails)(0xc4204003c0), Code:409}}
                
        finalization_test.go:64: details &v1.StatusDetails{Name:"foo123", Group:"mygroup.example.com", Kind:"noxus", UID:"", Causes:[]v1.StatusCause{v1.StatusCause{Type:"UnexpectedServerResponse", Message:"unknown", Field:""}}, RetryAfterSeconds:0}
        finalization_test.go:67: get2 object map[string]interface {}{"apiVersion":"mygroup.example.com/v1alpha1", "content":map[string]interface {}{"key":"value"}, "kind":"WishIHadChosenNoxu", "metadata":map[string]interface {}{"clusterName":"", "creationTimestamp":"2017-05-26T12:48:14Z", "finalizers":[]interface {}{"noxu.example.com/finalizer"}, "name":"foo123", "namespace":"", "resourceVersion":"5", "deletionGracePeriodSeconds":0, "deletionTimestamp":"2017-05-26T12:48:14Z", "selfLink":"/apis/mygroup.example.com/v1alpha1/foo123", "uid":"95114634-4211-11e7-bec5-80e65026e166"}, "num":map[string]interface {}{"num1":9223372036854775807, "num2":1000000}}

nikhita · 2017-05-26T13:59:54Z

@deads2k is better equipped to explain those errors but fwiw, the tests are passing for me everytime! :)

deads2k · 2017-05-26T14:19:03Z

staging/src/k8s.io/kube-apiextensions-server/test/integration/finalization_test.go

+	require.NoError(t, err)
+
+	// Check that the object is actually gone.
+	err = wait.PollImmediate(300*time.Millisecond, 3*time.Second, func() (bool, error) {


Remove the poll. It should not flake, but if it does, we'll chase it.

deads2k · 2017-05-26T14:19:15Z

@k8s-bot ok to test

deads2k · 2017-05-26T14:19:37Z

I haven't managed to see it flake. Let's see what CI says.

Fix the poll and squash please.

nikhita · 2017-05-26T17:05:23Z

staging/src/k8s.io/kube-apiextensions-server/test/integration/finalization_test.go

@@ -0,0 +1,73 @@
+package integration


Needs the copyright here. :)

sttts · 2017-05-29T08:28:50Z

@ash2k staging godeps want to be updated.

ash2k · 2017-05-29T10:46:07Z

@sttts hey, thanks for the tip - this is all new to me, I thought the failures were caused by something else, not my pr. How do I do this? I tried running ./hack/update-staging-godeps.sh but it outputs an error and I don't know how to proceed:

./hack/update-staging-godeps.sh 
Checking whether godeps are restored
Updating godeps for k8s.io/apimachinery
Rewriting Godeps.json to remove commits that don't really exist because we haven't pushed the prereqs yet
Updating godeps for k8s.io/apiserver
  godep: Package (github.com/emicklei/go-restful/swagger) not found
!!! [0529 20:42:48] Call tree:
!!! [0529 20:42:48]  1: ./hack/update-staging-godeps.sh:103 updateGodepManifest(...)
!!! Error in ./hack/update-staging-godeps.sh:63
  Error in ./hack/update-staging-godeps.sh:63. '((i<3-1))' exited with status 1 0
Call stack:
  1: ./hack/update-staging-godeps.sh:63 updateGodepManifest(...)
  2: ./hack/update-staging-godeps.sh:103 main(...)
Exiting with status 1

Looks like k8s.io/apiserver wants github.com/emicklei/go-restful/swagger but I haven't touched that directory.

sttts · 2017-05-29T11:10:24Z

hey, thanks for the tip - this is all new to me

No problem :)

Can you try to run hack/godep-restore.sh? Might be that this must be called once before.

ash2k · 2017-05-29T22:37:18Z

So the solution was to follow https://github.com/kubernetes/community/blob/master/contributors/devel/godep.md to the letter. Just using those scripts with my existing gopath didn't work. Magic!

ash2k · 2017-05-29T23:54:43Z

@deads2k please take a look at the CI test failure - it is something I'm getting locally too, so cannot just be my local (incorrect?) setup:

assertions.go:203: 
                        
	Error Trace:	finalization_test.go:75
	
	Error:		Received unexpected error "the server reported a conflict (put noxus.mygroup.example.com foo123)"

sttts · 2017-05-30T07:24:33Z

staging/src/k8s.io/kube-apiextensions-server/test/integration/finalization_test.go

+
+	// Removing the finalizers to allow the following delete remove the object.
+	// This step will fail if previous delete wrongly removed the object.
+	gottenNoxuInstance.SetFinalizers(nil)


You need a retry loop here which does the get-change-update pattern as long as errors.IsConflict(err) is true.

Background: the controllers in the master will update the object. If you use the your old copy in line 74 the apiserver will complain that your version is outdated. Even line 64 will modify the object (it sets the deletion timestamp).

That explains the spurious failures. I didn't know some other controllers will/may modify the object concurrently.
However then the question is why it was passing sometimes if line 64 always modifies it before the update? I tried to troubleshoot this earlier (see last block in comment #46439 (comment)) and found that resource version was the same before and after the failed update operation so I thought there should be no conflict even though the conflict is reported. Weird. Delete does not increment the resource version but is checked on conditional update?
Anyway, I've updated the code. Thanks for the help!

sttts · 2017-05-31T09:03:51Z

/approve
/lgtm

k8s-github-robot · 2017-05-31T09:04:02Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ash2k, sttts

Needs approval from an approver in each of these OWNERS Files:

~~staging/src/k8s.io/kube-apiextensions-server/OWNERS~~ [sttts]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

deads2k · 2017-05-31T14:04:18Z

staging/src/k8s.io/kube-apiextensions-server/test/integration/finalization_test.go

+
+	"k8s.io/apimachinery/pkg/api/errors"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	apiextensionsv1alpha1 "k8s.io/kube-apiextensions-server/pkg/apis/apiextensions/v1alpha1"


package moved during promotion to beta.

deads2k · 2017-05-31T14:04:26Z

@k8s-bot test this

Updates kubernetes#45511

sttts · 2017-06-01T07:08:14Z

@k8s-bot pull-kubernetes-e2e-gce-etcd3 test this

k8s-github-robot · 2017-06-01T12:43:40Z

Automatic merge from submit-queue (batch tested with PRs 43505, 45168, 46439, 46677, 46623)

ash2k · 2021-04-23T05:37:36Z

/kind cleanup

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 25, 2017

k8s-github-robot assigned sttts May 25, 2017

k8s-ci-robot added the sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. label May 25, 2017

k8s-github-robot assigned deads2k May 25, 2017

k8s-github-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. release-note-none Denotes a PR that doesn't merit a release note. labels May 25, 2017

ash2k force-pushed the crd-finalization-test branch 3 times, most recently from b2fe82b to 1a3eddc Compare May 26, 2017 05:12

ash2k changed the title ~~WIP: Test finalization for CRD~~ Test finalization for CRD May 26, 2017

ash2k changed the title ~~Test finalization for CRD~~ WIP: Test finalization for CRD May 26, 2017

deads2k reviewed May 26, 2017

View reviewed changes

k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label May 26, 2017

ash2k force-pushed the crd-finalization-test branch from e15f7f9 to 60cd40b Compare May 26, 2017 14:48

nikhita reviewed May 26, 2017

View reviewed changes

ash2k force-pushed the crd-finalization-test branch from 58f70c0 to 50bbb1c Compare May 27, 2017 02:20

sttts changed the title ~~Test finalization for CRD~~ Test finalization for CRs May 29, 2017

ash2k force-pushed the crd-finalization-test branch from 50bbb1c to b5a9ce5 Compare May 29, 2017 09:44

ash2k force-pushed the crd-finalization-test branch from b5a9ce5 to 4cad36f Compare May 29, 2017 22:31

k8s-github-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels May 29, 2017

sttts reviewed May 30, 2017

View reviewed changes

ash2k force-pushed the crd-finalization-test branch from 4cad36f to 848147f Compare May 30, 2017 11:38

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 31, 2017

k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 31, 2017

deads2k reviewed May 31, 2017

View reviewed changes

Test finalization of CRD

5ac9d86

Updates kubernetes#45511

ash2k force-pushed the crd-finalization-test branch from 848147f to 5ac9d86 Compare May 31, 2017 23:18

k8s-github-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 1, 2017

sttts added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 1, 2017

k8s-github-robot merged commit e5cdefa into kubernetes:master Jun 1, 2017

ash2k deleted the crd-finalization-test branch June 28, 2017 00:13

ash2k mentioned this pull request Jun 28, 2017

Fix Unstructured field accessor #48065

Merged

k8s-ci-robot added the kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. label Apr 23, 2021


		require.NotNil(t, gottenNoxuInstance.GetDeletionTimestamp())

		gottenNoxuInstance.SetFinalizers(nil)

Test finalization for CRs #46439

Test finalization for CRs #46439

Conversation

ash2k commented May 25, 2017

k8s-ci-robot commented May 25, 2017

ash2k commented May 25, 2017

ash2k commented May 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ash2k May 26, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deads2k commented May 26, 2017

ash2k commented May 26, 2017 • edited

ash2k commented May 26, 2017

ash2k commented May 26, 2017

nikhita commented May 26, 2017

Choose a reason for hiding this comment

deads2k commented May 26, 2017

deads2k commented May 26, 2017

Choose a reason for hiding this comment

sttts commented May 29, 2017

ash2k commented May 29, 2017

sttts commented May 29, 2017

ash2k commented May 29, 2017

ash2k commented May 29, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sttts commented May 31, 2017

k8s-github-robot commented May 31, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deads2k commented May 31, 2017

sttts commented Jun 1, 2017

k8s-github-robot commented Jun 1, 2017

ash2k commented Apr 23, 2021

ash2k May 26, 2017 •

edited

ash2k commented May 26, 2017 •

edited