E2E: Namespace soak test, confirms deletion of 100 NS happens within 150 seconds #8325
Conversation
jayunit100
changed the title
|
Results |
| totalNS := 100 | ||
| It("Namespaces should be deleted within 100 seconds", func() { | ||
| for n := 0; n < totalNS; n += 1 { | ||
| namespace, err = createTestingNS(fmt.Sprintf("nslifetest-%v", n), c) |
There was a problem hiding this comment.
you should try deleting a namespace with O(100) events in it. We do this for every density test and it takes soo long.
There was a problem hiding this comment.
yup i noticed that. great point !
I figured first i would start w/ just empty namespaces to verify that the cleanup is indeed happening ~ every minute.
Then we can add tests for dense namespaces and so on.
There was a problem hiding this comment.
The deletion of a namespace with O(100) events should improve dramatically when we are ready to merge the following: #7372
|
LGTM - hope to get the related PR in soon as well. The bottleneck on deleting a namespace with a large amount of content is its a network request per piece of content and I assume we are getting into rate limiting issues. The related PR fixes this by making a bulk deletion request to etcd for each resource type. |
E2E: Namespace soak test, confirms deletion of 100 NS happens within 150 seconds
|
This e2e test is failing consistently on our continuous integration system. I'm going to roll this PR back until it's stable. |
ghost
mentioned this pull request
|
The error detail is below. We can consider remerging this PR if #7372 causes it to succeed consistently in our continuous integration environment. ns-soak
• Failure [168.026 seconds] FAILED. There is still a remaining ns e2e-tests-nslifetest-0-7a153eaa-3044-4c12-ab5c-3743b96fcbf6 |
| @@ -0,0 +1,107 @@ | |||
| /* | |||
There was a problem hiding this comment.
For consistency, I'd prefer this to be in test/e2e/namespaces.go
$ ls -1 test/e2e
cadvisor.go
certs.go
density.go
docker_containers.go
downward_api.go
driver.go
empty_dir.go
es_cluster_logging.go
events.go
kubectl.go
kubelet.log
monitoring.go
networking.go
pd.go
pods.go
rc.go
scale.go
secrets.go
service.go
shell.go
ssh.go
util.go
|
hmmm thats funny i wonder why it gets pegged at 4 namespaces ? Can we disable the test (It -> PIt) instead of revert? That way we can still run it on our end. Or is it already reverted... |
Here's a namespace soak test which confirms that 100 created namespaces are all indeed deleted within 150 seconds .
The drain rate is definetly fixed thanks to @derekwaynecarr i beleive.
But in any case this will keep a regression from occuring