qa: stability: check multi launch and rm stability #491

grahamwhaley · 2017-08-29T14:06:49Z

launch a number of containers, and then 'rm -f' them all in one go.
Check we have the right number of 'components' running at each
stage.

Fixes: #490

Signed-off-by: Graham whaley graham.whaley@intel.com

clearcontainersbot · 2017-08-29T14:12:54Z

Popular Images qa-passed 👍

MarioCarrilloA · 2017-08-30T02:34:17Z

integration/stability/soak_parallel_rm.sh

+}
+
+function init() {
+	kill_all_containers


could be good to restart docker/cc-proxy services at this point

Hmm, yea, I guess. It almost seem backwards - make the system 'more stable' to try and 'find instabilities', but for repeatability it makes a lot of sense. Let me see if I can do something by re-using the stuff we have in the metrics lib files (would be good if we could share some of that across metrics and QA for instance).

maybe it is time to add a lib directory in the root of the repository, and start populating it and using it. But this can be another PR.

An idea from @chavafg - we can also check that there are no pending docker mounts left after the test - something else we have spotted as a potential issue (clearcontainers/packaging#110)

MarioCarrilloA · 2017-08-30T02:41:32Z

integration/stability/soak_parallel_rm.sh

+}
+
+function check_all_running() {
+	echo Checking ${how_many} containers have all relevant components


Missing quotes in this echo

strictly not needed, well not unless ${how_many} might expand into a command line argument or something ;-), but sure, good practice to have - I'll go add

MarioCarrilloA · 2017-08-30T02:44:32Z

integration/stability/soak_parallel_rm.sh

+}
+
+function kill_all_containers() {
+	present=$(docker ps -qa | wc -l)


Maybe count_containers() could be used here

oh, good point!

MarioCarrilloA · 2017-08-30T02:56:17Z

integration/stability/soak_parallel_rm.sh

+	how_many_proxys=$(ps --no-header -C ${PROXY_NAME} | wc -l)
+	if (( ${how_many_running} >= 1 )); then
+		# If we have any containers, then we expect to have a single proxy
+		if (( ${how_many_proxys} != 1 )); then


if this tests is designed for runc as a RUNTIME, could be ${how_many_proxys} > 1, this is due to runc does not use cc-proxy or something like that, then how_many_proxys will be 0 and this is an expected result

Indeed. Right now the primary goal is to run with CC, but if we were to support 'other' runtimes, then yes, we need to make the script more sentient.
I'm probably about to code something similar up in another script - I'll see if I get to apply that here as well.

grahamwhaley · 2017-09-20T09:38:19Z

I've rebase and pushed - let's see if we can get the CI to pass and ack/merge this?
We can then open an issue to add the mount-checks.
Oh, and Travis CI will fail as I will have rebased on the PR that just landed that removed the Travis config...

clearcontainersbot · 2017-09-21T16:03:36Z

kubernetes qa-passed 👍

clearcontainersbot · 2017-09-21T20:13:14Z

kubernetes qa-passed 👍

grahamwhaley · 2017-09-21T20:18:12Z

@MarioCarrilloA updated with changes
@devimc added a check for 'dangling mounts'. It is a bit of a simple check - just check we have the same number of mounts in the system after we've deleted all the containers that we had before we launched them. I did that as checking specifically for mounts related to docker and to check after each container launch and delete I believe would need fine knowledge of which graph driver was being used etc. This simple check should be good enough to catch any dangling mounts - it will just catch them at the end of each test cycle rather than at the exact point of failure.
@mcastelino if you are looking at some of the 'parts of the runtime' or 'mount points' left dangling issues then this test might help you reproduce them in a stable manner.

grahamwhaley · 2017-09-21T20:28:41Z

@MarioCarrilloA I should have said - I also made it so you should be able to run with other RUNTIMEs as well (which helped me sanity check the script ;-) )

clearcontainersbot · 2017-09-25T08:57:41Z

kubernetes qa-passed 👍

chavafg · 2017-10-02T13:32:38Z

lgtm

jodh-intel · 2017-10-04T13:03:09Z

integration/stability/soak_parallel_rm.sh

+# The goals are two fold:
+# - spot any stuck or non-started components
+# - catch any hang ups
+


Might be good to have a set -e in here. Note that I checked with that locally, but the script then exited prematurely so I had to change a few of the functions to add || true and also this...

((how_many++))

... had to change to:

how_many=$((how_many + 1))

hmmm. I'm a bit wary of setting -e here - if then docker or something fails because we have a bug then the script would just quit, without a chance of dumping any diagnostics. I think I'd rather have the script find that something 'had not happened', and then give it the chance to dump out the diagnostics.
Let me push an update to the script btw, which checks some other things (that cc-runtime list and /var/lib/virtcontainers/pods both have the correct number of items - and also I might have the 'check number of mounts' in as well - but I need to stare at that a bit)... hold on, I'll do a push - but, be warned, with the runtime as it is at the minute the test will fail on the vc/pods check I believe)

Given what this script is doing, yeah that's a fair point.

launch a number of containers, and then 'rm -f' them all in one go. Check we have the right number of 'components' running at each stage. Fixes: clearcontainers#490 Signed-off-by: Graham whaley <graham.whaley@intel.com>

grahamwhaley · 2017-10-04T13:18:47Z

@jodh-intel I've pushed the new version - it does also check the mount count.

clearcontainersbot · 2017-10-04T13:41:12Z

kubernetes qa-passed 👍

jodh-intel · 2017-10-04T13:50:09Z

integration/stability/soak_parallel_rm.sh

+# The goals are two fold:
+# - spot any stuck or non-started components
+# - catch any hang ups
+


Given what this script is doing, yeah that's a fair point.

devimc · 2017-10-04T14:27:23Z

integration/stability/soak_parallel_rm.sh

+	for ((i=1; i<= ITERATIONS; i++)); do {
+		echo "Start loop $i"
+		#spin them up
+		go


I wonder if this can cause clashes with the go command

heh - fair comment. I suspect there is some bash parse/ordering/precedent rule, but it's going to be easier for me to just rename the func ...

…xists k8s: check cni inteface exists before delete it

grahamwhaley requested review from chavafg and MarioCarrilloA August 29, 2017 14:06

grahamwhaley added the in progress label Aug 29, 2017

grahamwhaley mentioned this pull request Aug 29, 2017

docker rm -f hangs or leaves processes on multi-container remove #492

Closed

MarioCarrilloA reviewed Aug 30, 2017

View reviewed changes

chavafg mentioned this pull request Sep 19, 2017

kernel: fix 9p fs unlink-ftruncate clearcontainers/packaging#110

Merged

grahamwhaley force-pushed the 20170825_soak_parallel_rm branch from bd525e7 to 1fd7a10 Compare September 20, 2017 09:37

grahamwhaley mentioned this pull request Sep 20, 2017

running stress test the runtime is unable to unmount some rootfs clearcontainers/runtime#570

Open

grahamwhaley force-pushed the 20170825_soak_parallel_rm branch from 1fd7a10 to f1aace2 Compare September 21, 2017 19:49

jodh-intel reviewed Oct 4, 2017

View reviewed changes

qa: stability: check multi launch and rm stability

d9efcec

launch a number of containers, and then 'rm -f' them all in one go. Check we have the right number of 'components' running at each stage. Fixes: clearcontainers#490 Signed-off-by: Graham whaley <graham.whaley@intel.com>

grahamwhaley force-pushed the 20170825_soak_parallel_rm branch from f1aace2 to d9efcec Compare October 4, 2017 13:17

jodh-intel approved these changes Oct 4, 2017

View reviewed changes

devimc reviewed Oct 4, 2017

View reviewed changes

grahamwhaley mentioned this pull request Oct 12, 2017

net: netns mount/namespaces 'hang around' after containers stopped/deleted clearcontainers/runtime#716

Open

jodh-intel merged commit 380375f into clearcontainers:master Oct 13, 2017

jodh-intel removed the in progress label Oct 13, 2017

mcastelino pushed a commit to mcastelino/tests that referenced this pull request Jan 23, 2019

Merge pull request clearcontainers#491 from chavafg/topic/check-cni-e…

4469db1

…xists k8s: check cni inteface exists before delete it

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

qa: stability: check multi launch and rm stability #491

qa: stability: check multi launch and rm stability #491

grahamwhaley commented Aug 29, 2017

clearcontainersbot commented Aug 29, 2017

MarioCarrilloA Aug 30, 2017

grahamwhaley Aug 30, 2017

chavafg Sep 7, 2017

grahamwhaley Sep 19, 2017

MarioCarrilloA Aug 30, 2017

grahamwhaley Aug 30, 2017

MarioCarrilloA Aug 30, 2017

grahamwhaley Aug 30, 2017

MarioCarrilloA Aug 30, 2017 •

edited

Loading

grahamwhaley Aug 30, 2017

grahamwhaley commented Sep 20, 2017

clearcontainersbot commented Sep 21, 2017

clearcontainersbot commented Sep 21, 2017

grahamwhaley commented Sep 21, 2017

grahamwhaley commented Sep 21, 2017

clearcontainersbot commented Sep 25, 2017

chavafg commented Oct 2, 2017 •

edited by grahamwhaley

Loading

jodh-intel Oct 4, 2017

grahamwhaley Oct 4, 2017

jodh-intel Oct 4, 2017

grahamwhaley commented Oct 4, 2017

clearcontainersbot commented Oct 4, 2017

jodh-intel Oct 4, 2017

devimc Oct 4, 2017

grahamwhaley Oct 4, 2017

qa: stability: check multi launch and rm stability #491

qa: stability: check multi launch and rm stability #491

Conversation

grahamwhaley commented Aug 29, 2017

clearcontainersbot commented Aug 29, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MarioCarrilloA Aug 30, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grahamwhaley commented Sep 20, 2017

clearcontainersbot commented Sep 21, 2017

clearcontainersbot commented Sep 21, 2017

grahamwhaley commented Sep 21, 2017

grahamwhaley commented Sep 21, 2017

clearcontainersbot commented Sep 25, 2017

chavafg commented Oct 2, 2017 • edited by grahamwhaley Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grahamwhaley commented Oct 4, 2017

clearcontainersbot commented Oct 4, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MarioCarrilloA Aug 30, 2017 •

edited

Loading

chavafg commented Oct 2, 2017 •

edited by grahamwhaley

Loading