E2E Test Refactoring #437

whitleykeith · 2020-10-14T20:40:33Z

@dry923 @rsevilla87 @chaitanyaenr

python-based e2e testing framework. I wanted to go ahead and open a draft PR so you guys can give feedback/roast me 🔥

Changes:

New directory e2e which contains ported test benchmarks as well as initial framework to run the tests
auto-discovery of new test cases for pre-existing workloads
logic to run a benchmark, wait for completion, and return benchmark results
logic to run all benchmarks for a given workload type and return aggregated results
readable logs :)

Limitations / TODOs:

Not integrated with the check_es.py script yet, so no checks against elastic
Wait logic for benchmarks needs to be parameterized. All runs are hardcoded with a timeout of 500 seconds
Need to add support for custom setup/teardown functions
Need to add generic setup/teardown functions
Need to add logic to inject runtime variables like ES Server
Need to mark/parameterize tests
Need to add wrapper script to check git-diffs and run the correct workload tests
Full logging support instead of print statements, would like to have debug/info log levels and output related pod logs on failures

dry923 · 2020-10-30T19:24:46Z

/rerun all

comet-perf-ci · 2020-10-30T20:22:14Z

Test Report

Report generated on 30-Oct-2020 at 20:21:15 by pytest-md 📝

Summary

26 tests ran in 2385.16 seconds ⏱

26 passed 😃

26 passed 😃

e2e/tests/test_byowl.py

TestByowl.test_byowl[base] 48.16s ⏱

e2e/tests/test_backpack.py

Testbackpack.test_backpack[init] 96.38s ⏱

e2e/tests/test_backpack_ds.py

TestBackpackDaemonSet.test_backpack_daemonset[daemonset] 78.33s ⏱

e2e/tests/test_hammerdb.py

TestHammerDB.test_hammerdb[base] 234.66s ⏱

e2e/tests/test_fs_drift.py

TestFSDrift.test_fs_drift[hostpath] 114.42s ⏱

TestFSDrift.test_fs_drift[base] 111.39s ⏱

e2e/tests/test_fiod.py

TestFiod.test_fiod[bsrange] 288.89s ⏱

TestFiod.test_fiod[hostpath] 568.69s ⏱

TestFiod.test_fiod[base] 313.60s ⏱

e2e/tests/test_iperf3.py

TestIPerf3.test_iperf3[hostnetwork] 204.65s ⏱

TestIPerf3.test_iperf3[base] 216.68s ⏱

e2e/tests/test_kube_burner.py

TestKubeBurner.test_kube_burner[kubelet-density-metics] 255.76s ⏱

TestKubeBurner.test_kube_burner[kubelet-density-heavy-metrics] 277.04s ⏱

TestKubeBurner.test_kube_burner[cluster-density-metrics-aggregated] 322.48s ⏱

e2e/tests/test_pgbench.py

TestPgBench.test_pgbench[base] 234.70s ⏱

e2e/tests/test_sysbench.py

TestSysbench.test_sysbench[sysbench] 99.45s ⏱

e2e/tests/test_vegeta.py

TestVegeta.test_vegeta[hostnetwork] 129.88s ⏱

TestVegeta.test_vegeta[base] 108.35s ⏱

e2e/tests/test_smallfile.py

TestSmallFile.test_smallfile[hostpath] 120.43s ⏱

TestSmallFile.test_smallfile[base] 135.50s ⏱

e2e/tests/test_ycsb.py

TestYcsb.test_ycsb[base] 255.71s ⏱

e2e/tests/test_scale_openshift.py

TestScale.test_scale[down] 81.27s ⏱

TestScale.test_scale[up] 352.00s ⏱

e2e/tests/test_uperf.py

TestUperf.test_uperf[uperf_serviceip] 409.16s ⏱

TestUperf.test_uperf[base] 397.98s ⏱

TestUperf.test_uperf[uperf_resources] 309.90s ⏱

whitleykeith · 2020-10-30T20:29:11Z

@dry923 this is how a failure would report:

Test Report

Report generated on 30-Oct-2020 at 19:38:43 by pytest-md 📝

Summary

26 tests ran in 2527.54 seconds ⏱

1 failed 😰
25 passed 😃

1 failed 😰

e2e/tests/test_fiod.py

TestFiod.test_fiod[hostpath] 601.98s ⏱

[gw2] linux -- Python 3.8.5 /home/fedora/git/whitleykeith/benchmark-operator/venv/bin/python

self = <test_fiod.TestFiod object at 0x7ff18d2aa880>
run = <models.benchmark_run.BenchmarkRun object at 0x7ff18d4bc940>

    def test_fiod(self, run):
>       self.run_and_check_benchmark(run)

e2e/tests/test_fiod.py:42: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
e2e/models/test_base.py:92: in run_and_check_benchmark
    run.start(desired_state=desired_running_state, default_timeout=default_timeout)
e2e/models/benchmark_run.py:22: in start
    self.metadata = self.cluster.create_benchmark(self.yaml, desired_state, default_timeout=default_timeout)
e2e/util/k8s.py:126: in create_benchmark
    self.wait_for_benchmark(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <util.k8s.Cluster object at 0x7ff18d4bc880>, name = 'fiod-hostpath', namespace = 'my-ripsaw'
desired_state = 'Running', default_timeout = 600

    def wait_for_benchmark(self, name, namespace, desired_state="Completed", default_timeout=300):
        waiting_for_benchmark = True
        logging.info(f"Waiting for state: {desired_state}")
        timeout_interval = 0
        while waiting_for_benchmark:
            if timeout_interval >= default_timeout:
>               raise BenchmarkTimeoutError(name)
E               util.exceptions.BenchmarkTimeoutError: The benchmark fiod-hostpath timed out

e2e/util/k8s.py:97: BenchmarkTimeoutError

comet-perf-ci · 2020-10-30T21:45:01Z

Test Report

Report generated on 30-Oct-2020 at 21:44:11 by pytest-md 📝

Summary

28 tests ran in 4166.64 seconds ⏱

2 failed 😰
25 passed 😃
1 skipped 🙄

2 failed 😰

e2e/tests/test_service_mesh.py

TestServiceMesh.test_servicemesh[servicemesh] 601.57s ⏱

[gw0] linux -- Python 3.8.5 /var/lib/jenkins/workspace/ec2_jjb/venv/bin/python3

self = <test_service_mesh.TestServiceMesh object at 0x7fb66437d2e0>
run = <models.benchmark_run.BenchmarkRun object at 0x7fb6644003a0>

    def test_servicemesh(self, run):
>       self.run_and_check_benchmark(run)

e2e/tests/test_service_mesh.py:12: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
e2e/models/test_base.py:92: in run_and_check_benchmark
    run.start(desired_state=desired_running_state, default_timeout=default_timeout)
e2e/models/benchmark_run.py:22: in start
    self.metadata = self.cluster.create_benchmark(self.yaml, desired_state, default_timeout=default_timeout)
e2e/util/k8s.py:126: in create_benchmark
    self.wait_for_benchmark(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <util.k8s.Cluster object at 0x7fb664418b20>, name = 'servicemesh'
namespace = 'my-ripsaw', desired_state = 'Running', default_timeout = 600

    def wait_for_benchmark(self, name, namespace, desired_state="Completed", default_timeout=300):
        waiting_for_benchmark = True
        logging.info(f"Waiting for state: {desired_state}")
        timeout_interval = 0
        while waiting_for_benchmark:
            if timeout_interval >= default_timeout:
>               raise BenchmarkTimeoutError(name)
E               util.exceptions.BenchmarkTimeoutError: The benchmark servicemesh timed out

e2e/util/k8s.py:97: BenchmarkTimeoutError

e2e/tests/test_stressng.py

TestStressng.test_stressng[base] 631.75s ⏱

[gw2] linux -- Python 3.8.5 /var/lib/jenkins/workspace/ec2_jjb/venv/bin/python3

self = <test_stressng.TestStressng object at 0x7fa8e9536820>
run = <models.benchmark_run.BenchmarkRun object at 0x7fa8e95f32e0>
desired_running_state = 'Benchmark running', desired_complete_state = 'Complete'
default_timeout = 600

    def run_and_check_benchmark(self, run, desired_running_state="Running", desired_complete_state="Complete", default_timeout=default_timeout):
        run.start(desired_state=desired_running_state, default_timeout=default_timeout)
        try:
>           results = run.wait(desired_state=desired_complete_state, default_timeout=default_timeout)

e2e/models/test_base.py:94: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <models.benchmark_run.BenchmarkRun object at 0x7fa8e95f32e0>
desired_state = 'Complete', default_timeout = 600

    def wait(self, desired_state="Complete", default_timeout=300):
        if self.metadata == {}:
            raise BenchmarkNotStartedError(self.name)
>       self.cluster.wait_for_benchmark(self.resource_name, self.resource_namespace, desired_state=desired_state, default_timeout=default_timeout)

e2e/models/benchmark_run.py:28: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <util.k8s.Cluster object at 0x7fa8e96554f0>, name = 'stressng'
namespace = 'my-ripsaw', desired_state = 'Complete', default_timeout = 600

    def wait_for_benchmark(self, name, namespace, desired_state="Completed", default_timeout=300):
        waiting_for_benchmark = True
        logging.info(f"Waiting for state: {desired_state}")
        timeout_interval = 0
        while waiting_for_benchmark:
            if timeout_interval >= default_timeout:
>               raise BenchmarkTimeoutError(name)
E               util.exceptions.BenchmarkTimeoutError: The benchmark stressng timed out

e2e/util/k8s.py:97: BenchmarkTimeoutError

During handling of the above exception, another exception occurred:

self = <test_stressng.TestStressng object at 0x7fa8e9536820>
run = <models.benchmark_run.BenchmarkRun object at 0x7fa8e95f32e0>

    def test_stressng(self, run):
>       self.run_and_check_benchmark(run, desired_running_state="Benchmark running")

e2e/tests/test_stressng.py:12: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_stressng.TestStressng object at 0x7fa8e9536820>
run = <models.benchmark_run.BenchmarkRun object at 0x7fa8e95f32e0>
desired_running_state = 'Benchmark running', desired_complete_state = 'Complete'
default_timeout = 600

    def run_and_check_benchmark(self, run, desired_running_state="Running", desired_complete_state="Complete", default_timeout=default_timeout):
        run.start(desired_state=desired_running_state, default_timeout=default_timeout)
        try:
            results = run.wait(desired_state=desired_complete_state, default_timeout=default_timeout)
            assert results.get('status', "") == desired_complete_state
            self.check_metadata_collection(results['uuid'])
        except (BenchmarkFailedError, BenchmarkTimeoutError) as err:
>           assert False
E           assert False

e2e/models/test_base.py:98: AssertionError

25 passed 😃

e2e/tests/test_backpack_ds.py

TestBackpackDaemonSet.test_backpack_daemonset[daemonset] 121.54s ⏱

e2e/tests/test_fs_drift.py

TestFSDrift.test_fs_drift[hostpath] 129.49s ⏱

TestFSDrift.test_fs_drift[base] 93.42s ⏱

e2e/tests/test_backpack.py

Testbackpack.test_backpack[init] 147.51s ⏱

e2e/tests/test_byowl.py

TestByowl.test_byowl[base] 285.78s ⏱

e2e/tests/test_fiod.py

TestFiod.test_fiod[bsrange] 246.71s ⏱

TestFiod.test_fiod[hostpath] 592.63s ⏱

TestFiod.test_fiod[base] 274.33s ⏱

e2e/tests/test_iperf3.py

TestIPerf3.test_iperf3[hostnetwork] 183.55s ⏱

TestIPerf3.test_iperf3[base] 231.68s ⏱

e2e/tests/test_hammerdb.py

TestHammerDB.test_hammerdb[base] 276.85s ⏱

e2e/tests/test_kube_burner.py

TestKubeBurner.test_kube_burner[kubelet-density-metics] 231.68s ⏱

TestKubeBurner.test_kube_burner[kubelet-density-heavy-metrics] 309.93s ⏱

TestKubeBurner.test_kube_burner[cluster-density-metrics-aggregated] 325.10s ⏱

e2e/tests/test_pgbench.py

TestPgBench.test_pgbench[base] 243.73s ⏱

e2e/tests/test_smallfile.py

TestSmallFile.test_smallfile[hostpath] 135.41s ⏱

TestSmallFile.test_smallfile[base] 159.78s ⏱

e2e/tests/test_sysbench.py

TestSysbench.test_sysbench[sysbench] 87.27s ⏱

e2e/tests/test_vegeta.py

TestVegeta.test_vegeta[hostnetwork] 111.37s ⏱

TestVegeta.test_vegeta[base] 108.51s ⏱

e2e/tests/test_scale_openshift.py

TestScale.test_scale[up] 207.62s ⏱

e2e/tests/test_uperf.py

TestUperf.test_uperf[uperf_serviceip] 282.91s ⏱

TestUperf.test_uperf[base] 409.81s ⏱

TestUperf.test_uperf[uperf_resources] 267.81s ⏱

e2e/tests/test_ycsb.py

TestYcsb.test_ycsb[base] 135.50s ⏱

1 skipped 🙄

e2e/tests/test_scale_openshift.py

TestScale.test_scale[down] 0.00s ⏱

comet-perf-ci · 2020-11-02T14:14:34Z

!!! Couldn't read commit file !!!

comet-perf-ci · 2020-11-02T15:05:14Z

Test Report

Report generated on 02-Nov-2020 at 15:04:17 by pytest-md 📝

Summary

28 tests ran in 2800.21 seconds ⏱

26 passed 😃
2 skipped 🙄

26 passed 😃

e2e/tests/test_byowl.py

TestByowl.test_byowl[base] 48.15s ⏱

e2e/tests/test_backpack.py

Testbackpack.test_backpack[init] 72.32s ⏱

e2e/tests/test_backpack_ds.py

TestBackpackDaemonSet.test_backpack_daemonset[daemonset] 81.38s ⏱

e2e/tests/test_hammerdb.py

TestHammerDB.test_hammerdb[base] 213.68s ⏱

e2e/tests/test_fs_drift.py

TestFSDrift.test_fs_drift[hostpath] 117.37s ⏱

TestFSDrift.test_fs_drift[base] 120.37s ⏱

e2e/tests/test_fiod.py

TestFiod.test_fiod[bsrange] 267.82s ⏱

TestFiod.test_fiod[hostpath] 532.65s ⏱

TestFiod.test_fiod[base] 288.87s ⏱

e2e/tests/test_iperf3.py

TestIPerf3.test_iperf3[hostnetwork] 192.56s ⏱

TestIPerf3.test_iperf3[base] 222.62s ⏱

e2e/tests/test_kube_burner.py

TestKubeBurner.test_kube_burner[kubelet-density-metics] 261.81s ⏱

TestKubeBurner.test_kube_burner[kubelet-density-heavy-metrics] 264.79s ⏱

TestKubeBurner.test_kube_burner[cluster-density-metrics-aggregated] 316.07s ⏱

e2e/tests/test_pgbench.py

TestPgBench.test_pgbench[base] 225.64s ⏱

e2e/tests/test_smallfile.py

TestSmallFile.test_smallfile[hostpath] 132.46s ⏱

TestSmallFile.test_smallfile[base] 126.76s ⏱

e2e/tests/test_sysbench.py

TestSysbench.test_sysbench[sysbench] 84.25s ⏱

e2e/tests/test_vegeta.py

TestVegeta.test_vegeta[hostnetwork] 90.31s ⏱

TestVegeta.test_vegeta[base] 87.29s ⏱

e2e/tests/test_ycsb.py

TestYcsb.test_ycsb[base] 177.53s ⏱

e2e/tests/test_scale_openshift.py

TestScale.test_scale[up] 376.00s ⏱

TestScale.test_scale[down] 240.71s ⏱

e2e/tests/test_uperf.py

TestUperf.test_uperf[uperf_serviceip] 376.08s ⏱

TestUperf.test_uperf[base] 370.09s ⏱

TestUperf.test_uperf[uperf_resources] 270.85s ⏱

2 skipped 🙄

e2e/tests/test_service_mesh.py

TestServiceMesh.test_servicemesh[servicemesh] 0.00s ⏱

e2e/tests/test_stressng.py

TestStressng.test_stressng[base] 0.00s ⏱

.gitignore

jtaleric · 2020-11-03T17:23:16Z

e2e/benchmarks/byowl/base.yaml

  metadata:
-    collection: true
+    collection: false


why have the base default to no ES or meta collection?

Because if you run the tests without the proper ES environment variables set the tests will fail with metadata collection on.

So essentially the default is no metadata collection and the testing framework will turn it on if the appropriate ES variables are given. That way the tests work with and without ES

jtaleric · 2020-11-03T17:23:32Z

e2e/benchmarks/fiod/base.yaml

  namespace: my-ripsaw
 spec:
  elasticsearch:
-    server: ES_SERVER
-    port: ES_PORT
+    server: localhost


This base is different?

comet-perf-ci · 2020-11-03T20:16:25Z

Test Report

Report generated on 03-Nov-2020 at 20:15:26 by pytest-md 📝

Summary

28 tests ran in 4244.56 seconds ⏱

4 failed 😰
22 passed 😃
2 skipped 🙄

4 failed 😰

e2e/tests/test_byowl.py

TestByowl.test_byowl[base] 601.63s ⏱

[gw2] linux -- Python 3.8.5 /var/lib/jenkins/workspace/ec2_jjb/venv/bin/python3

self = <test_byowl.TestByowl object at 0x7f02593b2370>
run = <models.benchmark_run.BenchmarkRun object at 0x7f025949c850>

    def test_byowl(self, run):
>       self.run_and_check_benchmark(run)

e2e/tests/test_byowl.py:12: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
e2e/models/test_base.py:92: in run_and_check_benchmark
    run.start(desired_state=desired_running_state, default_timeout=default_timeout)
e2e/models/benchmark_run.py:22: in start
    self.metadata = self.cluster.create_benchmark(self.yaml, desired_state, default_timeout=default_timeout)
e2e/util/k8s.py:126: in create_benchmark
    self.wait_for_benchmark(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <util.k8s.Cluster object at 0x7f0259528c10>, name = 'byowl-benchmark'
namespace = 'my-ripsaw', desired_state = 'Running', default_timeout = 600

    def wait_for_benchmark(self, name, namespace, desired_state="Completed", default_timeout=300):
        waiting_for_benchmark = True
        logging.info(f"Waiting for state: {desired_state}")
        timeout_interval = 0
        while waiting_for_benchmark:
            if timeout_interval >= default_timeout:
>               raise BenchmarkTimeoutError(name)
E               util.exceptions.BenchmarkTimeoutError: The benchmark byowl-benchmark timed out

e2e/util/k8s.py:97: BenchmarkTimeoutError

e2e/tests/test_iperf3.py

TestIPerf3.test_iperf3[hostnetwork] 601.74s ⏱

[gw1] linux -- Python 3.8.5 /var/lib/jenkins/workspace/ec2_jjb/venv/bin/python3

self = <test_iperf3.TestIPerf3 object at 0x7fe988e64850>
run = <models.benchmark_run.BenchmarkRun object at 0x7fe988f642b0>

    def test_iperf3(self, run):
>       self.run_and_check_benchmark(run)

e2e/tests/test_iperf3.py:12: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
e2e/models/test_base.py:92: in run_and_check_benchmark
    run.start(desired_state=desired_running_state, default_timeout=default_timeout)
e2e/models/benchmark_run.py:22: in start
    self.metadata = self.cluster.create_benchmark(self.yaml, desired_state, default_timeout=default_timeout)
e2e/util/k8s.py:126: in create_benchmark
    self.wait_for_benchmark(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <util.k8s.Cluster object at 0x7fe989045370>, name = 'iperf-hostnetwork'
namespace = 'my-ripsaw', desired_state = 'Running', default_timeout = 600

    def wait_for_benchmark(self, name, namespace, desired_state="Completed", default_timeout=300):
        waiting_for_benchmark = True
        logging.info(f"Waiting for state: {desired_state}")
        timeout_interval = 0
        while waiting_for_benchmark:
            if timeout_interval >= default_timeout:
>               raise BenchmarkTimeoutError(name)
E               util.exceptions.BenchmarkTimeoutError: The benchmark iperf-hostnetwork timed out

e2e/util/k8s.py:97: BenchmarkTimeoutError

TestIPerf3.test_iperf3[base] 601.59s ⏱

[gw1] linux -- Python 3.8.5 /var/lib/jenkins/workspace/ec2_jjb/venv/bin/python3

self = <test_iperf3.TestIPerf3 object at 0x7fe988fa1640>
run = <models.benchmark_run.BenchmarkRun object at 0x7fe988f31fa0>

    def test_iperf3(self, run):
>       self.run_and_check_benchmark(run)

e2e/tests/test_iperf3.py:12: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
e2e/models/test_base.py:92: in run_and_check_benchmark
    run.start(desired_state=desired_running_state, default_timeout=default_timeout)
e2e/models/benchmark_run.py:22: in start
    self.metadata = self.cluster.create_benchmark(self.yaml, desired_state, default_timeout=default_timeout)
e2e/util/k8s.py:126: in create_benchmark
    self.wait_for_benchmark(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <util.k8s.Cluster object at 0x7fe989045370>, name = 'iperf-base'
namespace = 'my-ripsaw', desired_state = 'Running', default_timeout = 600

    def wait_for_benchmark(self, name, namespace, desired_state="Completed", default_timeout=300):
        waiting_for_benchmark = True
        logging.info(f"Waiting for state: {desired_state}")
        timeout_interval = 0
        while waiting_for_benchmark:
            if timeout_interval >= default_timeout:
>               raise BenchmarkTimeoutError(name)
E               util.exceptions.BenchmarkTimeoutError: The benchmark iperf-base timed out

e2e/util/k8s.py:97: BenchmarkTimeoutError

e2e/tests/test_sysbench.py

TestSysbench.test_sysbench[sysbench] 601.64s ⏱

[gw2] linux -- Python 3.8.5 /var/lib/jenkins/workspace/ec2_jjb/venv/bin/python3

self = <test_sysbench.TestSysbench object at 0x7f0259346310>
run = <models.benchmark_run.BenchmarkRun object at 0x7f02593d34c0>

    def test_sysbench(self, run):
>       self.run_and_check_benchmark(run)

e2e/tests/test_sysbench.py:12: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
e2e/models/test_base.py:92: in run_and_check_benchmark
    run.start(desired_state=desired_running_state, default_timeout=default_timeout)
e2e/models/benchmark_run.py:22: in start
    self.metadata = self.cluster.create_benchmark(self.yaml, desired_state, default_timeout=default_timeout)
e2e/util/k8s.py:126: in create_benchmark
    self.wait_for_benchmark(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <util.k8s.Cluster object at 0x7f025934f700>, name = 'sysbench'
namespace = 'my-ripsaw', desired_state = 'Running', default_timeout = 600

    def wait_for_benchmark(self, name, namespace, desired_state="Completed", default_timeout=300):
        waiting_for_benchmark = True
        logging.info(f"Waiting for state: {desired_state}")
        timeout_interval = 0
        while waiting_for_benchmark:
            if timeout_interval >= default_timeout:
>               raise BenchmarkTimeoutError(name)
E               util.exceptions.BenchmarkTimeoutError: The benchmark sysbench timed out

e2e/util/k8s.py:97: BenchmarkTimeoutError

22 passed 😃

e2e/tests/test_backpack.py

Testbackpack.test_backpack[init] 75.34s ⏱

e2e/tests/test_backpack_ds.py

TestBackpackDaemonSet.test_backpack_daemonset[daemonset] 90.39s ⏱

e2e/tests/test_fs_drift.py

TestFSDrift.test_fs_drift[hostpath] 141.46s ⏱

TestFSDrift.test_fs_drift[base] 99.32s ⏱

e2e/tests/test_fiod.py

TestFiod.test_fiod[bsrange] 270.79s ⏱

TestFiod.test_fiod[hostpath] 514.48s ⏱

TestFiod.test_fiod[base] 243.77s ⏱

e2e/tests/test_kube_burner.py

TestKubeBurner.test_kube_burner[kubelet-density-metics] 183.61s ⏱

TestKubeBurner.test_kube_burner[kubelet-density-heavy-metrics] 246.83s ⏱

TestKubeBurner.test_kube_burner[cluster-density-metrics-aggregated] 235.25s ⏱

e2e/tests/test_smallfile.py

TestSmallFile.test_smallfile[hostpath] 108.41s ⏱

TestSmallFile.test_smallfile[base] 135.46s ⏱

e2e/tests/test_hammerdb.py

TestHammerDB.test_hammerdb[base] 253.04s ⏱

e2e/tests/test_uperf.py

TestUperf.test_uperf[uperf_serviceip] 388.09s ⏱

TestUperf.test_uperf[base] 421.89s ⏱

TestUperf.test_uperf[uperf_resources] 354.97s ⏱

e2e/tests/test_scale_openshift.py

TestScale.test_scale[up] 321.87s ⏱

TestScale.test_scale[down] 213.66s ⏱

e2e/tests/test_vegeta.py

TestVegeta.test_vegeta[hostnetwork] 78.33s ⏱

TestVegeta.test_vegeta[base] 66.36s ⏱

e2e/tests/test_pgbench.py

TestPgBench.test_pgbench[base] 99.39s ⏱

e2e/tests/test_ycsb.py

TestYcsb.test_ycsb[base] 108.37s ⏱

2 skipped 🙄

e2e/tests/test_service_mesh.py

TestServiceMesh.test_servicemesh[servicemesh] 0.00s ⏱

e2e/tests/test_stressng.py

TestStressng.test_stressng[base] 0.00s ⏱

comet-perf-ci · 2020-11-03T21:29:21Z

Test Report

Report generated on 03-Nov-2020 at 21:28:33 by pytest-md 📝

Summary

28 tests ran in 2332.75 seconds ⏱

26 passed 😃
2 skipped 🙄

26 passed 😃

e2e/tests/test_byowl.py

TestByowl.test_byowl[base] 54.16s ⏱

e2e/tests/test_backpack_ds.py

TestBackpackDaemonSet.test_backpack_daemonset[daemonset] 84.35s ⏱

e2e/tests/test_backpack.py

Testbackpack.test_backpack[init] 75.31s ⏱

e2e/tests/test_fs_drift.py

TestFSDrift.test_fs_drift[hostpath] 144.45s ⏱

TestFSDrift.test_fs_drift[base] 114.38s ⏱

e2e/tests/test_hammerdb.py

TestHammerDB.test_hammerdb[base] 234.67s ⏱

e2e/tests/test_fiod.py

TestFiod.test_fiod[bsrange] 276.83s ⏱

TestFiod.test_fiod[hostpath] 589.67s ⏱

TestFiod.test_fiod[base] 300.99s ⏱

e2e/tests/test_iperf3.py

TestIPerf3.test_iperf3[hostnetwork] 183.54s ⏱

TestIPerf3.test_iperf3[base] 228.66s ⏱

e2e/tests/test_kube_burner.py

TestKubeBurner.test_kube_burner[kubelet-density-metics] 225.70s ⏱

TestKubeBurner.test_kube_burner[kubelet-density-heavy-metrics] 321.98s ⏱

TestKubeBurner.test_kube_burner[cluster-density-metrics-aggregated] 337.16s ⏱

e2e/tests/test_pgbench.py

TestPgBench.test_pgbench[base] 243.73s ⏱

e2e/tests/test_smallfile.py

TestSmallFile.test_smallfile[hostpath] 138.49s ⏱

TestSmallFile.test_smallfile[base] 117.38s ⏱

e2e/tests/test_sysbench.py

TestSysbench.test_sysbench[sysbench] 90.26s ⏱

e2e/tests/test_vegeta.py

TestVegeta.test_vegeta[hostnetwork] 126.43s ⏱

TestVegeta.test_vegeta[base] 96.37s ⏱

e2e/tests/test_ycsb.py

TestYcsb.test_ycsb[base] 192.56s ⏱

e2e/tests/test_uperf.py

TestUperf.test_uperf[uperf_serviceip] 409.20s ⏱

TestUperf.test_uperf[base] 397.61s ⏱

TestUperf.test_uperf[uperf_resources] 297.89s ⏱

e2e/tests/test_scale_openshift.py

TestScale.test_scale[up] 361.67s ⏱

TestScale.test_scale[down] 261.73s ⏱

2 skipped 🙄

e2e/tests/test_stressng.py

TestStressng.test_stressng[base] 0.00s ⏱

e2e/tests/test_service_mesh.py

TestServiceMesh.test_servicemesh[servicemesh] 0.00s ⏱

whitleykeith · 2020-11-12T15:31:20Z

@dry923 @chaitanyaenr can you guys add yourself as reviewers to this?

comet-perf-ci · 2020-11-19T20:17:19Z

Test Report

Report generated on 19-Nov-2020 at 20:16:51 by pytest-md 📝

Summary

29 tests ran in 1087.79 seconds ⏱

13 failed 😰
13 passed 😃
3 skipped 🙄

13 failed 😰

e2e/tests/test_smallfile.py

TestSmallFile.test_smallfile[hostpath] 0.00s ⏱

[gw1] linux -- Python 3.8.5 /var/lib/jenkins/workspace/ec2_jjb/venv/bin/python3

self = <test_smallfile.TestSmallFile object at 0x7f7de3489dc0>
run = <models.benchmark_run.BenchmarkRun object at 0x7f7de34a8430>

    def test_smallfile(self, run):
>       self.run_and_check_benchmark(run)

e2e/tests/test_smallfile.py:12: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
e2e/models/test_base.py:92: in run_and_check_benchmark
    run.start(desired_state=desired_running_state, default_timeout=default_timeout)
e2e/models/benchmark_run.py:22: in start
    self.metadata = self.cluster.create_benchmark(self.yaml, desired_state, default_timeout=default_timeout)
e2e/util/k8s.py:117: in create_benchmark
    self.crd_client.create_namespaced_custom_object(
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:225: in create_namespaced_custom_object
    return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs)  # noqa: E501
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:344: in create_namespaced_custom_object_with_http_info
    return self.api_client.call_api(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:348: in call_api
    return self.__call_api(resource_path, method,
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:180: in __call_api
    response_data = self.request(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:391: in request
    return self.rest_client.POST(url,
venv/lib/python3.8/site-packages/kubernetes/client/rest.py:274: in POST
    return self.request("POST", url,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <kubernetes.client.rest.RESTClientObject object at 0x7f7de34a8070>
method = 'POST'
url = 'https://api.jenkins-ci.perf-testing.devcluster.openshift.com:6443/apis/ripsaw.cloudbulldozer.io/v1alpha1/namespaces/my-ripsaw/benchmarks'
query_params = []
headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/12.0.0/python'}
body = {'apiVersion': 'ripsaw.cloudbulldozer.io/v1alpha1', 'kind': 'Benchmark', 'metadata': {'name': 'smallfile-hostpath', 'n...727m7ynd2xyt4y.us-west-2.es.amazonaws.com'}, 'hostpath': '/mnt/vda1/smallfile', 'metadata': {'collection': True}, ...}}
post_params = {}, _preload_content = True, _request_timeout = None

    def request(self, method, url, query_params=None, headers=None,
                body=None, post_params=None, _preload_content=True,
                _request_timeout=None):
        """Perform requests.
    
        :param method: http request method
        :param url: http request url
        :param query_params: query parameters in the url
        :param headers: http request headers
        :param body: request json body, for `application/json`
        :param post_params: request post parameters,
                            `application/x-www-form-urlencoded`
                            and `multipart/form-data`
        :param _preload_content: if False, the urllib3.HTTPResponse object will
                                 be returned without reading/decoding response
                                 data. Default is True.
        :param _request_timeout: timeout setting for this request. If one
                                 number provided, it will be total request
                                 timeout. It can also be a pair (tuple) of
                                 (connection, read) timeouts.
        """
        method = method.upper()
        assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT',
                          'PATCH', 'OPTIONS']
    
        if post_params and body:
            raise ApiValueError(
                "body parameter cannot be used with post_params parameter."
            )
    
        post_params = post_params or {}
        headers = headers or {}
    
        timeout = None
        if _request_timeout:
            if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)):  # noqa: E501,F821
                timeout = urllib3.Timeout(total=_request_timeout)
            elif (isinstance(_request_timeout, tuple) and
                  len(_request_timeout) == 2):
                timeout = urllib3.Timeout(
                    connect=_request_timeout[0], read=_request_timeout[1])
    
        if 'Content-Type' not in headers:
            headers['Content-Type'] = 'application/json'
    
        try:
            # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE`
            if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']:
                if query_params:
                    url += '?' + urlencode(query_params)
                if re.search('json', headers['Content-Type'], re.IGNORECASE):
                    if headers['Content-Type'] == 'application/json-patch+json':
                        if not isinstance(body, list):
                            headers['Content-Type'] = \
                                'application/strategic-merge-patch+json'
                    request_body = None
                    if body is not None:
                        request_body = json.dumps(body)
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'application/x-www-form-urlencoded':  # noqa: E501
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=False,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'multipart/form-data':
                    # must del headers['Content-Type'], or the correct
                    # Content-Type which generated by urllib3 will be
                    # overwritten.
                    del headers['Content-Type']
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=True,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                # Pass a `string` parameter directly in the body to support
                # other content types than Json when `body` argument is
                # provided in serialized form
                elif isinstance(body, str) or isinstance(body, bytes):
                    request_body = body
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                else:
                    # Cannot generate the request from given parameters
                    msg = """Cannot prepare a request message for provided
                             arguments. Please check that your arguments match
                             declared content type."""
                    raise ApiException(status=0, reason=msg)
            # For `GET`, `HEAD`
            else:
                r = self.pool_manager.request(method, url,
                                              fields=query_params,
                                              preload_content=_preload_content,
                                              timeout=timeout,
                                              headers=headers)
        except urllib3.exceptions.SSLError as e:
            msg = "{0}\n{1}".format(type(e).__name__, str(e))
            raise ApiException(status=0, reason=msg)
    
        if _preload_content:
            r = RESTResponse(r)
    
            # In the python 3, the response.data is bytes.
            # we need to decode it to string.
            if six.PY3:
                r.data = r.data.decode('utf8')
    
            # log response body
            logger.debug("response body: %s", r.data)
    
        if not 200 <= r.status <= 299:
>           raise ApiException(http_resp=r)
E           kubernetes.client.exceptions.ApiException: (403)
E           Reason: Forbidden
E           HTTP response headers: HTTPHeaderDict({'Audit-Id': '54c799e8-4e84-4390-bc63-0ff46a0f64d3', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '682d9336-3769-4f6e-8f0f-2f36581dea79', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'fb2241dd-f676-4aab-969d-77ab112dfaf0', 'Date': 'Thu, 19 Nov 2020 20:16:37 GMT', 'Content-Length': '490'})
E           HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"benchmarks.ripsaw.cloudbulldozer.io \"smallfile-hostpath\" is forbidden: unable to create new content in namespace my-ripsaw because it is being terminated","reason":"Forbidden","details":{"name":"smallfile-hostpath","group":"ripsaw.cloudbulldozer.io","kind":"benchmarks","causes":[{"reason":"NamespaceTerminating","message":"namespace my-ripsaw is being terminated","field":"metadata.namespace"}]},"code":403}

venv/lib/python3.8/site-packages/kubernetes/client/rest.py:233: ApiException

TestSmallFile.test_smallfile[base] 0.00s ⏱

[gw1] linux -- Python 3.8.5 /var/lib/jenkins/workspace/ec2_jjb/venv/bin/python3

self = <test_smallfile.TestSmallFile object at 0x7f7de3585a60>
run = <models.benchmark_run.BenchmarkRun object at 0x7f7de34dc310>

    def test_smallfile(self, run):
>       self.run_and_check_benchmark(run)

e2e/tests/test_smallfile.py:12: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
e2e/models/test_base.py:92: in run_and_check_benchmark
    run.start(desired_state=desired_running_state, default_timeout=default_timeout)
e2e/models/benchmark_run.py:22: in start
    self.metadata = self.cluster.create_benchmark(self.yaml, desired_state, default_timeout=default_timeout)
e2e/util/k8s.py:117: in create_benchmark
    self.crd_client.create_namespaced_custom_object(
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:225: in create_namespaced_custom_object
    return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs)  # noqa: E501
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:344: in create_namespaced_custom_object_with_http_info
    return self.api_client.call_api(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:348: in call_api
    return self.__call_api(resource_path, method,
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:180: in __call_api
    response_data = self.request(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:391: in request
    return self.rest_client.POST(url,
venv/lib/python3.8/site-packages/kubernetes/client/rest.py:274: in POST
    return self.request("POST", url,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <kubernetes.client.rest.RESTClientObject object at 0x7f7de34a8070>
method = 'POST'
url = 'https://api.jenkins-ci.perf-testing.devcluster.openshift.com:6443/apis/ripsaw.cloudbulldozer.io/v1alpha1/namespaces/my-ripsaw/benchmarks'
query_params = []
headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/12.0.0/python'}
body = {'apiVersion': 'ripsaw.cloudbulldozer.io/v1alpha1', 'kind': 'Benchmark', 'metadata': {'name': 'smallfile-base', 'names...taqo7727m7ynd2xyt4y.us-west-2.es.amazonaws.com'}, 'metadata': {'collection': True}, 'test_user': 'homer_simpson', ...}}
post_params = {}, _preload_content = True, _request_timeout = None

    def request(self, method, url, query_params=None, headers=None,
                body=None, post_params=None, _preload_content=True,
                _request_timeout=None):
        """Perform requests.
    
        :param method: http request method
        :param url: http request url
        :param query_params: query parameters in the url
        :param headers: http request headers
        :param body: request json body, for `application/json`
        :param post_params: request post parameters,
                            `application/x-www-form-urlencoded`
                            and `multipart/form-data`
        :param _preload_content: if False, the urllib3.HTTPResponse object will
                                 be returned without reading/decoding response
                                 data. Default is True.
        :param _request_timeout: timeout setting for this request. If one
                                 number provided, it will be total request
                                 timeout. It can also be a pair (tuple) of
                                 (connection, read) timeouts.
        """
        method = method.upper()
        assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT',
                          'PATCH', 'OPTIONS']
    
        if post_params and body:
            raise ApiValueError(
                "body parameter cannot be used with post_params parameter."
            )
    
        post_params = post_params or {}
        headers = headers or {}
    
        timeout = None
        if _request_timeout:
            if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)):  # noqa: E501,F821
                timeout = urllib3.Timeout(total=_request_timeout)
            elif (isinstance(_request_timeout, tuple) and
                  len(_request_timeout) == 2):
                timeout = urllib3.Timeout(
                    connect=_request_timeout[0], read=_request_timeout[1])
    
        if 'Content-Type' not in headers:
            headers['Content-Type'] = 'application/json'
    
        try:
            # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE`
            if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']:
                if query_params:
                    url += '?' + urlencode(query_params)
                if re.search('json', headers['Content-Type'], re.IGNORECASE):
                    if headers['Content-Type'] == 'application/json-patch+json':
                        if not isinstance(body, list):
                            headers['Content-Type'] = \
                                'application/strategic-merge-patch+json'
                    request_body = None
                    if body is not None:
                        request_body = json.dumps(body)
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'application/x-www-form-urlencoded':  # noqa: E501
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=False,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'multipart/form-data':
                    # must del headers['Content-Type'], or the correct
                    # Content-Type which generated by urllib3 will be
                    # overwritten.
                    del headers['Content-Type']
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=True,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                # Pass a `string` parameter directly in the body to support
                # other content types than Json when `body` argument is
                # provided in serialized form
                elif isinstance(body, str) or isinstance(body, bytes):
                    request_body = body
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                else:
                    # Cannot generate the request from given parameters
                    msg = """Cannot prepare a request message for provided
                             arguments. Please check that your arguments match
                             declared content type."""
                    raise ApiException(status=0, reason=msg)
            # For `GET`, `HEAD`
            else:
                r = self.pool_manager.request(method, url,
                                              fields=query_params,
                                              preload_content=_preload_content,
                                              timeout=timeout,
                                              headers=headers)
        except urllib3.exceptions.SSLError as e:
            msg = "{0}\n{1}".format(type(e).__name__, str(e))
            raise ApiException(status=0, reason=msg)
    
        if _preload_content:
            r = RESTResponse(r)
    
            # In the python 3, the response.data is bytes.
            # we need to decode it to string.
            if six.PY3:
                r.data = r.data.decode('utf8')
    
            # log response body
            logger.debug("response body: %s", r.data)
    
        if not 200 <= r.status <= 299:
>           raise ApiException(http_resp=r)
E           kubernetes.client.exceptions.ApiException: (403)
E           Reason: Forbidden
E           HTTP response headers: HTTPHeaderDict({'Audit-Id': '5aac997e-9def-4217-8fdf-cd0896d6ead1', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '682d9336-3769-4f6e-8f0f-2f36581dea79', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'fb2241dd-f676-4aab-969d-77ab112dfaf0', 'Date': 'Thu, 19 Nov 2020 20:16:41 GMT', 'Content-Length': '482'})
E           HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"benchmarks.ripsaw.cloudbulldozer.io \"smallfile-base\" is forbidden: unable to create new content in namespace my-ripsaw because it is being terminated","reason":"Forbidden","details":{"name":"smallfile-base","group":"ripsaw.cloudbulldozer.io","kind":"benchmarks","causes":[{"reason":"NamespaceTerminating","message":"namespace my-ripsaw is being terminated","field":"metadata.namespace"}]},"code":403}

venv/lib/python3.8/site-packages/kubernetes/client/rest.py:233: ApiException

e2e/tests/test_sysbench.py

TestSysbench.test_sysbench[sysbench] 0.00s ⏱

[gw2] linux -- Python 3.8.5 /var/lib/jenkins/workspace/ec2_jjb/venv/bin/python3

self = <test_sysbench.TestSysbench object at 0x7fee580b0880>
run = <models.benchmark_run.BenchmarkRun object at 0x7fee581e4670>

    def test_sysbench(self, run):
>       self.run_and_check_benchmark(run)

e2e/tests/test_sysbench.py:12: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
e2e/models/test_base.py:92: in run_and_check_benchmark
    run.start(desired_state=desired_running_state, default_timeout=default_timeout)
e2e/models/benchmark_run.py:22: in start
    self.metadata = self.cluster.create_benchmark(self.yaml, desired_state, default_timeout=default_timeout)
e2e/util/k8s.py:117: in create_benchmark
    self.crd_client.create_namespaced_custom_object(
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:225: in create_namespaced_custom_object
    return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs)  # noqa: E501
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:344: in create_namespaced_custom_object_with_http_info
    return self.api_client.call_api(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:348: in call_api
    return self.__call_api(resource_path, method,
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:180: in __call_api
    response_data = self.request(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:391: in request
    return self.rest_client.POST(url,
venv/lib/python3.8/site-packages/kubernetes/client/rest.py:274: in POST
    return self.request("POST", url,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <kubernetes.client.rest.RESTClientObject object at 0x7fee581e4430>
method = 'POST'
url = 'https://api.jenkins-ci.perf-testing.devcluster.openshift.com:6443/apis/ripsaw.cloudbulldozer.io/v1alpha1/namespaces/my-ripsaw/benchmarks'
query_params = []
headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/12.0.0/python'}
body = {'apiVersion': 'ripsaw.cloudbulldozer.io/v1alpha1', 'kind': 'Benchmark', 'metadata': {'name': 'sysbench', 'namespace':...': {'collection': False}, 'workload': {'args': {'tests': [{'name': 'cpu', 'parameters': {...}}]}, 'name': 'sysbench'}}}
post_params = {}, _preload_content = True, _request_timeout = None

    def request(self, method, url, query_params=None, headers=None,
                body=None, post_params=None, _preload_content=True,
                _request_timeout=None):
        """Perform requests.
    
        :param method: http request method
        :param url: http request url
        :param query_params: query parameters in the url
        :param headers: http request headers
        :param body: request json body, for `application/json`
        :param post_params: request post parameters,
                            `application/x-www-form-urlencoded`
                            and `multipart/form-data`
        :param _preload_content: if False, the urllib3.HTTPResponse object will
                                 be returned without reading/decoding response
                                 data. Default is True.
        :param _request_timeout: timeout setting for this request. If one
                                 number provided, it will be total request
                                 timeout. It can also be a pair (tuple) of
                                 (connection, read) timeouts.
        """
        method = method.upper()
        assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT',
                          'PATCH', 'OPTIONS']
    
        if post_params and body:
            raise ApiValueError(
                "body parameter cannot be used with post_params parameter."
            )
    
        post_params = post_params or {}
        headers = headers or {}
    
        timeout = None
        if _request_timeout:
            if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)):  # noqa: E501,F821
                timeout = urllib3.Timeout(total=_request_timeout)
            elif (isinstance(_request_timeout, tuple) and
                  len(_request_timeout) == 2):
                timeout = urllib3.Timeout(
                    connect=_request_timeout[0], read=_request_timeout[1])
    
        if 'Content-Type' not in headers:
            headers['Content-Type'] = 'application/json'
    
        try:
            # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE`
            if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']:
                if query_params:
                    url += '?' + urlencode(query_params)
                if re.search('json', headers['Content-Type'], re.IGNORECASE):
                    if headers['Content-Type'] == 'application/json-patch+json':
                        if not isinstance(body, list):
                            headers['Content-Type'] = \
                                'application/strategic-merge-patch+json'
                    request_body = None
                    if body is not None:
                        request_body = json.dumps(body)
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'application/x-www-form-urlencoded':  # noqa: E501
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=False,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'multipart/form-data':
                    # must del headers['Content-Type'], or the correct
                    # Content-Type which generated by urllib3 will be
                    # overwritten.
                    del headers['Content-Type']
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=True,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                # Pass a `string` parameter directly in the body to support
                # other content types than Json when `body` argument is
                # provided in serialized form
                elif isinstance(body, str) or isinstance(body, bytes):
                    request_body = body
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                else:
                    # Cannot generate the request from given parameters
                    msg = """Cannot prepare a request message for provided
                             arguments. Please check that your arguments match
                             declared content type."""
                    raise ApiException(status=0, reason=msg)
            # For `GET`, `HEAD`
            else:
                r = self.pool_manager.request(method, url,
                                              fields=query_params,
                                              preload_content=_preload_content,
                                              timeout=timeout,
                                              headers=headers)
        except urllib3.exceptions.SSLError as e:
            msg = "{0}\n{1}".format(type(e).__name__, str(e))
            raise ApiException(status=0, reason=msg)
    
        if _preload_content:
            r = RESTResponse(r)
    
            # In the python 3, the response.data is bytes.
            # we need to decode it to string.
            if six.PY3:
                r.data = r.data.decode('utf8')
    
            # log response body
            logger.debug("response body: %s", r.data)
    
        if not 200 <= r.status <= 299:
>           raise ApiException(http_resp=r)
E           kubernetes.client.exceptions.ApiException: (403)
E           Reason: Forbidden
E           HTTP response headers: HTTPHeaderDict({'Audit-Id': '11fa38d3-12d6-42b1-88da-d21a483acddb', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '682d9336-3769-4f6e-8f0f-2f36581dea79', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'fb2241dd-f676-4aab-969d-77ab112dfaf0', 'Date': 'Thu, 19 Nov 2020 20:16:38 GMT', 'Content-Length': '470'})
E           HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"benchmarks.ripsaw.cloudbulldozer.io \"sysbench\" is forbidden: unable to create new content in namespace my-ripsaw because it is being terminated","reason":"Forbidden","details":{"name":"sysbench","group":"ripsaw.cloudbulldozer.io","kind":"benchmarks","causes":[{"reason":"NamespaceTerminating","message":"namespace my-ripsaw is being terminated","field":"metadata.namespace"}]},"code":403}

venv/lib/python3.8/site-packages/kubernetes/client/rest.py:233: ApiException

e2e/tests/test_fiod.py

TestFiod.test_fiod[hostpath] 0.00s ⏱

[gw0] linux -- Python 3.8.5 /var/lib/jenkins/workspace/ec2_jjb/venv/bin/python3

self = <test_fiod.TestFiod object at 0x7f247bc85d30>
run = <models.benchmark_run.BenchmarkRun object at 0x7f2480193190>

    def test_fiod(self, run):
>       self.run_and_check_benchmark(run)

e2e/tests/test_fiod.py:42: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
e2e/models/test_base.py:92: in run_and_check_benchmark
    run.start(desired_state=desired_running_state, default_timeout=default_timeout)
e2e/models/benchmark_run.py:22: in start
    self.metadata = self.cluster.create_benchmark(self.yaml, desired_state, default_timeout=default_timeout)
e2e/util/k8s.py:117: in create_benchmark
    self.crd_client.create_namespaced_custom_object(
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:225: in create_namespaced_custom_object
    return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs)  # noqa: E501
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:344: in create_namespaced_custom_object_with_http_info
    return self.api_client.call_api(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:348: in call_api
    return self.__call_api(resource_path, method,
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:180: in __call_api
    response_data = self.request(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:391: in request
    return self.rest_client.POST(url,
venv/lib/python3.8/site-packages/kubernetes/client/rest.py:274: in POST
    return self.request("POST", url,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <kubernetes.client.rest.RESTClientObject object at 0x7f24800dd6a0>
method = 'POST'
url = 'https://api.jenkins-ci.perf-testing.devcluster.openshift.com:6443/apis/ripsaw.cloudbulldozer.io/v1alpha1/namespaces/my-ripsaw/benchmarks'
query_params = []
headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/12.0.0/python'}
body = {'apiVersion': 'ripsaw.cloudbulldozer.io/v1alpha1', 'kind': 'Benchmark', 'metadata': {'name': 'fiod-hostpath', 'namesp...e_based=1', 'runtime={{ workload_args.write_runtime }}', 'ramp_time={{ workload_args.write_ramp_time }}']}, ...], ...}}
post_params = {}, _preload_content = True, _request_timeout = None

    def request(self, method, url, query_params=None, headers=None,
                body=None, post_params=None, _preload_content=True,
                _request_timeout=None):
        """Perform requests.
    
        :param method: http request method
        :param url: http request url
        :param query_params: query parameters in the url
        :param headers: http request headers
        :param body: request json body, for `application/json`
        :param post_params: request post parameters,
                            `application/x-www-form-urlencoded`
                            and `multipart/form-data`
        :param _preload_content: if False, the urllib3.HTTPResponse object will
                                 be returned without reading/decoding response
                                 data. Default is True.
        :param _request_timeout: timeout setting for this request. If one
                                 number provided, it will be total request
                                 timeout. It can also be a pair (tuple) of
                                 (connection, read) timeouts.
        """
        method = method.upper()
        assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT',
                          'PATCH', 'OPTIONS']
    
        if post_params and body:
            raise ApiValueError(
                "body parameter cannot be used with post_params parameter."
            )
    
        post_params = post_params or {}
        headers = headers or {}
    
        timeout = None
        if _request_timeout:
            if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)):  # noqa: E501,F821
                timeout = urllib3.Timeout(total=_request_timeout)
            elif (isinstance(_request_timeout, tuple) and
                  len(_request_timeout) == 2):
                timeout = urllib3.Timeout(
                    connect=_request_timeout[0], read=_request_timeout[1])
    
        if 'Content-Type' not in headers:
            headers['Content-Type'] = 'application/json'
    
        try:
            # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE`
            if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']:
                if query_params:
                    url += '?' + urlencode(query_params)
                if re.search('json', headers['Content-Type'], re.IGNORECASE):
                    if headers['Content-Type'] == 'application/json-patch+json':
                        if not isinstance(body, list):
                            headers['Content-Type'] = \
                                'application/strategic-merge-patch+json'
                    request_body = None
                    if body is not None:
                        request_body = json.dumps(body)
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'application/x-www-form-urlencoded':  # noqa: E501
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=False,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'multipart/form-data':
                    # must del headers['Content-Type'], or the correct
                    # Content-Type which generated by urllib3 will be
                    # overwritten.
                    del headers['Content-Type']
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=True,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                # Pass a `string` parameter directly in the body to support
                # other content types than Json when `body` argument is
                # provided in serialized form
                elif isinstance(body, str) or isinstance(body, bytes):
                    request_body = body
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                else:
                    # Cannot generate the request from given parameters
                    msg = """Cannot prepare a request message for provided
                             arguments. Please check that your arguments match
                             declared content type."""
                    raise ApiException(status=0, reason=msg)
            # For `GET`, `HEAD`
            else:
                r = self.pool_manager.request(method, url,
                                              fields=query_params,
                                              preload_content=_preload_content,
                                              timeout=timeout,
                                              headers=headers)
        except urllib3.exceptions.SSLError as e:
            msg = "{0}\n{1}".format(type(e).__name__, str(e))
            raise ApiException(status=0, reason=msg)
    
        if _preload_content:
            r = RESTResponse(r)
    
            # In the python 3, the response.data is bytes.
            # we need to decode it to string.
            if six.PY3:
                r.data = r.data.decode('utf8')
    
            # log response body
            logger.debug("response body: %s", r.data)
    
        if not 200 <= r.status <= 299:
>           raise ApiException(http_resp=r)
E           kubernetes.client.exceptions.ApiException: (403)
E           Reason: Forbidden
E           HTTP response headers: HTTPHeaderDict({'Audit-Id': 'a14cfe18-47a3-47ad-9428-fbd0d8a4356b', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '682d9336-3769-4f6e-8f0f-2f36581dea79', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'fb2241dd-f676-4aab-969d-77ab112dfaf0', 'Date': 'Thu, 19 Nov 2020 20:16:39 GMT', 'Content-Length': '480'})
E           HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"benchmarks.ripsaw.cloudbulldozer.io \"fiod-hostpath\" is forbidden: unable to create new content in namespace my-ripsaw because it is being terminated","reason":"Forbidden","details":{"name":"fiod-hostpath","group":"ripsaw.cloudbulldozer.io","kind":"benchmarks","causes":[{"reason":"NamespaceTerminating","message":"namespace my-ripsaw is being terminated","field":"metadata.namespace"}]},"code":403}

venv/lib/python3.8/site-packages/kubernetes/client/rest.py:233: ApiException

TestFiod.test_fiod[base] 0.00s ⏱

[gw0] linux -- Python 3.8.5 /var/lib/jenkins/workspace/ec2_jjb/venv/bin/python3

self = <test_fiod.TestFiod object at 0x7f247bdfe9a0>
run = <models.benchmark_run.BenchmarkRun object at 0x7f2480193f40>

    def test_fiod(self, run):
>       self.run_and_check_benchmark(run)

e2e/tests/test_fiod.py:42: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
e2e/models/test_base.py:92: in run_and_check_benchmark
    run.start(desired_state=desired_running_state, default_timeout=default_timeout)
e2e/models/benchmark_run.py:22: in start
    self.metadata = self.cluster.create_benchmark(self.yaml, desired_state, default_timeout=default_timeout)
e2e/util/k8s.py:117: in create_benchmark
    self.crd_client.create_namespaced_custom_object(
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:225: in create_namespaced_custom_object
    return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs)  # noqa: E501
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:344: in create_namespaced_custom_object_with_http_info
    return self.api_client.call_api(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:348: in call_api
    return self.__call_api(resource_path, method,
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:180: in __call_api
    response_data = self.request(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:391: in request
    return self.rest_client.POST(url,
venv/lib/python3.8/site-packages/kubernetes/client/rest.py:274: in POST
    return self.request("POST", url,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <kubernetes.client.rest.RESTClientObject object at 0x7f24800dd6a0>
method = 'POST'
url = 'https://api.jenkins-ci.perf-testing.devcluster.openshift.com:6443/apis/ripsaw.cloudbulldozer.io/v1alpha1/namespaces/my-ripsaw/benchmarks'
query_params = []
headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/12.0.0/python'}
body = {'apiVersion': 'ripsaw.cloudbulldozer.io/v1alpha1', 'kind': 'Benchmark', 'metadata': {'name': 'fiod-base', 'namespace'...rgs.write_runtime }}', 'ramp_time={{ workload_args.write_ramp_time }}']}, ...], 'metadata': {'collection': True}, ...}}
post_params = {}, _preload_content = True, _request_timeout = None

    def request(self, method, url, query_params=None, headers=None,
                body=None, post_params=None, _preload_content=True,
                _request_timeout=None):
        """Perform requests.
    
        :param method: http request method
        :param url: http request url
        :param query_params: query parameters in the url
        :param headers: http request headers
        :param body: request json body, for `application/json`
        :param post_params: request post parameters,
                            `application/x-www-form-urlencoded`
                            and `multipart/form-data`
        :param _preload_content: if False, the urllib3.HTTPResponse object will
                                 be returned without reading/decoding response
                                 data. Default is True.
        :param _request_timeout: timeout setting for this request. If one
                                 number provided, it will be total request
                                 timeout. It can also be a pair (tuple) of
                                 (connection, read) timeouts.
        """
        method = method.upper()
        assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT',
                          'PATCH', 'OPTIONS']
    
        if post_params and body:
            raise ApiValueError(
                "body parameter cannot be used with post_params parameter."
            )
    
        post_params = post_params or {}
        headers = headers or {}
    
        timeout = None
        if _request_timeout:
            if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)):  # noqa: E501,F821
                timeout = urllib3.Timeout(total=_request_timeout)
            elif (isinstance(_request_timeout, tuple) and
                  len(_request_timeout) == 2):
                timeout = urllib3.Timeout(
                    connect=_request_timeout[0], read=_request_timeout[1])
    
        if 'Content-Type' not in headers:
            headers['Content-Type'] = 'application/json'
    
        try:
            # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE`
            if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']:
                if query_params:
                    url += '?' + urlencode(query_params)
                if re.search('json', headers['Content-Type'], re.IGNORECASE):
                    if headers['Content-Type'] == 'application/json-patch+json':
                        if not isinstance(body, list):
                            headers['Content-Type'] = \
                                'application/strategic-merge-patch+json'
                    request_body = None
                    if body is not None:
                        request_body = json.dumps(body)
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'application/x-www-form-urlencoded':  # noqa: E501
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=False,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'multipart/form-data':
                    # must del headers['Content-Type'], or the correct
                    # Content-Type which generated by urllib3 will be
                    # overwritten.
                    del headers['Content-Type']
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=True,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                # Pass a `string` parameter directly in the body to support
                # other content types than Json when `body` argument is
                # provided in serialized form
                elif isinstance(body, str) or isinstance(body, bytes):
                    request_body = body
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                else:
                    # Cannot generate the request from given parameters
                    msg = """Cannot prepare a request message for provided
                             arguments. Please check that your arguments match
                             declared content type."""
                    raise ApiException(status=0, reason=msg)
            # For `GET`, `HEAD`
            else:
                r = self.pool_manager.request(method, url,
                                              fields=query_params,
                                              preload_content=_preload_content,
                                              timeout=timeout,
                                              headers=headers)
        except urllib3.exceptions.SSLError as e:
            msg = "{0}\n{1}".format(type(e).__name__, str(e))
            raise ApiException(status=0, reason=msg)
    
        if _preload_content:
            r = RESTResponse(r)
    
            # In the python 3, the response.data is bytes.
            # we need to decode it to string.
            if six.PY3:
                r.data = r.data.decode('utf8')
    
            # log response body
            logger.debug("response body: %s", r.data)
    
        if not 200 <= r.status <= 299:
>           raise ApiException(http_resp=r)
E           kubernetes.client.exceptions.ApiException: (403)
E           Reason: Forbidden
E           HTTP response headers: HTTPHeaderDict({'Audit-Id': 'b40369ca-945f-47e0-b7d7-b2658eba0fa3', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '682d9336-3769-4f6e-8f0f-2f36581dea79', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'fb2241dd-f676-4aab-969d-77ab112dfaf0', 'Date': 'Thu, 19 Nov 2020 20:16:44 GMT', 'Content-Length': '472'})
E           HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"benchmarks.ripsaw.cloudbulldozer.io \"fiod-base\" is forbidden: unable to create new content in namespace my-ripsaw because it is being terminated","reason":"Forbidden","details":{"name":"fiod-base","group":"ripsaw.cloudbulldozer.io","kind":"benchmarks","causes":[{"reason":"NamespaceTerminating","message":"namespace my-ripsaw is being terminated","field":"metadata.namespace"}]},"code":403}

venv/lib/python3.8/site-packages/kubernetes/client/rest.py:233: ApiException

e2e/tests/test_vegeta.py

TestVegeta.test_vegeta[hostnetwork] 0.00s ⏱

[gw2] linux -- Python 3.8.5 /var/lib/jenkins/workspace/ec2_jjb/venv/bin/python3

self = <test_vegeta.TestVegeta object at 0x7fee53f95310>
run = <models.benchmark_run.BenchmarkRun object at 0x7fee53fcbca0>

    def test_vegeta(self, run):
>       self.run_and_check_benchmark(run)

e2e/tests/test_vegeta.py:12: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
e2e/models/test_base.py:92: in run_and_check_benchmark
    run.start(desired_state=desired_running_state, default_timeout=default_timeout)
e2e/models/benchmark_run.py:22: in start
    self.metadata = self.cluster.create_benchmark(self.yaml, desired_state, default_timeout=default_timeout)
e2e/util/k8s.py:117: in create_benchmark
    self.crd_client.create_namespaced_custom_object(
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:225: in create_namespaced_custom_object
    return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs)  # noqa: E501
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:344: in create_namespaced_custom_object_with_http_info
    return self.api_client.call_api(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:348: in call_api
    return self.__call_api(resource_path, method,
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:180: in __call_api
    response_data = self.request(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:391: in request
    return self.rest_client.POST(url,
venv/lib/python3.8/site-packages/kubernetes/client/rest.py:274: in POST
    return self.request("POST", url,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <kubernetes.client.rest.RESTClientObject object at 0x7fee53fcb970>
method = 'POST'
url = 'https://api.jenkins-ci.perf-testing.devcluster.openshift.com:6443/apis/ripsaw.cloudbulldozer.io/v1alpha1/namespaces/my-ripsaw/benchmarks'
query_params = []
headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/12.0.0/python'}
body = {'apiVersion': 'ripsaw.cloudbulldozer.io/v1alpha1', 'kind': 'Benchmark', 'metadata': {'name': 'vegeta-hostnetwork', 'n...twork': True, 'targets': [{'duration': 5, 'keepalive': True, 'name': '2w-ka', 'samples': 1, ...}]}, 'name': 'vegeta'}}}
post_params = {}, _preload_content = True, _request_timeout = None

    def request(self, method, url, query_params=None, headers=None,
                body=None, post_params=None, _preload_content=True,
                _request_timeout=None):
        """Perform requests.
    
        :param method: http request method
        :param url: http request url
        :param query_params: query parameters in the url
        :param headers: http request headers
        :param body: request json body, for `application/json`
        :param post_params: request post parameters,
                            `application/x-www-form-urlencoded`
                            and `multipart/form-data`
        :param _preload_content: if False, the urllib3.HTTPResponse object will
                                 be returned without reading/decoding response
                                 data. Default is True.
        :param _request_timeout: timeout setting for this request. If one
                                 number provided, it will be total request
                                 timeout. It can also be a pair (tuple) of
                                 (connection, read) timeouts.
        """
        method = method.upper()
        assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT',
                          'PATCH', 'OPTIONS']
    
        if post_params and body:
            raise ApiValueError(
                "body parameter cannot be used with post_params parameter."
            )
    
        post_params = post_params or {}
        headers = headers or {}
    
        timeout = None
        if _request_timeout:
            if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)):  # noqa: E501,F821
                timeout = urllib3.Timeout(total=_request_timeout)
            elif (isinstance(_request_timeout, tuple) and
                  len(_request_timeout) == 2):
                timeout = urllib3.Timeout(
                    connect=_request_timeout[0], read=_request_timeout[1])
    
        if 'Content-Type' not in headers:
            headers['Content-Type'] = 'application/json'
    
        try:
            # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE`
            if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']:
                if query_params:
                    url += '?' + urlencode(query_params)
                if re.search('json', headers['Content-Type'], re.IGNORECASE):
                    if headers['Content-Type'] == 'application/json-patch+json':
                        if not isinstance(body, list):
                            headers['Content-Type'] = \
                                'application/strategic-merge-patch+json'
                    request_body = None
                    if body is not None:
                        request_body = json.dumps(body)
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'application/x-www-form-urlencoded':  # noqa: E501
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=False,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'multipart/form-data':
                    # must del headers['Content-Type'], or the correct
                    # Content-Type which generated by urllib3 will be
                    # overwritten.
                    del headers['Content-Type']
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=True,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                # Pass a `string` parameter directly in the body to support
                # other content types than Json when `body` argument is
                # provided in serialized form
                elif isinstance(body, str) or isinstance(body, bytes):
                    request_body = body
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                else:
                    # Cannot generate the request from given parameters
                    msg = """Cannot prepare a request message for provided
                             arguments. Please check that your arguments match
                             declared content type."""
                    raise ApiException(status=0, reason=msg)
            # For `GET`, `HEAD`
            else:
                r = self.pool_manager.request(method, url,
                                              fields=query_params,
                                              preload_content=_preload_content,
                                              timeout=timeout,
                                              headers=headers)
        except urllib3.exceptions.SSLError as e:
            msg = "{0}\n{1}".format(type(e).__name__, str(e))
            raise ApiException(status=0, reason=msg)
    
        if _preload_content:
            r = RESTResponse(r)
    
            # In the python 3, the response.data is bytes.
            # we need to decode it to string.
            if six.PY3:
                r.data = r.data.decode('utf8')
    
            # log response body
            logger.debug("response body: %s", r.data)
    
        if not 200 <= r.status <= 299:
>           raise ApiException(http_resp=r)
E           kubernetes.client.exceptions.ApiException: (403)
E           Reason: Forbidden
E           HTTP response headers: HTTPHeaderDict({'Audit-Id': 'f4999596-1855-4285-b268-8a28131b2940', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '682d9336-3769-4f6e-8f0f-2f36581dea79', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'fb2241dd-f676-4aab-969d-77ab112dfaf0', 'Date': 'Thu, 19 Nov 2020 20:16:41 GMT', 'Content-Length': '490'})
E           HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"benchmarks.ripsaw.cloudbulldozer.io \"vegeta-hostnetwork\" is forbidden: unable to create new content in namespace my-ripsaw because it is being terminated","reason":"Forbidden","details":{"name":"vegeta-hostnetwork","group":"ripsaw.cloudbulldozer.io","kind":"benchmarks","causes":[{"reason":"NamespaceTerminating","message":"namespace my-ripsaw is being terminated","field":"metadata.namespace"}]},"code":403}

venv/lib/python3.8/site-packages/kubernetes/client/rest.py:233: ApiException

TestVegeta.test_vegeta[base] 0.00s ⏱

[gw2] linux -- Python 3.8.5 /var/lib/jenkins/workspace/ec2_jjb/venv/bin/python3

self = <test_vegeta.TestVegeta object at 0x7fee53feb2e0>
run = <models.benchmark_run.BenchmarkRun object at 0x7fee5808c0a0>

    def test_vegeta(self, run):
>       self.run_and_check_benchmark(run)

e2e/tests/test_vegeta.py:12: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
e2e/models/test_base.py:92: in run_and_check_benchmark
    run.start(desired_state=desired_running_state, default_timeout=default_timeout)
e2e/models/benchmark_run.py:22: in start
    self.metadata = self.cluster.create_benchmark(self.yaml, desired_state, default_timeout=default_timeout)
e2e/util/k8s.py:117: in create_benchmark
    self.crd_client.create_namespaced_custom_object(
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:225: in create_namespaced_custom_object
    return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs)  # noqa: E501
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:344: in create_namespaced_custom_object_with_http_info
    return self.api_client.call_api(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:348: in call_api
    return self.__call_api(resource_path, method,
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:180: in __call_api
    response_data = self.request(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:391: in request
    return self.rest_client.POST(url,
venv/lib/python3.8/site-packages/kubernetes/client/rest.py:274: in POST
    return self.request("POST", url,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <kubernetes.client.rest.RESTClientObject object at 0x7fee53fcb970>
method = 'POST'
url = 'https://api.jenkins-ci.perf-testing.devcluster.openshift.com:6443/apis/ripsaw.cloudbulldozer.io/v1alpha1/namespaces/my-ripsaw/benchmarks'
query_params = []
headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/12.0.0/python'}
body = {'apiVersion': 'ripsaw.cloudbulldozer.io/v1alpha1', 'kind': 'Benchmark', 'metadata': {'name': 'vegeta-base', 'namespac...e': '2w-ka', 'samples': 1, ...}, {'duration': 5, 'name': '2w-noka', 'urls': [...], 'workers': 2}]}, 'name': 'vegeta'}}}
post_params = {}, _preload_content = True, _request_timeout = None

    def request(self, method, url, query_params=None, headers=None,
                body=None, post_params=None, _preload_content=True,
                _request_timeout=None):
        """Perform requests.
    
        :param method: http request method
        :param url: http request url
        :param query_params: query parameters in the url
        :param headers: http request headers
        :param body: request json body, for `application/json`
        :param post_params: request post parameters,
                            `application/x-www-form-urlencoded`
                            and `multipart/form-data`
        :param _preload_content: if False, the urllib3.HTTPResponse object will
                                 be returned without reading/decoding response
                                 data. Default is True.
        :param _request_timeout: timeout setting for this request. If one
                                 number provided, it will be total request
                                 timeout. It can also be a pair (tuple) of
                                 (connection, read) timeouts.
        """
        method = method.upper()
        assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT',
                          'PATCH', 'OPTIONS']
    
        if post_params and body:
            raise ApiValueError(
                "body parameter cannot be used with post_params parameter."
            )
    
        post_params = post_params or {}
        headers = headers or {}
    
        timeout = None
        if _request_timeout:
            if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)):  # noqa: E501,F821
                timeout = urllib3.Timeout(total=_request_timeout)
            elif (isinstance(_request_timeout, tuple) and
                  len(_request_timeout) == 2):
                timeout = urllib3.Timeout(
                    connect=_request_timeout[0], read=_request_timeout[1])
    
        if 'Content-Type' not in headers:
            headers['Content-Type'] = 'application/json'
    
        try:
            # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE`
            if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']:
                if query_params:
                    url += '?' + urlencode(query_params)
                if re.search('json', headers['Content-Type'], re.IGNORECASE):
                    if headers['Content-Type'] == 'application/json-patch+json':
                        if not isinstance(body, list):
                            headers['Content-Type'] = \
                                'application/strategic-merge-patch+json'
                    request_body = None
                    if body is not None:
                        request_body = json.dumps(body)
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'application/x-www-form-urlencoded':  # noqa: E501
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=False,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'multipart/form-data':
                    # must del headers['Content-Type'], or the correct
                    # Content-Type which generated by urllib3 will be
                    # overwritten.
                    del headers['Content-Type']
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=True,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                # Pass a `string` parameter directly in the body to support
                # other content types than Json when `body` argument is
                # provided in serialized form
                elif isinstance(body, str) or isinstance(body, bytes):
                    request_body = body
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                else:
                    # Cannot generate the request from given parameters
                    msg = """Cannot prepare a request message for provided
                             arguments. Please check that your arguments match
                             declared content type."""
                    raise ApiException(status=0, reason=msg)
            # For `GET`, `HEAD`
            else:
                r = self.pool_manager.request(method, url,
                                              fields=query_params,
                                              preload_content=_preload_content,
                                              timeout=timeout,
                                              headers=headers)
        except urllib3.exceptions.SSLError as e:
            msg = "{0}\n{1}".format(type(e).__name__, str(e))
            raise ApiException(status=0, reason=msg)
    
        if _preload_content:
            r = RESTResponse(r)
    
            # In the python 3, the response.data is bytes.
            # we need to decode it to string.
            if six.PY3:
                r.data = r.data.decode('utf8')
    
            # log response body
            logger.debug("response body: %s", r.data)
    
        if not 200 <= r.status <= 299:
>           raise ApiException(http_resp=r)
E           kubernetes.client.exceptions.ApiException: (403)
E           Reason: Forbidden
E           HTTP response headers: HTTPHeaderDict({'Audit-Id': 'a601dc29-fc22-4fa4-8d64-a4213bbec998', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '682d9336-3769-4f6e-8f0f-2f36581dea79', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'fb2241dd-f676-4aab-969d-77ab112dfaf0', 'Date': 'Thu, 19 Nov 2020 20:16:43 GMT', 'Content-Length': '476'})
E           HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"benchmarks.ripsaw.cloudbulldozer.io \"vegeta-base\" is forbidden: unable to create new content in namespace my-ripsaw because it is being terminated","reason":"Forbidden","details":{"name":"vegeta-base","group":"ripsaw.cloudbulldozer.io","kind":"benchmarks","causes":[{"reason":"NamespaceTerminating","message":"namespace my-ripsaw is being terminated","field":"metadata.namespace"}]},"code":403}

venv/lib/python3.8/site-packages/kubernetes/client/rest.py:233: ApiException

e2e/tests/test_uperf.py

TestUperf.test_uperf[uperf_serviceip] 0.00s ⏱

[gw1] linux -- Python 3.8.5 /var/lib/jenkins/workspace/ec2_jjb/venv/bin/python3

self = <test_uperf.TestUperf object at 0x7f7de30970a0>
run = <models.benchmark_run.BenchmarkRun object at 0x7f7de35b5d60>

    def test_uperf(self, run):
>       self.run_and_check_benchmark(run)

e2e/tests/test_uperf.py:12: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
e2e/models/test_base.py:92: in run_and_check_benchmark
    run.start(desired_state=desired_running_state, default_timeout=default_timeout)
e2e/models/benchmark_run.py:22: in start
    self.metadata = self.cluster.create_benchmark(self.yaml, desired_state, default_timeout=default_timeout)
e2e/util/k8s.py:117: in create_benchmark
    self.crd_client.create_namespaced_custom_object(
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:225: in create_namespaced_custom_object
    return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs)  # noqa: E501
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:344: in create_namespaced_custom_object_with_http_info
    return self.api_client.call_api(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:348: in call_api
    return self.__call_api(resource_path, method,
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:180: in __call_api
    response_data = self.request(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:391: in request
    return self.rest_client.POST(url,
venv/lib/python3.8/site-packages/kubernetes/client/rest.py:274: in POST
    return self.request("POST", url,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <kubernetes.client.rest.RESTClientObject object at 0x7f7de35b5a30>
method = 'POST'
url = 'https://api.jenkins-ci.perf-testing.devcluster.openshift.com:6443/apis/ripsaw.cloudbulldozer.io/v1alpha1/namespaces/my-ripsaw/benchmarks'
query_params = []
headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/12.0.0/python'}
body = {'apiVersion': 'ripsaw.cloudbulldozer.io/v1alpha1', 'kind': 'Benchmark', 'metadata': {'name': 'uperf-serviceip', 'name...ad': {'args': {'hostnetwork': False, 'multus': {'enabled': False}, 'nthrs': [1, 2], 'pair': 1, ...}, 'name': 'uperf'}}}
post_params = {}, _preload_content = True, _request_timeout = None

    def request(self, method, url, query_params=None, headers=None,
                body=None, post_params=None, _preload_content=True,
                _request_timeout=None):
        """Perform requests.
    
        :param method: http request method
        :param url: http request url
        :param query_params: query parameters in the url
        :param headers: http request headers
        :param body: request json body, for `application/json`
        :param post_params: request post parameters,
                            `application/x-www-form-urlencoded`
                            and `multipart/form-data`
        :param _preload_content: if False, the urllib3.HTTPResponse object will
                                 be returned without reading/decoding response
                                 data. Default is True.
        :param _request_timeout: timeout setting for this request. If one
                                 number provided, it will be total request
                                 timeout. It can also be a pair (tuple) of
                                 (connection, read) timeouts.
        """
        method = method.upper()
        assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT',
                          'PATCH', 'OPTIONS']
    
        if post_params and body:
            raise ApiValueError(
                "body parameter cannot be used with post_params parameter."
            )
    
        post_params = post_params or {}
        headers = headers or {}
    
        timeout = None
        if _request_timeout:
            if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)):  # noqa: E501,F821
                timeout = urllib3.Timeout(total=_request_timeout)
            elif (isinstance(_request_timeout, tuple) and
                  len(_request_timeout) == 2):
                timeout = urllib3.Timeout(
                    connect=_request_timeout[0], read=_request_timeout[1])
    
        if 'Content-Type' not in headers:
            headers['Content-Type'] = 'application/json'
    
        try:
            # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE`
            if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']:
                if query_params:
                    url += '?' + urlencode(query_params)
                if re.search('json', headers['Content-Type'], re.IGNORECASE):
                    if headers['Content-Type'] == 'application/json-patch+json':
                        if not isinstance(body, list):
                            headers['Content-Type'] = \
                                'application/strategic-merge-patch+json'
                    request_body = None
                    if body is not None:
                        request_body = json.dumps(body)
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'application/x-www-form-urlencoded':  # noqa: E501
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=False,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'multipart/form-data':
                    # must del headers['Content-Type'], or the correct
                    # Content-Type which generated by urllib3 will be
                    # overwritten.
                    del headers['Content-Type']
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=True,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                # Pass a `string` parameter directly in the body to support
                # other content types than Json when `body` argument is
                # provided in serialized form
                elif isinstance(body, str) or isinstance(body, bytes):
                    request_body = body
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                else:
                    # Cannot generate the request from given parameters
                    msg = """Cannot prepare a request message for provided
                             arguments. Please check that your arguments match
                             declared content type."""
                    raise ApiException(status=0, reason=msg)
            # For `GET`, `HEAD`
            else:
                r = self.pool_manager.request(method, url,
                                              fields=query_params,
                                              preload_content=_preload_content,
                                              timeout=timeout,
                                              headers=headers)
        except urllib3.exceptions.SSLError as e:
            msg = "{0}\n{1}".format(type(e).__name__, str(e))
            raise ApiException(status=0, reason=msg)
    
        if _preload_content:
            r = RESTResponse(r)
    
            # In the python 3, the response.data is bytes.
            # we need to decode it to string.
            if six.PY3:
                r.data = r.data.decode('utf8')
    
            # log response body
            logger.debug("response body: %s", r.data)
    
        if not 200 <= r.status <= 299:
>           raise ApiException(http_resp=r)
E           kubernetes.client.exceptions.ApiException: (403)
E           Reason: Forbidden
E           HTTP response headers: HTTPHeaderDict({'Audit-Id': 'b3f5e472-2de9-4d6d-83dc-0020844530b7', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '682d9336-3769-4f6e-8f0f-2f36581dea79', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'fb2241dd-f676-4aab-969d-77ab112dfaf0', 'Date': 'Thu, 19 Nov 2020 20:16:44 GMT', 'Content-Length': '484'})
E           HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"benchmarks.ripsaw.cloudbulldozer.io \"uperf-serviceip\" is forbidden: unable to create new content in namespace my-ripsaw because it is being terminated","reason":"Forbidden","details":{"name":"uperf-serviceip","group":"ripsaw.cloudbulldozer.io","kind":"benchmarks","causes":[{"reason":"NamespaceTerminating","message":"namespace my-ripsaw is being terminated","field":"metadata.namespace"}]},"code":403}

venv/lib/python3.8/site-packages/kubernetes/client/rest.py:233: ApiException

TestUperf.test_uperf[base] 0.00s ⏱

[gw1] linux -- Python 3.8.5 /var/lib/jenkins/workspace/ec2_jjb/venv/bin/python3

self = <test_uperf.TestUperf object at 0x7f7de30d81f0>
run = <models.benchmark_run.BenchmarkRun object at 0x7f7de35e87c0>

    def test_uperf(self, run):
>       self.run_and_check_benchmark(run)

e2e/tests/test_uperf.py:12: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
e2e/models/test_base.py:92: in run_and_check_benchmark
    run.start(desired_state=desired_running_state, default_timeout=default_timeout)
e2e/models/benchmark_run.py:22: in start
    self.metadata = self.cluster.create_benchmark(self.yaml, desired_state, default_timeout=default_timeout)
e2e/util/k8s.py:117: in create_benchmark
    self.crd_client.create_namespaced_custom_object(
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:225: in create_namespaced_custom_object
    return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs)  # noqa: E501
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:344: in create_namespaced_custom_object_with_http_info
    return self.api_client.call_api(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:348: in call_api
    return self.__call_api(resource_path, method,
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:180: in __call_api
    response_data = self.request(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:391: in request
    return self.rest_client.POST(url,
venv/lib/python3.8/site-packages/kubernetes/client/rest.py:274: in POST
    return self.request("POST", url,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <kubernetes.client.rest.RESTClientObject object at 0x7f7de35b5a30>
method = 'POST'
url = 'https://api.jenkins-ci.perf-testing.devcluster.openshift.com:6443/apis/ripsaw.cloudbulldozer.io/v1alpha1/namespaces/my-ripsaw/benchmarks'
query_params = []
headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/12.0.0/python'}
body = {'apiVersion': 'ripsaw.cloudbulldozer.io/v1alpha1', 'kind': 'Benchmark', 'metadata': {'name': 'uperf-base', 'namespace...ad': {'args': {'hostnetwork': False, 'multus': {'enabled': False}, 'nthrs': [1, 2], 'pair': 1, ...}, 'name': 'uperf'}}}
post_params = {}, _preload_content = True, _request_timeout = None

    def request(self, method, url, query_params=None, headers=None,
                body=None, post_params=None, _preload_content=True,
                _request_timeout=None):
        """Perform requests.
    
        :param method: http request method
        :param url: http request url
        :param query_params: query parameters in the url
        :param headers: http request headers
        :param body: request json body, for `application/json`
        :param post_params: request post parameters,
                            `application/x-www-form-urlencoded`
                            and `multipart/form-data`
        :param _preload_content: if False, the urllib3.HTTPResponse object will
                                 be returned without reading/decoding response
                                 data. Default is True.
        :param _request_timeout: timeout setting for this request. If one
                                 number provided, it will be total request
                                 timeout. It can also be a pair (tuple) of
                                 (connection, read) timeouts.
        """
        method = method.upper()
        assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT',
                          'PATCH', 'OPTIONS']
    
        if post_params and body:
            raise ApiValueError(
                "body parameter cannot be used with post_params parameter."
            )
    
        post_params = post_params or {}
        headers = headers or {}
    
        timeout = None
        if _request_timeout:
            if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)):  # noqa: E501,F821
                timeout = urllib3.Timeout(total=_request_timeout)
            elif (isinstance(_request_timeout, tuple) and
                  len(_request_timeout) == 2):
                timeout = urllib3.Timeout(
                    connect=_request_timeout[0], read=_request_timeout[1])
    
        if 'Content-Type' not in headers:
            headers['Content-Type'] = 'application/json'
    
        try:
            # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE`
            if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']:
                if query_params:
                    url += '?' + urlencode(query_params)
                if re.search('json', headers['Content-Type'], re.IGNORECASE):
                    if headers['Content-Type'] == 'application/json-patch+json':
                        if not isinstance(body, list):
                            headers['Content-Type'] = \
                                'application/strategic-merge-patch+json'
                    request_body = None
                    if body is not None:
                        request_body = json.dumps(body)
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'application/x-www-form-urlencoded':  # noqa: E501
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=False,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'multipart/form-data':
                    # must del headers['Content-Type'], or the correct
                    # Content-Type which generated by urllib3 will be
                    # overwritten.
                    del headers['Content-Type']
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=True,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                # Pass a `string` parameter directly in the body to support
                # other content types than Json when `body` argument is
                # provided in serialized form
                elif isinstance(body, str) or isinstance(body, bytes):
                    request_body = body
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                else:
                    # Cannot generate the request from given parameters
                    msg = """Cannot prepare a request message for provided
                             arguments. Please check that your arguments match
                             declared content type."""
                    raise ApiException(status=0, reason=msg)
            # For `GET`, `HEAD`
            else:
                r = self.pool_manager.request(method, url,
                                              fields=query_params,
                                              preload_content=_preload_content,
                                              timeout=timeout,
                                              headers=headers)
        except urllib3.exceptions.SSLError as e:
            msg = "{0}\n{1}".format(type(e).__name__, str(e))
            raise ApiException(status=0, reason=msg)
    
        if _preload_content:
            r = RESTResponse(r)
    
            # In the python 3, the response.data is bytes.
            # we need to decode it to string.
            if six.PY3:
                r.data = r.data.decode('utf8')
    
            # log response body
            logger.debug("response body: %s", r.data)
    
        if not 200 <= r.status <= 299:
>           raise ApiException(http_resp=r)
E           kubernetes.client.exceptions.ApiException: (403)
E           Reason: Forbidden
E           HTTP response headers: HTTPHeaderDict({'Audit-Id': '85242447-7f3d-4246-9b42-f8905fab6d36', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '682d9336-3769-4f6e-8f0f-2f36581dea79', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'fb2241dd-f676-4aab-969d-77ab112dfaf0', 'Date': 'Thu, 19 Nov 2020 20:16:46 GMT', 'Content-Length': '474'})
E           HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"benchmarks.ripsaw.cloudbulldozer.io \"uperf-base\" is forbidden: unable to create new content in namespace my-ripsaw because it is being terminated","reason":"Forbidden","details":{"name":"uperf-base","group":"ripsaw.cloudbulldozer.io","kind":"benchmarks","causes":[{"reason":"NamespaceTerminating","message":"namespace my-ripsaw is being terminated","field":"metadata.namespace"}]},"code":403}

venv/lib/python3.8/site-packages/kubernetes/client/rest.py:233: ApiException

TestUperf.test_uperf[uperf_networkpolicy] 0.00s ⏱

[gw1] linux -- Python 3.8.5 /var/lib/jenkins/workspace/ec2_jjb/venv/bin/python3

self = <test_uperf.TestUperf object at 0x7f7de3238c70>
run = <models.benchmark_run.BenchmarkRun object at 0x7f7de35e8c40>

    def test_uperf(self, run):
>       self.run_and_check_benchmark(run)

e2e/tests/test_uperf.py:12: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
e2e/models/test_base.py:92: in run_and_check_benchmark
    run.start(desired_state=desired_running_state, default_timeout=default_timeout)
e2e/models/benchmark_run.py:22: in start
    self.metadata = self.cluster.create_benchmark(self.yaml, desired_state, default_timeout=default_timeout)
e2e/util/k8s.py:117: in create_benchmark
    self.crd_client.create_namespaced_custom_object(
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:225: in create_namespaced_custom_object
    return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs)  # noqa: E501
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:344: in create_namespaced_custom_object_with_http_info
    return self.api_client.call_api(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:348: in call_api
    return self.__call_api(resource_path, method,
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:180: in __call_api
    response_data = self.request(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:391: in request
    return self.rest_client.POST(url,
venv/lib/python3.8/site-packages/kubernetes/client/rest.py:274: in POST
    return self.request("POST", url,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <kubernetes.client.rest.RESTClientObject object at 0x7f7de35b5a30>
method = 'POST'
url = 'https://api.jenkins-ci.perf-testing.devcluster.openshift.com:6443/apis/ripsaw.cloudbulldozer.io/v1alpha1/namespaces/my-ripsaw/benchmarks'
query_params = []
headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/12.0.0/python'}
body = {'apiVersion': 'ripsaw.cloudbulldozer.io/v1alpha1', 'kind': 'Benchmark', 'metadata': {'name': 'example-benchmark', 'na...: {'hostnetwork': False, 'multus': {'enabled': False}, 'networkpolicy': True, 'nthrs': [1, 2], ...}, 'name': 'uperf'}}}
post_params = {}, _preload_content = True, _request_timeout = None

    def request(self, method, url, query_params=None, headers=None,
                body=None, post_params=None, _preload_content=True,
                _request_timeout=None):
        """Perform requests.
    
        :param method: http request method
        :param url: http request url
        :param query_params: query parameters in the url
        :param headers: http request headers
        :param body: request json body, for `application/json`
        :param post_params: request post parameters,
                            `application/x-www-form-urlencoded`
                            and `multipart/form-data`
        :param _preload_content: if False, the urllib3.HTTPResponse object will
                                 be returned without reading/decoding response
                                 data. Default is True.
        :param _request_timeout: timeout setting for this request. If one
                                 number provided, it will be total request
                                 timeout. It can also be a pair (tuple) of
                                 (connection, read) timeouts.
        """
        method = method.upper()
        assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT',
                          'PATCH', 'OPTIONS']
    
        if post_params and body:
            raise ApiValueError(
                "body parameter cannot be used with post_params parameter."
            )
    
        post_params = post_params or {}
        headers = headers or {}
    
        timeout = None
        if _request_timeout:
            if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)):  # noqa: E501,F821
                timeout = urllib3.Timeout(total=_request_timeout)
            elif (isinstance(_request_timeout, tuple) and
                  len(_request_timeout) == 2):
                timeout = urllib3.Timeout(
                    connect=_request_timeout[0], read=_request_timeout[1])
    
        if 'Content-Type' not in headers:
            headers['Content-Type'] = 'application/json'
    
        try:
            # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE`
            if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']:
                if query_params:
                    url += '?' + urlencode(query_params)
                if re.search('json', headers['Content-Type'], re.IGNORECASE):
                    if headers['Content-Type'] == 'application/json-patch+json':
                        if not isinstance(body, list):
                            headers['Content-Type'] = \
                                'application/strategic-merge-patch+json'
                    request_body = None
                    if body is not None:
                        request_body = json.dumps(body)
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'application/x-www-form-urlencoded':  # noqa: E501
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=False,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'multipart/form-data':
                    # must del headers['Content-Type'], or the correct
                    # Content-Type which generated by urllib3 will be
                    # overwritten.
                    del headers['Content-Type']
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=True,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                # Pass a `string` parameter directly in the body to support
                # other content types than Json when `body` argument is
                # provided in serialized form
                elif isinstance(body, str) or isinstance(body, bytes):
                    request_body = body
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                else:
                    # Cannot generate the request from given parameters
                    msg = """Cannot prepare a request message for provided
                             arguments. Please check that your arguments match
                             declared content type."""
                    raise ApiException(status=0, reason=msg)
            # For `GET`, `HEAD`
            else:
                r = self.pool_manager.request(method, url,
                                              fields=query_params,
                                              preload_content=_preload_content,
                                              timeout=timeout,
                                              headers=headers)
        except urllib3.exceptions.SSLError as e:
            msg = "{0}\n{1}".format(type(e).__name__, str(e))
            raise ApiException(status=0, reason=msg)
    
        if _preload_content:
            r = RESTResponse(r)
    
            # In the python 3, the response.data is bytes.
            # we need to decode it to string.
            if six.PY3:
                r.data = r.data.decode('utf8')
    
            # log response body
            logger.debug("response body: %s", r.data)
    
        if not 200 <= r.status <= 299:
>           raise ApiException(http_resp=r)
E           kubernetes.client.exceptions.ApiException: (403)
E           Reason: Forbidden
E           HTTP response headers: HTTPHeaderDict({'Audit-Id': '62d17003-bd03-40cd-88f5-695ba1cb8177', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '682d9336-3769-4f6e-8f0f-2f36581dea79', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'fb2241dd-f676-4aab-969d-77ab112dfaf0', 'Date': 'Thu, 19 Nov 2020 20:16:48 GMT', 'Content-Length': '488'})
E           HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"benchmarks.ripsaw.cloudbulldozer.io \"example-benchmark\" is forbidden: unable to create new content in namespace my-ripsaw because it is being terminated","reason":"Forbidden","details":{"name":"example-benchmark","group":"ripsaw.cloudbulldozer.io","kind":"benchmarks","causes":[{"reason":"NamespaceTerminating","message":"namespace my-ripsaw is being terminated","field":"metadata.namespace"}]},"code":403}

venv/lib/python3.8/site-packages/kubernetes/client/rest.py:233: ApiException

TestUperf.test_uperf[uperf_resources] 0.00s ⏱

[gw1] linux -- Python 3.8.5 /var/lib/jenkins/workspace/ec2_jjb/venv/bin/python3

self = <test_uperf.TestUperf object at 0x7f7de2f216a0>
run = <models.benchmark_run.BenchmarkRun object at 0x7f7de362f9a0>

    def test_uperf(self, run):
>       self.run_and_check_benchmark(run)

e2e/tests/test_uperf.py:12: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
e2e/models/test_base.py:92: in run_and_check_benchmark
    run.start(desired_state=desired_running_state, default_timeout=default_timeout)
e2e/models/benchmark_run.py:22: in start
    self.metadata = self.cluster.create_benchmark(self.yaml, desired_state, default_timeout=default_timeout)
e2e/util/k8s.py:117: in create_benchmark
    self.crd_client.create_namespaced_custom_object(
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:225: in create_namespaced_custom_object
    return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs)  # noqa: E501
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:344: in create_namespaced_custom_object_with_http_info
    return self.api_client.call_api(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:348: in call_api
    return self.__call_api(resource_path, method,
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:180: in __call_api
    response_data = self.request(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:391: in request
    return self.rest_client.POST(url,
venv/lib/python3.8/site-packages/kubernetes/client/rest.py:274: in POST
    return self.request("POST", url,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <kubernetes.client.rest.RESTClientObject object at 0x7f7de35b5a30>
method = 'POST'
url = 'https://api.jenkins-ci.perf-testing.devcluster.openshift.com:6443/apis/ripsaw.cloudbulldozer.io/v1alpha1/namespaces/my-ripsaw/benchmarks'
query_params = []
headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/12.0.0/python'}
body = {'apiVersion': 'ripsaw.cloudbulldozer.io/v1alpha1', 'kind': 'Benchmark', 'metadata': {'name': 'uperf-resources', 'name...00m', 'memory': '100Mi'}}, 'hostnetwork': True, 'multus': {'enabled': False}, 'nthrs': [1, 2], ...}, 'name': 'uperf'}}}
post_params = {}, _preload_content = True, _request_timeout = None

    def request(self, method, url, query_params=None, headers=None,
                body=None, post_params=None, _preload_content=True,
                _request_timeout=None):
        """Perform requests.
    
        :param method: http request method
        :param url: http request url
        :param query_params: query parameters in the url
        :param headers: http request headers
        :param body: request json body, for `application/json`
        :param post_params: request post parameters,
                            `application/x-www-form-urlencoded`
                            and `multipart/form-data`
        :param _preload_content: if False, the urllib3.HTTPResponse object will
                                 be returned without reading/decoding response
                                 data. Default is True.
        :param _request_timeout: timeout setting for this request. If one
                                 number provided, it will be total request
                                 timeout. It can also be a pair (tuple) of
                                 (connection, read) timeouts.
        """
        method = method.upper()
        assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT',
                          'PATCH', 'OPTIONS']
    
        if post_params and body:
            raise ApiValueError(
                "body parameter cannot be used with post_params parameter."
            )
    
        post_params = post_params or {}
        headers = headers or {}
    
        timeout = None
        if _request_timeout:
            if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)):  # noqa: E501,F821
                timeout = urllib3.Timeout(total=_request_timeout)
            elif (isinstance(_request_timeout, tuple) and
                  len(_request_timeout) == 2):
                timeout = urllib3.Timeout(
                    connect=_request_timeout[0], read=_request_timeout[1])
    
        if 'Content-Type' not in headers:
            headers['Content-Type'] = 'application/json'
    
        try:
            # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE`
            if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']:
                if query_params:
                    url += '?' + urlencode(query_params)
                if re.search('json', headers['Content-Type'], re.IGNORECASE):
                    if headers['Content-Type'] == 'application/json-patch+json':
                        if not isinstance(body, list):
                            headers['Content-Type'] = \
                                'application/strategic-merge-patch+json'
                    request_body = None
                    if body is not None:
                        request_body = json.dumps(body)
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'application/x-www-form-urlencoded':  # noqa: E501
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=False,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'multipart/form-data':
                    # must del headers['Content-Type'], or the correct
                    # Content-Type which generated by urllib3 will be
                    # overwritten.
                    del headers['Content-Type']
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=True,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                # Pass a `string` parameter directly in the body to support
                # other content types than Json when `body` argument is
                # provided in serialized form
                elif isinstance(body, str) or isinstance(body, bytes):
                    request_body = body
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                else:
                    # Cannot generate the request from given parameters
                    msg = """Cannot prepare a request message for provided
                             arguments. Please check that your arguments match
                             declared content type."""
                    raise ApiException(status=0, reason=msg)
            # For `GET`, `HEAD`
            else:
                r = self.pool_manager.request(method, url,
                                              fields=query_params,
                                              preload_content=_preload_content,
                                              timeout=timeout,
                                              headers=headers)
        except urllib3.exceptions.SSLError as e:
            msg = "{0}\n{1}".format(type(e).__name__, str(e))
            raise ApiException(status=0, reason=msg)
    
        if _preload_content:
            r = RESTResponse(r)
    
            # In the python 3, the response.data is bytes.
            # we need to decode it to string.
            if six.PY3:
                r.data = r.data.decode('utf8')
    
            # log response body
            logger.debug("response body: %s", r.data)
    
        if not 200 <= r.status <= 299:
>           raise ApiException(http_resp=r)
E           kubernetes.client.exceptions.ApiException: (403)
E           Reason: Forbidden
E           HTTP response headers: HTTPHeaderDict({'Audit-Id': '10f0c955-c352-49a9-a428-ccd790054913', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '682d9336-3769-4f6e-8f0f-2f36581dea79', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'fb2241dd-f676-4aab-969d-77ab112dfaf0', 'Date': 'Thu, 19 Nov 2020 20:16:50 GMT', 'Content-Length': '484'})
E           HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"benchmarks.ripsaw.cloudbulldozer.io \"uperf-resources\" is forbidden: unable to create new content in namespace my-ripsaw because it is being terminated","reason":"Forbidden","details":{"name":"uperf-resources","group":"ripsaw.cloudbulldozer.io","kind":"benchmarks","causes":[{"reason":"NamespaceTerminating","message":"namespace my-ripsaw is being terminated","field":"metadata.namespace"}]},"code":403}

venv/lib/python3.8/site-packages/kubernetes/client/rest.py:233: ApiException

e2e/tests/test_ycsb.py

TestYcsb.test_ycsb[base] 0.00s ⏱

[gw2] linux -- Python 3.8.5 /var/lib/jenkins/workspace/ec2_jjb/venv/bin/python3

self = <test_ycsb.TestYcsb object at 0x7fee53cf7df0>
run = <models.benchmark_run.BenchmarkRun object at 0x7fee5814e430>

    def test_ycsb(self, run):
>       self.run_and_check_benchmark(run, desired_running_state="Running Load", desired_complete_state="Workload Complete")

e2e/tests/test_ycsb.py:12: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
e2e/models/test_base.py:92: in run_and_check_benchmark
    run.start(desired_state=desired_running_state, default_timeout=default_timeout)
e2e/models/benchmark_run.py:22: in start
    self.metadata = self.cluster.create_benchmark(self.yaml, desired_state, default_timeout=default_timeout)
e2e/util/k8s.py:117: in create_benchmark
    self.crd_client.create_namespaced_custom_object(
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:225: in create_namespaced_custom_object
    return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs)  # noqa: E501
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:344: in create_namespaced_custom_object_with_http_info
    return self.api_client.call_api(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:348: in call_api
    return self.__call_api(resource_path, method,
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:180: in __call_api
    response_data = self.request(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:391: in request
    return self.rest_client.POST(url,
venv/lib/python3.8/site-packages/kubernetes/client/rest.py:274: in POST
    return self.request("POST", url,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <kubernetes.client.rest.RESTClientObject object at 0x7fee5814e340>
method = 'POST'
url = 'https://api.jenkins-ci.perf-testing.devcluster.openshift.com:6443/apis/ripsaw.cloudbulldozer.io/v1alpha1/namespaces/my-ripsaw/benchmarks'
query_params = []
headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/12.0.0/python'}
body = {'apiVersion': 'ripsaw.cloudbulldozer.io/v1alpha1', 'kind': 'Benchmark', 'metadata': {'name': 'ycsb-base', 'namespace'...godb', 'operationcount': 100, 'options_load': '-p mongodb.url="mongodb://mongo-0.mongo/ycsb?"', ...}, 'name': 'ycsb'}}}
post_params = {}, _preload_content = True, _request_timeout = None

    def request(self, method, url, query_params=None, headers=None,
                body=None, post_params=None, _preload_content=True,
                _request_timeout=None):
        """Perform requests.
    
        :param method: http request method
        :param url: http request url
        :param query_params: query parameters in the url
        :param headers: http request headers
        :param body: request json body, for `application/json`
        :param post_params: request post parameters,
                            `application/x-www-form-urlencoded`
                            and `multipart/form-data`
        :param _preload_content: if False, the urllib3.HTTPResponse object will
                                 be returned without reading/decoding response
                                 data. Default is True.
        :param _request_timeout: timeout setting for this request. If one
                                 number provided, it will be total request
                                 timeout. It can also be a pair (tuple) of
                                 (connection, read) timeouts.
        """
        method = method.upper()
        assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT',
                          'PATCH', 'OPTIONS']
    
        if post_params and body:
            raise ApiValueError(
                "body parameter cannot be used with post_params parameter."
            )
    
        post_params = post_params or {}
        headers = headers or {}
    
        timeout = None
        if _request_timeout:
            if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)):  # noqa: E501,F821
                timeout = urllib3.Timeout(total=_request_timeout)
            elif (isinstance(_request_timeout, tuple) and
                  len(_request_timeout) == 2):
                timeout = urllib3.Timeout(
                    connect=_request_timeout[0], read=_request_timeout[1])
    
        if 'Content-Type' not in headers:
            headers['Content-Type'] = 'application/json'
    
        try:
            # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE`
            if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']:
                if query_params:
                    url += '?' + urlencode(query_params)
                if re.search('json', headers['Content-Type'], re.IGNORECASE):
                    if headers['Content-Type'] == 'application/json-patch+json':
                        if not isinstance(body, list):
                            headers['Content-Type'] = \
                                'application/strategic-merge-patch+json'
                    request_body = None
                    if body is not None:
                        request_body = json.dumps(body)
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'application/x-www-form-urlencoded':  # noqa: E501
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=False,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'multipart/form-data':
                    # must del headers['Content-Type'], or the correct
                    # Content-Type which generated by urllib3 will be
                    # overwritten.
                    del headers['Content-Type']
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=True,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                # Pass a `string` parameter directly in the body to support
                # other content types than Json when `body` argument is
                # provided in serialized form
                elif isinstance(body, str) or isinstance(body, bytes):
                    request_body = body
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                else:
                    # Cannot generate the request from given parameters
                    msg = """Cannot prepare a request message for provided
                             arguments. Please check that your arguments match
                             declared content type."""
                    raise ApiException(status=0, reason=msg)
            # For `GET`, `HEAD`
            else:
                r = self.pool_manager.request(method, url,
                                              fields=query_params,
                                              preload_content=_preload_content,
                                              timeout=timeout,
                                              headers=headers)
        except urllib3.exceptions.SSLError as e:
            msg = "{0}\n{1}".format(type(e).__name__, str(e))
            raise ApiException(status=0, reason=msg)
    
        if _preload_content:
            r = RESTResponse(r)
    
            # In the python 3, the response.data is bytes.
            # we need to decode it to string.
            if six.PY3:
                r.data = r.data.decode('utf8')
    
            # log response body
            logger.debug("response body: %s", r.data)
    
        if not 200 <= r.status <= 299:
>           raise ApiException(http_resp=r)
E           kubernetes.client.exceptions.ApiException: (403)
E           Reason: Forbidden
E           HTTP response headers: HTTPHeaderDict({'Audit-Id': '01867a67-12d9-4eb8-b8f5-46fe2ea3e62d', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '682d9336-3769-4f6e-8f0f-2f36581dea79', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'fb2241dd-f676-4aab-969d-77ab112dfaf0', 'Date': 'Thu, 19 Nov 2020 20:16:47 GMT', 'Content-Length': '472'})
E           HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"benchmarks.ripsaw.cloudbulldozer.io \"ycsb-base\" is forbidden: unable to create new content in namespace my-ripsaw because it is being terminated","reason":"Forbidden","details":{"name":"ycsb-base","group":"ripsaw.cloudbulldozer.io","kind":"benchmarks","causes":[{"reason":"NamespaceTerminating","message":"namespace my-ripsaw is being terminated","field":"metadata.namespace"}]},"code":403}

venv/lib/python3.8/site-packages/kubernetes/client/rest.py:233: ApiException

e2e/tests/test_scale_openshift.py

TestScale.test_scale[up] 0.00s ⏱

[gw0] linux -- Python 3.8.5 /var/lib/jenkins/workspace/ec2_jjb/venv/bin/python3

self = <test_scale_openshift.TestScale object at 0x7f247bf485b0>
run = <models.benchmark_run.BenchmarkRun object at 0x7f247bf97160>

    def test_scale(self, run):
>       self.run_and_check_benchmark(run, desired_running_state="Scaling Cluster")

e2e/tests/test_scale_openshift.py:12: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
e2e/models/test_base.py:92: in run_and_check_benchmark
    run.start(desired_state=desired_running_state, default_timeout=default_timeout)
e2e/models/benchmark_run.py:22: in start
    self.metadata = self.cluster.create_benchmark(self.yaml, desired_state, default_timeout=default_timeout)
e2e/util/k8s.py:117: in create_benchmark
    self.crd_client.create_namespaced_custom_object(
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:225: in create_namespaced_custom_object
    return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs)  # noqa: E501
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:344: in create_namespaced_custom_object_with_http_info
    return self.api_client.call_api(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:348: in call_api
    return self.__call_api(resource_path, method,
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:180: in __call_api
    response_data = self.request(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:391: in request
    return self.rest_client.POST(url,
venv/lib/python3.8/site-packages/kubernetes/client/rest.py:274: in POST
    return self.request("POST", url,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <kubernetes.client.rest.RESTClientObject object at 0x7f2480051c40>
method = 'POST'
url = 'https://api.jenkins-ci.perf-testing.devcluster.openshift.com:6443/apis/ripsaw.cloudbulldozer.io/v1alpha1/namespaces/my-ripsaw/benchmarks'
query_params = []
headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/12.0.0/python'}
body = {'apiVersion': 'ripsaw.cloudbulldozer.io/v1alpha1', 'kind': 'Benchmark', 'metadata': {'name': 'scale-up', 'namespace':... True}, 'workload': {'args': {'poll_interval': 2, 'scale': 1, 'serviceaccount': 'scaler'}, 'name': 'scale_openshift'}}}
post_params = {}, _preload_content = True, _request_timeout = None

    def request(self, method, url, query_params=None, headers=None,
                body=None, post_params=None, _preload_content=True,
                _request_timeout=None):
        """Perform requests.
    
        :param method: http request method
        :param url: http request url
        :param query_params: query parameters in the url
        :param headers: http request headers
        :param body: request json body, for `application/json`
        :param post_params: request post parameters,
                            `application/x-www-form-urlencoded`
                            and `multipart/form-data`
        :param _preload_content: if False, the urllib3.HTTPResponse object will
                                 be returned without reading/decoding response
                                 data. Default is True.
        :param _request_timeout: timeout setting for this request. If one
                                 number provided, it will be total request
                                 timeout. It can also be a pair (tuple) of
                                 (connection, read) timeouts.
        """
        method = method.upper()
        assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT',
                          'PATCH', 'OPTIONS']
    
        if post_params and body:
            raise ApiValueError(
                "body parameter cannot be used with post_params parameter."
            )
    
        post_params = post_params or {}
        headers = headers or {}
    
        timeout = None
        if _request_timeout:
            if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)):  # noqa: E501,F821
                timeout = urllib3.Timeout(total=_request_timeout)
            elif (isinstance(_request_timeout, tuple) and
                  len(_request_timeout) == 2):
                timeout = urllib3.Timeout(
                    connect=_request_timeout[0], read=_request_timeout[1])
    
        if 'Content-Type' not in headers:
            headers['Content-Type'] = 'application/json'
    
        try:
            # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE`
            if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']:
                if query_params:
                    url += '?' + urlencode(query_params)
                if re.search('json', headers['Content-Type'], re.IGNORECASE):
                    if headers['Content-Type'] == 'application/json-patch+json':
                        if not isinstance(body, list):
                            headers['Content-Type'] = \
                                'application/strategic-merge-patch+json'
                    request_body = None
                    if body is not None:
                        request_body = json.dumps(body)
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'application/x-www-form-urlencoded':  # noqa: E501
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=False,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'multipart/form-data':
                    # must del headers['Content-Type'], or the correct
                    # Content-Type which generated by urllib3 will be
                    # overwritten.
                    del headers['Content-Type']
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=True,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                # Pass a `string` parameter directly in the body to support
                # other content types than Json when `body` argument is
                # provided in serialized form
                elif isinstance(body, str) or isinstance(body, bytes):
                    request_body = body
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                else:
                    # Cannot generate the request from given parameters
                    msg = """Cannot prepare a request message for provided
                             arguments. Please check that your arguments match
                             declared content type."""
                    raise ApiException(status=0, reason=msg)
            # For `GET`, `HEAD`
            else:
                r = self.pool_manager.request(method, url,
                                              fields=query_params,
                                              preload_content=_preload_content,
                                              timeout=timeout,
                                              headers=headers)
        except urllib3.exceptions.SSLError as e:
            msg = "{0}\n{1}".format(type(e).__name__, str(e))
            raise ApiException(status=0, reason=msg)
    
        if _preload_content:
            r = RESTResponse(r)
    
            # In the python 3, the response.data is bytes.
            # we need to decode it to string.
            if six.PY3:
                r.data = r.data.decode('utf8')
    
            # log response body
            logger.debug("response body: %s", r.data)
    
        if not 200 <= r.status <= 299:
>           raise ApiException(http_resp=r)
E           kubernetes.client.exceptions.ApiException: (403)
E           Reason: Forbidden
E           HTTP response headers: HTTPHeaderDict({'Audit-Id': 'dab6115b-64f6-4949-b7f9-4f60f885b5f3', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '682d9336-3769-4f6e-8f0f-2f36581dea79', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'fb2241dd-f676-4aab-969d-77ab112dfaf0', 'Date': 'Thu, 19 Nov 2020 20:16:48 GMT', 'Content-Length': '470'})
E           HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"benchmarks.ripsaw.cloudbulldozer.io \"scale-up\" is forbidden: unable to create new content in namespace my-ripsaw because it is being terminated","reason":"Forbidden","details":{"name":"scale-up","group":"ripsaw.cloudbulldozer.io","kind":"benchmarks","causes":[{"reason":"NamespaceTerminating","message":"namespace my-ripsaw is being terminated","field":"metadata.namespace"}]},"code":403}

venv/lib/python3.8/site-packages/kubernetes/client/rest.py:233: ApiException

13 passed 😃

e2e/tests/test_backpack.py

Testbackpack.test_backpack[init] 87.39s ⏱

e2e/tests/test_byowl.py

TestByowl.test_byowl[base] 99.27s ⏱

e2e/tests/test_backpack_ds.py

TestBackpackDaemonSet.test_backpack_daemonset[daemonset] 72.41s ⏱

e2e/tests/test_fs_drift.py

TestFSDrift.test_fs_drift[hostpath] 135.42s ⏱

TestFSDrift.test_fs_drift[base] 117.60s ⏱

e2e/tests/test_hammerdb.py

TestHammerDB.test_hammerdb[base] 265.06s ⏱

e2e/tests/test_fiod.py

TestFiod.test_fiod[bsrange] 318.96s ⏱

e2e/tests/test_kube_burner.py

TestKubeBurner.test_kube_burner[kubelet-density-metics] 144.42s ⏱

TestKubeBurner.test_kube_burner[kubelet-density-heavy-metrics] 174.51s ⏱

TestKubeBurner.test_kube_burner[cluster-density-metrics-aggregated] 228.77s ⏱

e2e/tests/test_iperf3.py

TestIPerf3.test_iperf3[hostnetwork] 192.61s ⏱

TestIPerf3.test_iperf3[base] 177.52s ⏱

e2e/tests/test_pgbench.py

TestPgBench.test_pgbench[base] 252.86s ⏱

3 skipped 🙄

e2e/tests/test_stressng.py

TestStressng.test_stressng[base] 0.00s ⏱

e2e/tests/test_service_mesh.py

TestServiceMesh.test_servicemesh[servicemesh] 0.00s ⏱

e2e/tests/test_scale_openshift.py

TestScale.test_scale[down] 0.00s ⏱

whitleykeith · 2020-11-19T20:45:28Z

/rerun all

comet-perf-ci · 2020-11-23T11:25:50Z

Test Report

Report generated on 19-Nov-2020 at 20:57:26 by pytest-md 📝

Summary

13 tests ran in 456.07 seconds ⏱

8 failed 😰
2 passed 😃
3 skipped 🙄

8 failed 😰

e2e/tests/test_fiod.py

TestFiod.test_fiod[hostpath] 0.00s ⏱

self = <test_fiod.TestFiod object at 0x7f3825dc7ee0>
run = <models.benchmark_run.BenchmarkRun object at 0x7f38264be5e0>

    def test_fiod(self, run):
>       self.run_and_check_benchmark(run)

e2e/tests/test_fiod.py:42: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
e2e/models/test_base.py:92: in run_and_check_benchmark
    run.start(desired_state=desired_running_state, default_timeout=default_timeout)
e2e/models/benchmark_run.py:22: in start
    self.metadata = self.cluster.create_benchmark(self.yaml, desired_state, default_timeout=default_timeout)
e2e/util/k8s.py:117: in create_benchmark
    self.crd_client.create_namespaced_custom_object(
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:225: in create_namespaced_custom_object
    return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs)  # noqa: E501
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:344: in create_namespaced_custom_object_with_http_info
    return self.api_client.call_api(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:348: in call_api
    return self.__call_api(resource_path, method,
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:180: in __call_api
    response_data = self.request(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:391: in request
    return self.rest_client.POST(url,
venv/lib/python3.8/site-packages/kubernetes/client/rest.py:274: in POST
    return self.request("POST", url,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <kubernetes.client.rest.RESTClientObject object at 0x7f382636f580>
method = 'POST'
url = 'https://api.jenkins-ci.perf-testing.devcluster.openshift.com:6443/apis/ripsaw.cloudbulldozer.io/v1alpha1/namespaces/my-ripsaw/benchmarks'
query_params = []
headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/12.0.0/python'}
body = {'apiVersion': 'ripsaw.cloudbulldozer.io/v1alpha1', 'kind': 'Benchmark', 'metadata': {'name': 'fiod-hostpath', 'namesp...e_based=1', 'runtime={{ workload_args.write_runtime }}', 'ramp_time={{ workload_args.write_ramp_time }}']}, ...], ...}}
post_params = {}, _preload_content = True, _request_timeout = None

    def request(self, method, url, query_params=None, headers=None,
                body=None, post_params=None, _preload_content=True,
                _request_timeout=None):
        """Perform requests.
    
        :param method: http request method
        :param url: http request url
        :param query_params: query parameters in the url
        :param headers: http request headers
        :param body: request json body, for `application/json`
        :param post_params: request post parameters,
                            `application/x-www-form-urlencoded`
                            and `multipart/form-data`
        :param _preload_content: if False, the urllib3.HTTPResponse object will
                                 be returned without reading/decoding response
                                 data. Default is True.
        :param _request_timeout: timeout setting for this request. If one
                                 number provided, it will be total request
                                 timeout. It can also be a pair (tuple) of
                                 (connection, read) timeouts.
        """
        method = method.upper()
        assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT',
                          'PATCH', 'OPTIONS']
    
        if post_params and body:
            raise ApiValueError(
                "body parameter cannot be used with post_params parameter."
            )
    
        post_params = post_params or {}
        headers = headers or {}
    
        timeout = None
        if _request_timeout:
            if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)):  # noqa: E501,F821
                timeout = urllib3.Timeout(total=_request_timeout)
            elif (isinstance(_request_timeout, tuple) and
                  len(_request_timeout) == 2):
                timeout = urllib3.Timeout(
                    connect=_request_timeout[0], read=_request_timeout[1])
    
        if 'Content-Type' not in headers:
            headers['Content-Type'] = 'application/json'
    
        try:
            # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE`
            if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']:
                if query_params:
                    url += '?' + urlencode(query_params)
                if re.search('json', headers['Content-Type'], re.IGNORECASE):
                    if headers['Content-Type'] == 'application/json-patch+json':
                        if not isinstance(body, list):
                            headers['Content-Type'] = \
                                'application/strategic-merge-patch+json'
                    request_body = None
                    if body is not None:
                        request_body = json.dumps(body)
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'application/x-www-form-urlencoded':  # noqa: E501
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=False,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'multipart/form-data':
                    # must del headers['Content-Type'], or the correct
                    # Content-Type which generated by urllib3 will be
                    # overwritten.
                    del headers['Content-Type']
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=True,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                # Pass a `string` parameter directly in the body to support
                # other content types than Json when `body` argument is
                # provided in serialized form
                elif isinstance(body, str) or isinstance(body, bytes):
                    request_body = body
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                else:
                    # Cannot generate the request from given parameters
                    msg = """Cannot prepare a request message for provided
                             arguments. Please check that your arguments match
                             declared content type."""
                    raise ApiException(status=0, reason=msg)
            # For `GET`, `HEAD`
            else:
                r = self.pool_manager.request(method, url,
                                              fields=query_params,
                                              preload_content=_preload_content,
                                              timeout=timeout,
                                              headers=headers)
        except urllib3.exceptions.SSLError as e:
            msg = "{0}\n{1}".format(type(e).__name__, str(e))
            raise ApiException(status=0, reason=msg)
    
        if _preload_content:
            r = RESTResponse(r)
    
            # In the python 3, the response.data is bytes.
            # we need to decode it to string.
            if six.PY3:
                r.data = r.data.decode('utf8')
    
            # log response body
            logger.debug("response body: %s", r.data)
    
        if not 200 <= r.status <= 299:
>           raise ApiException(http_resp=r)
E           kubernetes.client.exceptions.ApiException: (403)
E           Reason: Forbidden
E           HTTP response headers: HTTPHeaderDict({'Audit-Id': '58cfa482-7afa-40a3-beb7-40995c4910fb', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '682d9336-3769-4f6e-8f0f-2f36581dea79', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'fb2241dd-f676-4aab-969d-77ab112dfaf0', 'Date': 'Thu, 19 Nov 2020 20:57:09 GMT', 'Content-Length': '480'})
E           HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"benchmarks.ripsaw.cloudbulldozer.io \"fiod-hostpath\" is forbidden: unable to create new content in namespace my-ripsaw because it is being terminated","reason":"Forbidden","details":{"name":"fiod-hostpath","group":"ripsaw.cloudbulldozer.io","kind":"benchmarks","causes":[{"reason":"NamespaceTerminating","message":"namespace my-ripsaw is being terminated","field":"metadata.namespace"}]},"code":403}

venv/lib/python3.8/site-packages/kubernetes/client/rest.py:233: ApiException

TestFiod.test_fiod[base] 0.00s ⏱

self = <test_fiod.TestFiod object at 0x7f3825dc2070>
run = <models.benchmark_run.BenchmarkRun object at 0x7f382668dfa0>

    def test_fiod(self, run):
>       self.run_and_check_benchmark(run)

e2e/tests/test_fiod.py:42: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
e2e/models/test_base.py:92: in run_and_check_benchmark
    run.start(desired_state=desired_running_state, default_timeout=default_timeout)
e2e/models/benchmark_run.py:22: in start
    self.metadata = self.cluster.create_benchmark(self.yaml, desired_state, default_timeout=default_timeout)
e2e/util/k8s.py:117: in create_benchmark
    self.crd_client.create_namespaced_custom_object(
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:225: in create_namespaced_custom_object
    return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs)  # noqa: E501
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:344: in create_namespaced_custom_object_with_http_info
    return self.api_client.call_api(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:348: in call_api
    return self.__call_api(resource_path, method,
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:180: in __call_api
    response_data = self.request(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:391: in request
    return self.rest_client.POST(url,
venv/lib/python3.8/site-packages/kubernetes/client/rest.py:274: in POST
    return self.request("POST", url,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <kubernetes.client.rest.RESTClientObject object at 0x7f382636f580>
method = 'POST'
url = 'https://api.jenkins-ci.perf-testing.devcluster.openshift.com:6443/apis/ripsaw.cloudbulldozer.io/v1alpha1/namespaces/my-ripsaw/benchmarks'
query_params = []
headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/12.0.0/python'}
body = {'apiVersion': 'ripsaw.cloudbulldozer.io/v1alpha1', 'kind': 'Benchmark', 'metadata': {'name': 'fiod-base', 'namespace'...rgs.write_runtime }}', 'ramp_time={{ workload_args.write_ramp_time }}']}, ...], 'metadata': {'collection': True}, ...}}
post_params = {}, _preload_content = True, _request_timeout = None

    def request(self, method, url, query_params=None, headers=None,
                body=None, post_params=None, _preload_content=True,
                _request_timeout=None):
        """Perform requests.
    
        :param method: http request method
        :param url: http request url
        :param query_params: query parameters in the url
        :param headers: http request headers
        :param body: request json body, for `application/json`
        :param post_params: request post parameters,
                            `application/x-www-form-urlencoded`
                            and `multipart/form-data`
        :param _preload_content: if False, the urllib3.HTTPResponse object will
                                 be returned without reading/decoding response
                                 data. Default is True.
        :param _request_timeout: timeout setting for this request. If one
                                 number provided, it will be total request
                                 timeout. It can also be a pair (tuple) of
                                 (connection, read) timeouts.
        """
        method = method.upper()
        assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT',
                          'PATCH', 'OPTIONS']
    
        if post_params and body:
            raise ApiValueError(
                "body parameter cannot be used with post_params parameter."
            )
    
        post_params = post_params or {}
        headers = headers or {}
    
        timeout = None
        if _request_timeout:
            if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)):  # noqa: E501,F821
                timeout = urllib3.Timeout(total=_request_timeout)
            elif (isinstance(_request_timeout, tuple) and
                  len(_request_timeout) == 2):
                timeout = urllib3.Timeout(
                    connect=_request_timeout[0], read=_request_timeout[1])
    
        if 'Content-Type' not in headers:
            headers['Content-Type'] = 'application/json'
    
        try:
            # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE`
            if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']:
                if query_params:
                    url += '?' + urlencode(query_params)
                if re.search('json', headers['Content-Type'], re.IGNORECASE):
                    if headers['Content-Type'] == 'application/json-patch+json':
                        if not isinstance(body, list):
                            headers['Content-Type'] = \
                                'application/strategic-merge-patch+json'
                    request_body = None
                    if body is not None:
                        request_body = json.dumps(body)
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'application/x-www-form-urlencoded':  # noqa: E501
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=False,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'multipart/form-data':
                    # must del headers['Content-Type'], or the correct
                    # Content-Type which generated by urllib3 will be
                    # overwritten.
                    del headers['Content-Type']
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=True,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                # Pass a `string` parameter directly in the body to support
                # other content types than Json when `body` argument is
                # provided in serialized form
                elif isinstance(body, str) or isinstance(body, bytes):
                    request_body = body
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                else:
                    # Cannot generate the request from given parameters
                    msg = """Cannot prepare a request message for provided
                             arguments. Please check that your arguments match
                             declared content type."""
                    raise ApiException(status=0, reason=msg)
            # For `GET`, `HEAD`
            else:
                r = self.pool_manager.request(method, url,
                                              fields=query_params,
                                              preload_content=_preload_content,
                                              timeout=timeout,
                                              headers=headers)
        except urllib3.exceptions.SSLError as e:
            msg = "{0}\n{1}".format(type(e).__name__, str(e))
            raise ApiException(status=0, reason=msg)
    
        if _preload_content:
            r = RESTResponse(r)
    
            # In the python 3, the response.data is bytes.
            # we need to decode it to string.
            if six.PY3:
                r.data = r.data.decode('utf8')
    
            # log response body
            logger.debug("response body: %s", r.data)
    
        if not 200 <= r.status <= 299:
>           raise ApiException(http_resp=r)
E           kubernetes.client.exceptions.ApiException: (403)
E           Reason: Forbidden
E           HTTP response headers: HTTPHeaderDict({'Audit-Id': '14d27055-28e1-432b-993f-fe6ed57adf07', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '682d9336-3769-4f6e-8f0f-2f36581dea79', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'fb2241dd-f676-4aab-969d-77ab112dfaf0', 'Date': 'Thu, 19 Nov 2020 20:57:13 GMT', 'Content-Length': '472'})
E           HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"benchmarks.ripsaw.cloudbulldozer.io \"fiod-base\" is forbidden: unable to create new content in namespace my-ripsaw because it is being terminated","reason":"Forbidden","details":{"name":"fiod-base","group":"ripsaw.cloudbulldozer.io","kind":"benchmarks","causes":[{"reason":"NamespaceTerminating","message":"namespace my-ripsaw is being terminated","field":"metadata.namespace"}]},"code":403}

venv/lib/python3.8/site-packages/kubernetes/client/rest.py:233: ApiException

e2e/tests/test_uperf.py

TestUperf.test_uperf[uperf_serviceip] 0.00s ⏱

self = <test_uperf.TestUperf object at 0x7f38260b2820>
run = <models.benchmark_run.BenchmarkRun object at 0x7f38262f8c10>

    def test_uperf(self, run):
>       self.run_and_check_benchmark(run)

e2e/tests/test_uperf.py:12: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
e2e/models/test_base.py:92: in run_and_check_benchmark
    run.start(desired_state=desired_running_state, default_timeout=default_timeout)
e2e/models/benchmark_run.py:22: in start
    self.metadata = self.cluster.create_benchmark(self.yaml, desired_state, default_timeout=default_timeout)
e2e/util/k8s.py:117: in create_benchmark
    self.crd_client.create_namespaced_custom_object(
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:225: in create_namespaced_custom_object
    return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs)  # noqa: E501
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:344: in create_namespaced_custom_object_with_http_info
    return self.api_client.call_api(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:348: in call_api
    return self.__call_api(resource_path, method,
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:180: in __call_api
    response_data = self.request(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:391: in request
    return self.rest_client.POST(url,
venv/lib/python3.8/site-packages/kubernetes/client/rest.py:274: in POST
    return self.request("POST", url,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <kubernetes.client.rest.RESTClientObject object at 0x7f38262f8910>
method = 'POST'
url = 'https://api.jenkins-ci.perf-testing.devcluster.openshift.com:6443/apis/ripsaw.cloudbulldozer.io/v1alpha1/namespaces/my-ripsaw/benchmarks'
query_params = []
headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/12.0.0/python'}
body = {'apiVersion': 'ripsaw.cloudbulldozer.io/v1alpha1', 'kind': 'Benchmark', 'metadata': {'name': 'uperf-serviceip', 'name...ad': {'args': {'hostnetwork': False, 'multus': {'enabled': False}, 'nthrs': [1, 2], 'pair': 1, ...}, 'name': 'uperf'}}}
post_params = {}, _preload_content = True, _request_timeout = None

    def request(self, method, url, query_params=None, headers=None,
                body=None, post_params=None, _preload_content=True,
                _request_timeout=None):
        """Perform requests.
    
        :param method: http request method
        :param url: http request url
        :param query_params: query parameters in the url
        :param headers: http request headers
        :param body: request json body, for `application/json`
        :param post_params: request post parameters,
                            `application/x-www-form-urlencoded`
                            and `multipart/form-data`
        :param _preload_content: if False, the urllib3.HTTPResponse object will
                                 be returned without reading/decoding response
                                 data. Default is True.
        :param _request_timeout: timeout setting for this request. If one
                                 number provided, it will be total request
                                 timeout. It can also be a pair (tuple) of
                                 (connection, read) timeouts.
        """
        method = method.upper()
        assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT',
                          'PATCH', 'OPTIONS']
    
        if post_params and body:
            raise ApiValueError(
                "body parameter cannot be used with post_params parameter."
            )
    
        post_params = post_params or {}
        headers = headers or {}
    
        timeout = None
        if _request_timeout:
            if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)):  # noqa: E501,F821
                timeout = urllib3.Timeout(total=_request_timeout)
            elif (isinstance(_request_timeout, tuple) and
                  len(_request_timeout) == 2):
                timeout = urllib3.Timeout(
                    connect=_request_timeout[0], read=_request_timeout[1])
    
        if 'Content-Type' not in headers:
            headers['Content-Type'] = 'application/json'
    
        try:
            # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE`
            if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']:
                if query_params:
                    url += '?' + urlencode(query_params)
                if re.search('json', headers['Content-Type'], re.IGNORECASE):
                    if headers['Content-Type'] == 'application/json-patch+json':
                        if not isinstance(body, list):
                            headers['Content-Type'] = \
                                'application/strategic-merge-patch+json'
                    request_body = None
                    if body is not None:
                        request_body = json.dumps(body)
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'application/x-www-form-urlencoded':  # noqa: E501
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=False,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'multipart/form-data':
                    # must del headers['Content-Type'], or the correct
                    # Content-Type which generated by urllib3 will be
                    # overwritten.
                    del headers['Content-Type']
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=True,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                # Pass a `string` parameter directly in the body to support
                # other content types than Json when `body` argument is
                # provided in serialized form
                elif isinstance(body, str) or isinstance(body, bytes):
                    request_body = body
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                else:
                    # Cannot generate the request from given parameters
                    msg = """Cannot prepare a request message for provided
                             arguments. Please check that your arguments match
                             declared content type."""
                    raise ApiException(status=0, reason=msg)
            # For `GET`, `HEAD`
            else:
                r = self.pool_manager.request(method, url,
                                              fields=query_params,
                                              preload_content=_preload_content,
                                              timeout=timeout,
                                              headers=headers)
        except urllib3.exceptions.SSLError as e:
            msg = "{0}\n{1}".format(type(e).__name__, str(e))
            raise ApiException(status=0, reason=msg)
    
        if _preload_content:
            r = RESTResponse(r)
    
            # In the python 3, the response.data is bytes.
            # we need to decode it to string.
            if six.PY3:
                r.data = r.data.decode('utf8')
    
            # log response body
            logger.debug("response body: %s", r.data)
    
        if not 200 <= r.status <= 299:
>           raise ApiException(http_resp=r)
E           kubernetes.client.exceptions.ApiException: (403)
E           Reason: Forbidden
E           HTTP response headers: HTTPHeaderDict({'Audit-Id': 'd67e1f5c-2332-4bc0-8849-3a71bf17592f', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '682d9336-3769-4f6e-8f0f-2f36581dea79', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'fb2241dd-f676-4aab-969d-77ab112dfaf0', 'Date': 'Thu, 19 Nov 2020 20:57:15 GMT', 'Content-Length': '484'})
E           HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"benchmarks.ripsaw.cloudbulldozer.io \"uperf-serviceip\" is forbidden: unable to create new content in namespace my-ripsaw because it is being terminated","reason":"Forbidden","details":{"name":"uperf-serviceip","group":"ripsaw.cloudbulldozer.io","kind":"benchmarks","causes":[{"reason":"NamespaceTerminating","message":"namespace my-ripsaw is being terminated","field":"metadata.namespace"}]},"code":403}

venv/lib/python3.8/site-packages/kubernetes/client/rest.py:233: ApiException

TestUperf.test_uperf[base] 0.00s ⏱

self = <test_uperf.TestUperf object at 0x7f3826164160>
run = <models.benchmark_run.BenchmarkRun object at 0x7f38263c6160>

    def test_uperf(self, run):
>       self.run_and_check_benchmark(run)

e2e/tests/test_uperf.py:12: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
e2e/models/test_base.py:92: in run_and_check_benchmark
    run.start(desired_state=desired_running_state, default_timeout=default_timeout)
e2e/models/benchmark_run.py:22: in start
    self.metadata = self.cluster.create_benchmark(self.yaml, desired_state, default_timeout=default_timeout)
e2e/util/k8s.py:117: in create_benchmark
    self.crd_client.create_namespaced_custom_object(
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:225: in create_namespaced_custom_object
    return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs)  # noqa: E501
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:344: in create_namespaced_custom_object_with_http_info
    return self.api_client.call_api(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:348: in call_api
    return self.__call_api(resource_path, method,
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:180: in __call_api
    response_data = self.request(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:391: in request
    return self.rest_client.POST(url,
venv/lib/python3.8/site-packages/kubernetes/client/rest.py:274: in POST
    return self.request("POST", url,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <kubernetes.client.rest.RESTClientObject object at 0x7f38262f8910>
method = 'POST'
url = 'https://api.jenkins-ci.perf-testing.devcluster.openshift.com:6443/apis/ripsaw.cloudbulldozer.io/v1alpha1/namespaces/my-ripsaw/benchmarks'
query_params = []
headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/12.0.0/python'}
body = {'apiVersion': 'ripsaw.cloudbulldozer.io/v1alpha1', 'kind': 'Benchmark', 'metadata': {'name': 'uperf-base', 'namespace...ad': {'args': {'hostnetwork': False, 'multus': {'enabled': False}, 'nthrs': [1, 2], 'pair': 1, ...}, 'name': 'uperf'}}}
post_params = {}, _preload_content = True, _request_timeout = None

    def request(self, method, url, query_params=None, headers=None,
                body=None, post_params=None, _preload_content=True,
                _request_timeout=None):
        """Perform requests.
    
        :param method: http request method
        :param url: http request url
        :param query_params: query parameters in the url
        :param headers: http request headers
        :param body: request json body, for `application/json`
        :param post_params: request post parameters,
                            `application/x-www-form-urlencoded`
                            and `multipart/form-data`
        :param _preload_content: if False, the urllib3.HTTPResponse object will
                                 be returned without reading/decoding response
                                 data. Default is True.
        :param _request_timeout: timeout setting for this request. If one
                                 number provided, it will be total request
                                 timeout. It can also be a pair (tuple) of
                                 (connection, read) timeouts.
        """
        method = method.upper()
        assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT',
                          'PATCH', 'OPTIONS']
    
        if post_params and body:
            raise ApiValueError(
                "body parameter cannot be used with post_params parameter."
            )
    
        post_params = post_params or {}
        headers = headers or {}
    
        timeout = None
        if _request_timeout:
            if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)):  # noqa: E501,F821
                timeout = urllib3.Timeout(total=_request_timeout)
            elif (isinstance(_request_timeout, tuple) and
                  len(_request_timeout) == 2):
                timeout = urllib3.Timeout(
                    connect=_request_timeout[0], read=_request_timeout[1])
    
        if 'Content-Type' not in headers:
            headers['Content-Type'] = 'application/json'
    
        try:
            # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE`
            if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']:
                if query_params:
                    url += '?' + urlencode(query_params)
                if re.search('json', headers['Content-Type'], re.IGNORECASE):
                    if headers['Content-Type'] == 'application/json-patch+json':
                        if not isinstance(body, list):
                            headers['Content-Type'] = \
                                'application/strategic-merge-patch+json'
                    request_body = None
                    if body is not None:
                        request_body = json.dumps(body)
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'application/x-www-form-urlencoded':  # noqa: E501
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=False,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'multipart/form-data':
                    # must del headers['Content-Type'], or the correct
                    # Content-Type which generated by urllib3 will be
                    # overwritten.
                    del headers['Content-Type']
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=True,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                # Pass a `string` parameter directly in the body to support
                # other content types than Json when `body` argument is
                # provided in serialized form
                elif isinstance(body, str) or isinstance(body, bytes):
                    request_body = body
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                else:
                    # Cannot generate the request from given parameters
                    msg = """Cannot prepare a request message for provided
                             arguments. Please check that your arguments match
                             declared content type."""
                    raise ApiException(status=0, reason=msg)
            # For `GET`, `HEAD`
            else:
                r = self.pool_manager.request(method, url,
                                              fields=query_params,
                                              preload_content=_preload_content,
                                              timeout=timeout,
                                              headers=headers)
        except urllib3.exceptions.SSLError as e:
            msg = "{0}\n{1}".format(type(e).__name__, str(e))
            raise ApiException(status=0, reason=msg)
    
        if _preload_content:
            r = RESTResponse(r)
    
            # In the python 3, the response.data is bytes.
            # we need to decode it to string.
            if six.PY3:
                r.data = r.data.decode('utf8')
    
            # log response body
            logger.debug("response body: %s", r.data)
    
        if not 200 <= r.status <= 299:
>           raise ApiException(http_resp=r)
E           kubernetes.client.exceptions.ApiException: (403)
E           Reason: Forbidden
E           HTTP response headers: HTTPHeaderDict({'Audit-Id': '4ef47f74-c31d-4992-89cd-223c02fa2921', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '682d9336-3769-4f6e-8f0f-2f36581dea79', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'fb2241dd-f676-4aab-969d-77ab112dfaf0', 'Date': 'Thu, 19 Nov 2020 20:57:17 GMT', 'Content-Length': '474'})
E           HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"benchmarks.ripsaw.cloudbulldozer.io \"uperf-base\" is forbidden: unable to create new content in namespace my-ripsaw because it is being terminated","reason":"Forbidden","details":{"name":"uperf-base","group":"ripsaw.cloudbulldozer.io","kind":"benchmarks","causes":[{"reason":"NamespaceTerminating","message":"namespace my-ripsaw is being terminated","field":"metadata.namespace"}]},"code":403}

venv/lib/python3.8/site-packages/kubernetes/client/rest.py:233: ApiException

TestUperf.test_uperf[uperf_networkpolicy] 0.00s ⏱

self = <test_uperf.TestUperf object at 0x7f3825f73790>
run = <models.benchmark_run.BenchmarkRun object at 0x7f38262d91c0>

    def test_uperf(self, run):
>       self.run_and_check_benchmark(run)

e2e/tests/test_uperf.py:12: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
e2e/models/test_base.py:92: in run_and_check_benchmark
    run.start(desired_state=desired_running_state, default_timeout=default_timeout)
e2e/models/benchmark_run.py:22: in start
    self.metadata = self.cluster.create_benchmark(self.yaml, desired_state, default_timeout=default_timeout)
e2e/util/k8s.py:117: in create_benchmark
    self.crd_client.create_namespaced_custom_object(
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:225: in create_namespaced_custom_object
    return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs)  # noqa: E501
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:344: in create_namespaced_custom_object_with_http_info
    return self.api_client.call_api(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:348: in call_api
    return self.__call_api(resource_path, method,
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:180: in __call_api
    response_data = self.request(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:391: in request
    return self.rest_client.POST(url,
venv/lib/python3.8/site-packages/kubernetes/client/rest.py:274: in POST
    return self.request("POST", url,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <kubernetes.client.rest.RESTClientObject object at 0x7f38262f8910>
method = 'POST'
url = 'https://api.jenkins-ci.perf-testing.devcluster.openshift.com:6443/apis/ripsaw.cloudbulldozer.io/v1alpha1/namespaces/my-ripsaw/benchmarks'
query_params = []
headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/12.0.0/python'}
body = {'apiVersion': 'ripsaw.cloudbulldozer.io/v1alpha1', 'kind': 'Benchmark', 'metadata': {'name': 'example-benchmark', 'na...: {'hostnetwork': False, 'multus': {'enabled': False}, 'networkpolicy': True, 'nthrs': [1, 2], ...}, 'name': 'uperf'}}}
post_params = {}, _preload_content = True, _request_timeout = None

    def request(self, method, url, query_params=None, headers=None,
                body=None, post_params=None, _preload_content=True,
                _request_timeout=None):
        """Perform requests.
    
        :param method: http request method
        :param url: http request url
        :param query_params: query parameters in the url
        :param headers: http request headers
        :param body: request json body, for `application/json`
        :param post_params: request post parameters,
                            `application/x-www-form-urlencoded`
                            and `multipart/form-data`
        :param _preload_content: if False, the urllib3.HTTPResponse object will
                                 be returned without reading/decoding response
                                 data. Default is True.
        :param _request_timeout: timeout setting for this request. If one
                                 number provided, it will be total request
                                 timeout. It can also be a pair (tuple) of
                                 (connection, read) timeouts.
        """
        method = method.upper()
        assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT',
                          'PATCH', 'OPTIONS']
    
        if post_params and body:
            raise ApiValueError(
                "body parameter cannot be used with post_params parameter."
            )
    
        post_params = post_params or {}
        headers = headers or {}
    
        timeout = None
        if _request_timeout:
            if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)):  # noqa: E501,F821
                timeout = urllib3.Timeout(total=_request_timeout)
            elif (isinstance(_request_timeout, tuple) and
                  len(_request_timeout) == 2):
                timeout = urllib3.Timeout(
                    connect=_request_timeout[0], read=_request_timeout[1])
    
        if 'Content-Type' not in headers:
            headers['Content-Type'] = 'application/json'
    
        try:
            # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE`
            if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']:
                if query_params:
                    url += '?' + urlencode(query_params)
                if re.search('json', headers['Content-Type'], re.IGNORECASE):
                    if headers['Content-Type'] == 'application/json-patch+json':
                        if not isinstance(body, list):
                            headers['Content-Type'] = \
                                'application/strategic-merge-patch+json'
                    request_body = None
                    if body is not None:
                        request_body = json.dumps(body)
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'application/x-www-form-urlencoded':  # noqa: E501
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=False,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'multipart/form-data':
                    # must del headers['Content-Type'], or the correct
                    # Content-Type which generated by urllib3 will be
                    # overwritten.
                    del headers['Content-Type']
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=True,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                # Pass a `string` parameter directly in the body to support
                # other content types than Json when `body` argument is
                # provided in serialized form
                elif isinstance(body, str) or isinstance(body, bytes):
                    request_body = body
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                else:
                    # Cannot generate the request from given parameters
                    msg = """Cannot prepare a request message for provided
                             arguments. Please check that your arguments match
                             declared content type."""
                    raise ApiException(status=0, reason=msg)
            # For `GET`, `HEAD`
            else:
                r = self.pool_manager.request(method, url,
                                              fields=query_params,
                                              preload_content=_preload_content,
                                              timeout=timeout,
                                              headers=headers)
        except urllib3.exceptions.SSLError as e:
            msg = "{0}\n{1}".format(type(e).__name__, str(e))
            raise ApiException(status=0, reason=msg)
    
        if _preload_content:
            r = RESTResponse(r)
    
            # In the python 3, the response.data is bytes.
            # we need to decode it to string.
            if six.PY3:
                r.data = r.data.decode('utf8')
    
            # log response body
            logger.debug("response body: %s", r.data)
    
        if not 200 <= r.status <= 299:
>           raise ApiException(http_resp=r)
E           kubernetes.client.exceptions.ApiException: (403)
E           Reason: Forbidden
E           HTTP response headers: HTTPHeaderDict({'Audit-Id': 'c24b67a4-b206-4c4c-9cea-6682ebd9e7e0', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '682d9336-3769-4f6e-8f0f-2f36581dea79', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'fb2241dd-f676-4aab-969d-77ab112dfaf0', 'Date': 'Thu, 19 Nov 2020 20:57:18 GMT', 'Content-Length': '488'})
E           HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"benchmarks.ripsaw.cloudbulldozer.io \"example-benchmark\" is forbidden: unable to create new content in namespace my-ripsaw because it is being terminated","reason":"Forbidden","details":{"name":"example-benchmark","group":"ripsaw.cloudbulldozer.io","kind":"benchmarks","causes":[{"reason":"NamespaceTerminating","message":"namespace my-ripsaw is being terminated","field":"metadata.namespace"}]},"code":403}

venv/lib/python3.8/site-packages/kubernetes/client/rest.py:233: ApiException

TestUperf.test_uperf[uperf_resources] 0.00s ⏱

self = <test_uperf.TestUperf object at 0x7f3826206e20>
run = <models.benchmark_run.BenchmarkRun object at 0x7f38262d9fd0>

    def test_uperf(self, run):
>       self.run_and_check_benchmark(run)

e2e/tests/test_uperf.py:12: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
e2e/models/test_base.py:92: in run_and_check_benchmark
    run.start(desired_state=desired_running_state, default_timeout=default_timeout)
e2e/models/benchmark_run.py:22: in start
    self.metadata = self.cluster.create_benchmark(self.yaml, desired_state, default_timeout=default_timeout)
e2e/util/k8s.py:117: in create_benchmark
    self.crd_client.create_namespaced_custom_object(
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:225: in create_namespaced_custom_object
    return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs)  # noqa: E501
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:344: in create_namespaced_custom_object_with_http_info
    return self.api_client.call_api(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:348: in call_api
    return self.__call_api(resource_path, method,
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:180: in __call_api
    response_data = self.request(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:391: in request
    return self.rest_client.POST(url,
venv/lib/python3.8/site-packages/kubernetes/client/rest.py:274: in POST
    return self.request("POST", url,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <kubernetes.client.rest.RESTClientObject object at 0x7f38262f8910>
method = 'POST'
url = 'https://api.jenkins-ci.perf-testing.devcluster.openshift.com:6443/apis/ripsaw.cloudbulldozer.io/v1alpha1/namespaces/my-ripsaw/benchmarks'
query_params = []
headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/12.0.0/python'}
body = {'apiVersion': 'ripsaw.cloudbulldozer.io/v1alpha1', 'kind': 'Benchmark', 'metadata': {'name': 'uperf-resources', 'name...00m', 'memory': '100Mi'}}, 'hostnetwork': True, 'multus': {'enabled': False}, 'nthrs': [1, 2], ...}, 'name': 'uperf'}}}
post_params = {}, _preload_content = True, _request_timeout = None

    def request(self, method, url, query_params=None, headers=None,
                body=None, post_params=None, _preload_content=True,
                _request_timeout=None):
        """Perform requests.
    
        :param method: http request method
        :param url: http request url
        :param query_params: query parameters in the url
        :param headers: http request headers
        :param body: request json body, for `application/json`
        :param post_params: request post parameters,
                            `application/x-www-form-urlencoded`
                            and `multipart/form-data`
        :param _preload_content: if False, the urllib3.HTTPResponse object will
                                 be returned without reading/decoding response
                                 data. Default is True.
        :param _request_timeout: timeout setting for this request. If one
                                 number provided, it will be total request
                                 timeout. It can also be a pair (tuple) of
                                 (connection, read) timeouts.
        """
        method = method.upper()
        assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT',
                          'PATCH', 'OPTIONS']
    
        if post_params and body:
            raise ApiValueError(
                "body parameter cannot be used with post_params parameter."
            )
    
        post_params = post_params or {}
        headers = headers or {}
    
        timeout = None
        if _request_timeout:
            if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)):  # noqa: E501,F821
                timeout = urllib3.Timeout(total=_request_timeout)
            elif (isinstance(_request_timeout, tuple) and
                  len(_request_timeout) == 2):
                timeout = urllib3.Timeout(
                    connect=_request_timeout[0], read=_request_timeout[1])
    
        if 'Content-Type' not in headers:
            headers['Content-Type'] = 'application/json'
    
        try:
            # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE`
            if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']:
                if query_params:
                    url += '?' + urlencode(query_params)
                if re.search('json', headers['Content-Type'], re.IGNORECASE):
                    if headers['Content-Type'] == 'application/json-patch+json':
                        if not isinstance(body, list):
                            headers['Content-Type'] = \
                                'application/strategic-merge-patch+json'
                    request_body = None
                    if body is not None:
                        request_body = json.dumps(body)
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'application/x-www-form-urlencoded':  # noqa: E501
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=False,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'multipart/form-data':
                    # must del headers['Content-Type'], or the correct
                    # Content-Type which generated by urllib3 will be
                    # overwritten.
                    del headers['Content-Type']
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=True,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                # Pass a `string` parameter directly in the body to support
                # other content types than Json when `body` argument is
                # provided in serialized form
                elif isinstance(body, str) or isinstance(body, bytes):
                    request_body = body
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                else:
                    # Cannot generate the request from given parameters
                    msg = """Cannot prepare a request message for provided
                             arguments. Please check that your arguments match
                             declared content type."""
                    raise ApiException(status=0, reason=msg)
            # For `GET`, `HEAD`
            else:
                r = self.pool_manager.request(method, url,
                                              fields=query_params,
                                              preload_content=_preload_content,
                                              timeout=timeout,
                                              headers=headers)
        except urllib3.exceptions.SSLError as e:
            msg = "{0}\n{1}".format(type(e).__name__, str(e))
            raise ApiException(status=0, reason=msg)
    
        if _preload_content:
            r = RESTResponse(r)
    
            # In the python 3, the response.data is bytes.
            # we need to decode it to string.
            if six.PY3:
                r.data = r.data.decode('utf8')
    
            # log response body
            logger.debug("response body: %s", r.data)
    
        if not 200 <= r.status <= 299:
>           raise ApiException(http_resp=r)
E           kubernetes.client.exceptions.ApiException: (403)
E           Reason: Forbidden
E           HTTP response headers: HTTPHeaderDict({'Audit-Id': 'abb38690-fa15-4a27-a4f7-1b757de49bbc', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '682d9336-3769-4f6e-8f0f-2f36581dea79', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'fb2241dd-f676-4aab-969d-77ab112dfaf0', 'Date': 'Thu, 19 Nov 2020 20:57:19 GMT', 'Content-Length': '484'})
E           HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"benchmarks.ripsaw.cloudbulldozer.io \"uperf-resources\" is forbidden: unable to create new content in namespace my-ripsaw because it is being terminated","reason":"Forbidden","details":{"name":"uperf-resources","group":"ripsaw.cloudbulldozer.io","kind":"benchmarks","causes":[{"reason":"NamespaceTerminating","message":"namespace my-ripsaw is being terminated","field":"metadata.namespace"}]},"code":403}

venv/lib/python3.8/site-packages/kubernetes/client/rest.py:233: ApiException

e2e/tests/test_ycsb.py

TestYcsb.test_ycsb[base] 0.00s ⏱

self = <test_ycsb.TestYcsb object at 0x7f3825e2f760>
run = <models.benchmark_run.BenchmarkRun object at 0x7f382623e5e0>

    def test_ycsb(self, run):
>       self.run_and_check_benchmark(run, desired_running_state="Running Load", desired_complete_state="Workload Complete")

e2e/tests/test_ycsb.py:12: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
e2e/models/test_base.py:92: in run_and_check_benchmark
    run.start(desired_state=desired_running_state, default_timeout=default_timeout)
e2e/models/benchmark_run.py:22: in start
    self.metadata = self.cluster.create_benchmark(self.yaml, desired_state, default_timeout=default_timeout)
e2e/util/k8s.py:117: in create_benchmark
    self.crd_client.create_namespaced_custom_object(
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:225: in create_namespaced_custom_object
    return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs)  # noqa: E501
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:344: in create_namespaced_custom_object_with_http_info
    return self.api_client.call_api(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:348: in call_api
    return self.__call_api(resource_path, method,
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:180: in __call_api
    response_data = self.request(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:391: in request
    return self.rest_client.POST(url,
venv/lib/python3.8/site-packages/kubernetes/client/rest.py:274: in POST
    return self.request("POST", url,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <kubernetes.client.rest.RESTClientObject object at 0x7f382623e280>
method = 'POST'
url = 'https://api.jenkins-ci.perf-testing.devcluster.openshift.com:6443/apis/ripsaw.cloudbulldozer.io/v1alpha1/namespaces/my-ripsaw/benchmarks'
query_params = []
headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/12.0.0/python'}
body = {'apiVersion': 'ripsaw.cloudbulldozer.io/v1alpha1', 'kind': 'Benchmark', 'metadata': {'name': 'ycsb-base', 'namespace'...godb', 'operationcount': 100, 'options_load': '-p mongodb.url="mongodb://mongo-0.mongo/ycsb?"', ...}, 'name': 'ycsb'}}}
post_params = {}, _preload_content = True, _request_timeout = None

    def request(self, method, url, query_params=None, headers=None,
                body=None, post_params=None, _preload_content=True,
                _request_timeout=None):
        """Perform requests.
    
        :param method: http request method
        :param url: http request url
        :param query_params: query parameters in the url
        :param headers: http request headers
        :param body: request json body, for `application/json`
        :param post_params: request post parameters,
                            `application/x-www-form-urlencoded`
                            and `multipart/form-data`
        :param _preload_content: if False, the urllib3.HTTPResponse object will
                                 be returned without reading/decoding response
                                 data. Default is True.
        :param _request_timeout: timeout setting for this request. If one
                                 number provided, it will be total request
                                 timeout. It can also be a pair (tuple) of
                                 (connection, read) timeouts.
        """
        method = method.upper()
        assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT',
                          'PATCH', 'OPTIONS']
    
        if post_params and body:
            raise ApiValueError(
                "body parameter cannot be used with post_params parameter."
            )
    
        post_params = post_params or {}
        headers = headers or {}
    
        timeout = None
        if _request_timeout:
            if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)):  # noqa: E501,F821
                timeout = urllib3.Timeout(total=_request_timeout)
            elif (isinstance(_request_timeout, tuple) and
                  len(_request_timeout) == 2):
                timeout = urllib3.Timeout(
                    connect=_request_timeout[0], read=_request_timeout[1])
    
        if 'Content-Type' not in headers:
            headers['Content-Type'] = 'application/json'
    
        try:
            # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE`
            if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']:
                if query_params:
                    url += '?' + urlencode(query_params)
                if re.search('json', headers['Content-Type'], re.IGNORECASE):
                    if headers['Content-Type'] == 'application/json-patch+json':
                        if not isinstance(body, list):
                            headers['Content-Type'] = \
                                'application/strategic-merge-patch+json'
                    request_body = None
                    if body is not None:
                        request_body = json.dumps(body)
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'application/x-www-form-urlencoded':  # noqa: E501
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=False,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'multipart/form-data':
                    # must del headers['Content-Type'], or the correct
                    # Content-Type which generated by urllib3 will be
                    # overwritten.
                    del headers['Content-Type']
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=True,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                # Pass a `string` parameter directly in the body to support
                # other content types than Json when `body` argument is
                # provided in serialized form
                elif isinstance(body, str) or isinstance(body, bytes):
                    request_body = body
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                else:
                    # Cannot generate the request from given parameters
                    msg = """Cannot prepare a request message for provided
                             arguments. Please check that your arguments match
                             declared content type."""
                    raise ApiException(status=0, reason=msg)
            # For `GET`, `HEAD`
            else:
                r = self.pool_manager.request(method, url,
                                              fields=query_params,
                                              preload_content=_preload_content,
                                              timeout=timeout,
                                              headers=headers)
        except urllib3.exceptions.SSLError as e:
            msg = "{0}\n{1}".format(type(e).__name__, str(e))
            raise ApiException(status=0, reason=msg)
    
        if _preload_content:
            r = RESTResponse(r)
    
            # In the python 3, the response.data is bytes.
            # we need to decode it to string.
            if six.PY3:
                r.data = r.data.decode('utf8')
    
            # log response body
            logger.debug("response body: %s", r.data)
    
        if not 200 <= r.status <= 299:
>           raise ApiException(http_resp=r)
E           kubernetes.client.exceptions.ApiException: (403)
E           Reason: Forbidden
E           HTTP response headers: HTTPHeaderDict({'Audit-Id': '709e268c-46d6-452a-8384-f7b9de14b269', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '682d9336-3769-4f6e-8f0f-2f36581dea79', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'fb2241dd-f676-4aab-969d-77ab112dfaf0', 'Date': 'Thu, 19 Nov 2020 20:57:22 GMT', 'Content-Length': '472'})
E           HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"benchmarks.ripsaw.cloudbulldozer.io \"ycsb-base\" is forbidden: unable to create new content in namespace my-ripsaw because it is being terminated","reason":"Forbidden","details":{"name":"ycsb-base","group":"ripsaw.cloudbulldozer.io","kind":"benchmarks","causes":[{"reason":"NamespaceTerminating","message":"namespace my-ripsaw is being terminated","field":"metadata.namespace"}]},"code":403}

venv/lib/python3.8/site-packages/kubernetes/client/rest.py:233: ApiException

e2e/tests/test_scale_openshift.py

TestScale.test_scale[up] 0.00s ⏱

self = <test_scale_openshift.TestScale object at 0x7f3825e1e940>
run = <models.benchmark_run.BenchmarkRun object at 0x7f3826234160>

    def test_scale(self, run):
>       self.run_and_check_benchmark(run, desired_running_state="Scaling Cluster")

e2e/tests/test_scale_openshift.py:12: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
e2e/models/test_base.py:92: in run_and_check_benchmark
    run.start(desired_state=desired_running_state, default_timeout=default_timeout)
e2e/models/benchmark_run.py:22: in start
    self.metadata = self.cluster.create_benchmark(self.yaml, desired_state, default_timeout=default_timeout)
e2e/util/k8s.py:117: in create_benchmark
    self.crd_client.create_namespaced_custom_object(
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:225: in create_namespaced_custom_object
    return self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, **kwargs)  # noqa: E501
venv/lib/python3.8/site-packages/kubernetes/client/api/custom_objects_api.py:344: in create_namespaced_custom_object_with_http_info
    return self.api_client.call_api(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:348: in call_api
    return self.__call_api(resource_path, method,
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:180: in __call_api
    response_data = self.request(
venv/lib/python3.8/site-packages/kubernetes/client/api_client.py:391: in request
    return self.rest_client.POST(url,
venv/lib/python3.8/site-packages/kubernetes/client/rest.py:274: in POST
    return self.request("POST", url,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <kubernetes.client.rest.RESTClientObject object at 0x7f3826325be0>
method = 'POST'
url = 'https://api.jenkins-ci.perf-testing.devcluster.openshift.com:6443/apis/ripsaw.cloudbulldozer.io/v1alpha1/namespaces/my-ripsaw/benchmarks'
query_params = []
headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'User-Agent': 'OpenAPI-Generator/12.0.0/python'}
body = {'apiVersion': 'ripsaw.cloudbulldozer.io/v1alpha1', 'kind': 'Benchmark', 'metadata': {'name': 'scale-up', 'namespace':... True}, 'workload': {'args': {'poll_interval': 2, 'scale': 1, 'serviceaccount': 'scaler'}, 'name': 'scale_openshift'}}}
post_params = {}, _preload_content = True, _request_timeout = None

    def request(self, method, url, query_params=None, headers=None,
                body=None, post_params=None, _preload_content=True,
                _request_timeout=None):
        """Perform requests.
    
        :param method: http request method
        :param url: http request url
        :param query_params: query parameters in the url
        :param headers: http request headers
        :param body: request json body, for `application/json`
        :param post_params: request post parameters,
                            `application/x-www-form-urlencoded`
                            and `multipart/form-data`
        :param _preload_content: if False, the urllib3.HTTPResponse object will
                                 be returned without reading/decoding response
                                 data. Default is True.
        :param _request_timeout: timeout setting for this request. If one
                                 number provided, it will be total request
                                 timeout. It can also be a pair (tuple) of
                                 (connection, read) timeouts.
        """
        method = method.upper()
        assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT',
                          'PATCH', 'OPTIONS']
    
        if post_params and body:
            raise ApiValueError(
                "body parameter cannot be used with post_params parameter."
            )
    
        post_params = post_params or {}
        headers = headers or {}
    
        timeout = None
        if _request_timeout:
            if isinstance(_request_timeout, (int, ) if six.PY3 else (int, long)):  # noqa: E501,F821
                timeout = urllib3.Timeout(total=_request_timeout)
            elif (isinstance(_request_timeout, tuple) and
                  len(_request_timeout) == 2):
                timeout = urllib3.Timeout(
                    connect=_request_timeout[0], read=_request_timeout[1])
    
        if 'Content-Type' not in headers:
            headers['Content-Type'] = 'application/json'
    
        try:
            # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE`
            if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']:
                if query_params:
                    url += '?' + urlencode(query_params)
                if re.search('json', headers['Content-Type'], re.IGNORECASE):
                    if headers['Content-Type'] == 'application/json-patch+json':
                        if not isinstance(body, list):
                            headers['Content-Type'] = \
                                'application/strategic-merge-patch+json'
                    request_body = None
                    if body is not None:
                        request_body = json.dumps(body)
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'application/x-www-form-urlencoded':  # noqa: E501
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=False,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                elif headers['Content-Type'] == 'multipart/form-data':
                    # must del headers['Content-Type'], or the correct
                    # Content-Type which generated by urllib3 will be
                    # overwritten.
                    del headers['Content-Type']
                    r = self.pool_manager.request(
                        method, url,
                        fields=post_params,
                        encode_multipart=True,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                # Pass a `string` parameter directly in the body to support
                # other content types than Json when `body` argument is
                # provided in serialized form
                elif isinstance(body, str) or isinstance(body, bytes):
                    request_body = body
                    r = self.pool_manager.request(
                        method, url,
                        body=request_body,
                        preload_content=_preload_content,
                        timeout=timeout,
                        headers=headers)
                else:
                    # Cannot generate the request from given parameters
                    msg = """Cannot prepare a request message for provided
                             arguments. Please check that your arguments match
                             declared content type."""
                    raise ApiException(status=0, reason=msg)
            # For `GET`, `HEAD`
            else:
                r = self.pool_manager.request(method, url,
                                              fields=query_params,
                                              preload_content=_preload_content,
                                              timeout=timeout,
                                              headers=headers)
        except urllib3.exceptions.SSLError as e:
            msg = "{0}\n{1}".format(type(e).__name__, str(e))
            raise ApiException(status=0, reason=msg)
    
        if _preload_content:
            r = RESTResponse(r)
    
            # In the python 3, the response.data is bytes.
            # we need to decode it to string.
            if six.PY3:
                r.data = r.data.decode('utf8')
    
            # log response body
            logger.debug("response body: %s", r.data)
    
        if not 200 <= r.status <= 299:
>           raise ApiException(http_resp=r)
E           kubernetes.client.exceptions.ApiException: (403)
E           Reason: Forbidden
E           HTTP response headers: HTTPHeaderDict({'Audit-Id': '161d7ba5-0e0c-4749-b12a-4cefb65151fb', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '682d9336-3769-4f6e-8f0f-2f36581dea79', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'fb2241dd-f676-4aab-969d-77ab112dfaf0', 'Date': 'Thu, 19 Nov 2020 20:57:25 GMT', 'Content-Length': '470'})
E           HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"benchmarks.ripsaw.cloudbulldozer.io \"scale-up\" is forbidden: unable to create new content in namespace my-ripsaw because it is being terminated","reason":"Forbidden","details":{"name":"scale-up","group":"ripsaw.cloudbulldozer.io","kind":"benchmarks","causes":[{"reason":"NamespaceTerminating","message":"namespace my-ripsaw is being terminated","field":"metadata.namespace"}]},"code":403}

venv/lib/python3.8/site-packages/kubernetes/client/rest.py:233: ApiException

2 passed 😃

e2e/tests/test_backpack.py

Testbackpack.test_backpack[init] 81.41s ⏱

e2e/tests/test_fiod.py

TestFiod.test_fiod[bsrange] 300.88s ⏱

3 skipped 🙄

e2e/tests/test_service_mesh.py

TestServiceMesh.test_servicemesh[servicemesh] 0.00s ⏱

e2e/tests/test_stressng.py

TestStressng.test_stressng[base] 0.00s ⏱

e2e/tests/test_scale_openshift.py

TestScale.test_scale[down] 0.00s ⏱

whitleykeith · 2020-11-24T14:48:14Z

/rerun all

comet-perf-ci · 2020-11-24T16:14:33Z

Test Report

Report generated on 24-Nov-2020 at 16:13:44 by pytest-md 📝

Summary

29 tests ran in 2620.13 seconds ⏱

27 passed 😃
2 skipped 🙄

27 passed 😃

e2e/tests/test_byowl.py

TestByowl.test_byowl[base] 57.16s ⏱

e2e/tests/test_backpack_ds.py

TestBackpackDaemonSet.test_backpack_daemonset[daemonset] 87.51s ⏱

e2e/tests/test_backpack.py

Testbackpack.test_backpack[init] 75.34s ⏱

e2e/tests/test_fs_drift.py

TestFSDrift.test_fs_drift[hostpath] 117.36s ⏱

TestFSDrift.test_fs_drift[base] 120.40s ⏱

e2e/tests/test_hammerdb.py

TestHammerDB.test_hammerdb[base] 243.90s ⏱

e2e/tests/test_fiod.py

TestFiod.test_fiod[bsrange] 267.85s ⏱

TestFiod.test_fiod[hostpath] 616.93s ⏱

TestFiod.test_fiod[base] 282.88s ⏱

e2e/tests/test_kube_burner.py

TestKubeBurner.test_kube_burner[kubelet-density-metics] 120.41s ⏱

TestKubeBurner.test_kube_burner[kubelet-density-heavy-metrics] 186.63s ⏱

TestKubeBurner.test_kube_burner[cluster-density-metrics-aggregated] 268.00s ⏱

e2e/tests/test_iperf3.py

TestIPerf3.test_iperf3[hostnetwork] 177.52s ⏱

TestIPerf3.test_iperf3[base] 234.66s ⏱

e2e/tests/test_pgbench.py

TestPgBench.test_pgbench[base] 267.81s ⏱

e2e/tests/test_smallfile.py

TestSmallFile.test_smallfile[hostpath] 159.49s ⏱

TestSmallFile.test_smallfile[base] 126.41s ⏱

e2e/tests/test_sysbench.py

TestSysbench.test_sysbench[sysbench] 96.29s ⏱

e2e/tests/test_vegeta.py

TestVegeta.test_vegeta[hostnetwork] 102.29s ⏱

TestVegeta.test_vegeta[base] 96.28s ⏱

e2e/tests/test_ycsb.py

TestYcsb.test_ycsb[base] 189.56s ⏱

e2e/tests/test_uperf.py

TestUperf.test_uperf[uperf_serviceip] 391.06s ⏱

TestUperf.test_uperf[base] 394.09s ⏱

TestUperf.test_uperf[uperf_networkpolicy] 369.99s ⏱

TestUperf.test_uperf[uperf_resources] 270.75s ⏱

e2e/tests/test_scale_openshift.py

TestScale.test_scale[up] 373.06s ⏱

TestScale.test_scale[down] 123.44s ⏱

2 skipped 🙄

e2e/tests/test_stressng.py

TestStressng.test_stressng[base] 0.00s ⏱

e2e/tests/test_service_mesh.py

TestServiceMesh.test_servicemesh[servicemesh] 0.00s ⏱

comet-perf-ci · 2020-12-04T15:31:13Z

Test Report

Report generated on 04-Dec-2020 at 15:30:25 by pytest-md 📝

Summary

29 tests ran in 2621.57 seconds ⏱

1 failed 😰
26 passed 😃
2 skipped 🙄

1 failed 😰

e2e/tests/test_kube_burner.py

TestKubeBurner.test_kube_burner[cluster-density-metrics-aggregated] 144.38s ⏱

[gw1] linux -- Python 3.8.5 /var/lib/jenkins/workspace/ec2_jjb/venv/bin/python3

self = <test_kube_burner.TestKubeBurner object at 0x7f162ee4c2b0>
run = <models.benchmark_run.BenchmarkRun object at 0x7f162efff400>
desired_running_state = 'Running', desired_complete_state = 'Complete'
default_timeout = 600

    def run_and_check_benchmark(self, run, desired_running_state="Running", desired_complete_state="Complete", default_timeout=default_timeout):
        run.start(desired_state=desired_running_state, default_timeout=default_timeout)
        try:
>           results = run.wait(desired_state=desired_complete_state, default_timeout=default_timeout)

e2e/models/test_base.py:94: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <models.benchmark_run.BenchmarkRun object at 0x7f162efff400>
desired_state = 'Complete', default_timeout = 600

    def wait(self, desired_state="Complete", default_timeout=300):
        if self.metadata == {}:
            raise BenchmarkNotStartedError(self.name)
>       self.cluster.wait_for_benchmark(self.resource_name, self.resource_namespace, desired_state=desired_state, default_timeout=default_timeout)

e2e/models/benchmark_run.py:28: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <util.k8s.Cluster object at 0x7f162f025d00>
name = 'kube-burner-cluster-density-metrics-aggregated', namespace = 'my-ripsaw'
desired_state = 'Complete', default_timeout = 600

    def wait_for_benchmark(self, name, namespace, desired_state="Completed", default_timeout=300):
        waiting_for_benchmark = True
        logging.info(f"Waiting for state: {desired_state}")
        timeout_interval = 0
        while waiting_for_benchmark:
            if timeout_interval >= default_timeout:
                raise BenchmarkTimeoutError(name)
            benchmark = self.get_benchmark(name, namespace)
            bench_status = benchmark.get('status', {})
            uuid = bench_status.get('uuid', "Not Assigned Yet")
            current_state = bench_status.get('state', "")
            logging.info(
                f"{benchmark['metadata']['namespace']}\t{benchmark['metadata']['name']}\t{uuid}\t{current_state}")
            if current_state == "Failed":
>               raise BenchmarkFailedError(
                    benchmark['metadata']['name'], benchmark['status']['uuid'])
E               util.exceptions.BenchmarkFailedError: The benchmark kube-burner-cluster-density-metrics-aggregated with uuid 511a927e-ec2a-53c3-ba09-81f92252da90 failed

e2e/util/k8s.py:105: BenchmarkFailedError

During handling of the above exception, another exception occurred:

self = <test_kube_burner.TestKubeBurner object at 0x7f162ee4c2b0>
run = <models.benchmark_run.BenchmarkRun object at 0x7f162efff400>

    def test_kube_burner(self, run):
>       self.run_and_check_benchmark(run)

e2e/tests/test_kube_burner.py:27: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_kube_burner.TestKubeBurner object at 0x7f162ee4c2b0>
run = <models.benchmark_run.BenchmarkRun object at 0x7f162efff400>
desired_running_state = 'Running', desired_complete_state = 'Complete'
default_timeout = 600

    def run_and_check_benchmark(self, run, desired_running_state="Running", desired_complete_state="Complete", default_timeout=default_timeout):
        run.start(desired_state=desired_running_state, default_timeout=default_timeout)
        try:
            results = run.wait(desired_state=desired_complete_state, default_timeout=default_timeout)
            assert results.get('status', "") == desired_complete_state
            self.check_metadata_collection(results['uuid'])
        except (BenchmarkFailedError, BenchmarkTimeoutError) as err:
>           assert False
E           assert False

e2e/models/test_base.py:98: AssertionError

26 passed 😃

e2e/tests/test_byowl.py

TestByowl.test_byowl[base] 54.17s ⏱

e2e/tests/test_backpack_ds.py

TestBackpackDaemonSet.test_backpack_daemonset[daemonset] 75.30s ⏱

e2e/tests/test_backpack.py

Testbackpack.test_backpack[init] 75.47s ⏱

e2e/tests/test_fs_drift.py

TestFSDrift.test_fs_drift[hostpath] 114.38s ⏱

TestFSDrift.test_fs_drift[base] 123.41s ⏱

e2e/tests/test_hammerdb.py

TestHammerDB.test_hammerdb[base] 262.21s ⏱

e2e/tests/test_kube_burner.py

TestKubeBurner.test_kube_burner[kubelet-density-metics] 144.46s ⏱

TestKubeBurner.test_kube_burner[kubelet-density-heavy-metrics] 195.54s ⏱

e2e/tests/test_fiod.py

TestFiod.test_fiod[bsrange] 327.89s ⏱

TestFiod.test_fiod[hostpath] 583.60s ⏱

TestFiod.test_fiod[base] 291.84s ⏱

e2e/tests/test_iperf3.py

TestIPerf3.test_iperf3[hostnetwork] 228.72s ⏱

TestIPerf3.test_iperf3[base] 228.71s ⏱

e2e/tests/test_pgbench.py

TestPgBench.test_pgbench[base] 262.13s ⏱

e2e/tests/test_sysbench.py

TestSysbench.test_sysbench[sysbench] 102.31s ⏱

e2e/tests/test_vegeta.py

TestVegeta.test_vegeta[hostnetwork] 132.38s ⏱

TestVegeta.test_vegeta[base] 102.28s ⏱

e2e/tests/test_uperf.py

TestUperf.test_uperf[uperf_serviceip] 430.28s ⏱

TestUperf.test_uperf[base] 469.40s ⏱

TestUperf.test_uperf[uperf_networkpolicy] 394.18s ⏱

TestUperf.test_uperf[uperf_resources] 270.84s ⏱

e2e/tests/test_smallfile.py

TestSmallFile.test_smallfile[hostpath] 129.37s ⏱

TestSmallFile.test_smallfile[base] 153.47s ⏱

e2e/tests/test_ycsb.py

TestYcsb.test_ycsb[base] 189.56s ⏱

e2e/tests/test_scale_openshift.py

TestScale.test_scale[up] 397.05s ⏱

TestScale.test_scale[down] 123.36s ⏱

2 skipped 🙄

e2e/tests/test_service_mesh.py

TestServiceMesh.test_servicemesh[servicemesh] 0.00s ⏱

e2e/tests/test_stressng.py

TestStressng.test_stressng[base] 0.00s ⏱

…ppens after scale_down

…dont support metadata collection

comet-perf-ci · 2021-01-14T01:16:25Z

!!! Couldn't read commit file !!!

stale · 2021-01-30T02:58:32Z

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

whitleykeith · 2021-02-01T15:43:08Z

should I close this PR and reopen it to make it smaller? @jtaleric @dry923 @rsevilla87

stale · 2021-02-16T15:52:31Z

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

whitleykeith force-pushed the ci-refactor-v2 branch 3 times, most recently from 26f6cb0 to 5af4a70 Compare October 22, 2020 14:49

whitleykeith force-pushed the ci-refactor-v2 branch 2 times, most recently from 45af110 to b0e78c0 Compare October 30, 2020 19:22

whitleykeith marked this pull request as ready for review October 30, 2020 19:22

dry923 added the ok to test Kick off our CI framework label Oct 30, 2020

jtaleric reviewed Nov 3, 2020

View reviewed changes

.gitignore Show resolved Hide resolved

jtaleric reviewed Nov 3, 2020

View reviewed changes

whitleykeith force-pushed the ci-refactor-v2 branch from 47ad40e to 72b2a68 Compare November 3, 2020 19:01

whitleykeith force-pushed the ci-refactor-v2 branch from 36ac084 to 60e4719 Compare November 19, 2020 19:56

whitleykeith force-pushed the ci-refactor-v2 branch from 60e4719 to b267548 Compare December 4, 2020 14:44

rsevilla87 requested review from dry923 and rsevilla87 December 4, 2020 15:32

Keith Whitley added 20 commits January 11, 2021 10:27

changing backpack ds desired state

ca679b5

removing sleep

b0a31a9

using right client in k8s.py

9ff1e0e

using right list comprehension syntax

bdf599d

ensuring scale tests run before all tests and that scale_up always ha…

916ffff

…ppens after scale_down

keeping this branch up with master

1d1153e

cleanup pt2

98ef6b3

skipping flaky

c5c5a5e

changing how scale tests are ordered

0e222ef

ensuring scale_down always runs after scale_up

ae4b8fb

fixing typo

21657ac

removing unecessary values and default metadata.collection to true

3e27be1

changing default metadata.collection to false on specific tests that …

549c9fd

…dont support metadata collection

fixing keypath seperator since some keys use periods

8e91408

actually fixing keypath issue

1b10b18

increasing timeout for kubeburner tests

7470857

increasing timeout for kubeburner tests

664475d

removing duplicate task in kube-burner role

4997b32

forcing rebuild

71aab7d

changing things from reviews

2967370

whitleykeith force-pushed the ci-refactor-v2 branch from 85d7bac to 2967370 Compare January 11, 2021 15:32

stale bot added the not_ready label Jan 30, 2021

stale bot removed the not_ready label Feb 1, 2021

stale bot added the not_ready label Feb 16, 2021

stale bot closed this Feb 27, 2021

whitleykeith mentioned this pull request Mar 17, 2021

Refactor benchmarking scripts into a versioned CLI Tool cloud-bulldozer/e2e-benchmarking#143

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

E2E Test Refactoring #437

E2E Test Refactoring #437

whitleykeith commented Oct 14, 2020 •

edited

Loading

dry923 commented Oct 30, 2020

comet-perf-ci commented Oct 30, 2020

whitleykeith commented Oct 30, 2020

comet-perf-ci commented Oct 30, 2020

comet-perf-ci commented Nov 2, 2020

comet-perf-ci commented Nov 2, 2020

jtaleric Nov 3, 2020

whitleykeith Dec 21, 2020

jtaleric Nov 3, 2020

comet-perf-ci commented Nov 3, 2020

comet-perf-ci commented Nov 3, 2020

whitleykeith commented Nov 12, 2020

comet-perf-ci commented Nov 19, 2020

whitleykeith commented Nov 19, 2020

comet-perf-ci commented Nov 23, 2020

whitleykeith commented Nov 24, 2020

comet-perf-ci commented Nov 24, 2020

comet-perf-ci commented Dec 4, 2020

comet-perf-ci commented Jan 14, 2021

stale bot commented Jan 30, 2021

whitleykeith commented Feb 1, 2021

stale bot commented Feb 16, 2021

E2E Test Refactoring #437

E2E Test Refactoring #437

Conversation

whitleykeith commented Oct 14, 2020 • edited Loading

dry923 commented Oct 30, 2020

comet-perf-ci commented Oct 30, 2020

Test Report

Summary

26 passed 😃

e2e/tests/test_byowl.py

e2e/tests/test_backpack.py

e2e/tests/test_backpack_ds.py

e2e/tests/test_hammerdb.py

e2e/tests/test_fs_drift.py

e2e/tests/test_fiod.py

e2e/tests/test_iperf3.py

e2e/tests/test_kube_burner.py

e2e/tests/test_pgbench.py

e2e/tests/test_sysbench.py

e2e/tests/test_vegeta.py

e2e/tests/test_smallfile.py

e2e/tests/test_ycsb.py

e2e/tests/test_scale_openshift.py

e2e/tests/test_uperf.py

whitleykeith commented Oct 30, 2020

Test Report

Summary

1 failed 😰

e2e/tests/test_fiod.py

comet-perf-ci commented Oct 30, 2020

Test Report

Summary

2 failed 😰

e2e/tests/test_service_mesh.py

e2e/tests/test_stressng.py

25 passed 😃

e2e/tests/test_backpack_ds.py

e2e/tests/test_fs_drift.py

e2e/tests/test_backpack.py

e2e/tests/test_byowl.py

e2e/tests/test_fiod.py

e2e/tests/test_iperf3.py

e2e/tests/test_hammerdb.py

e2e/tests/test_kube_burner.py

e2e/tests/test_pgbench.py

e2e/tests/test_smallfile.py

e2e/tests/test_sysbench.py

e2e/tests/test_vegeta.py

e2e/tests/test_scale_openshift.py

e2e/tests/test_uperf.py

e2e/tests/test_ycsb.py

1 skipped 🙄

e2e/tests/test_scale_openshift.py

comet-perf-ci commented Nov 2, 2020

comet-perf-ci commented Nov 2, 2020

Test Report

Summary

26 passed 😃

e2e/tests/test_byowl.py

e2e/tests/test_backpack.py

e2e/tests/test_backpack_ds.py

e2e/tests/test_hammerdb.py

e2e/tests/test_fs_drift.py

e2e/tests/test_fiod.py

e2e/tests/test_iperf3.py

e2e/tests/test_kube_burner.py

e2e/tests/test_pgbench.py

e2e/tests/test_smallfile.py

e2e/tests/test_sysbench.py

e2e/tests/test_vegeta.py

e2e/tests/test_ycsb.py

e2e/tests/test_scale_openshift.py

e2e/tests/test_uperf.py

2 skipped 🙄

e2e/tests/test_service_mesh.py

e2e/tests/test_stressng.py

jtaleric Nov 3, 2020

Choose a reason for hiding this comment

whitleykeith Dec 21, 2020

Choose a reason for hiding this comment

whitleykeith commented Oct 14, 2020 •

edited

Loading