Automates BZ OCS podDisruptionBudget prevents successful OCP upgrades #3888

Shrivaibavi · 2021-02-22T17:58:25Z

Automates BZ https://bugzilla.redhat.com/show_bug.cgi?id=1861104

Signed-off-by: Shrivaibavi Raghaventhiran sraghave@redhat.com

ocs_ci/ocs/cluster.py

tests/ecosystem/upgrade/test_upgrade.py

tests/manage/z_cluster/nodes/test_simultaneous_ocs_node_drains.py

ocs_ci/ocs/cluster.py

tests/ecosystem/upgrade/test_upgrade.py

tests/manage/z_cluster/nodes/test_simultaneous_ocs_node_drains.py

ocs_ci/ocs/cluster.py

ocs_ci/ocs/constants.py

tests/ecosystem/upgrade/test_upgrade.py

ocs_ci/ocs/cluster.py

tests/manage/z_cluster/nodes/test_simultaneous_ocs_node_drains.py

ocs_ci/ocs/cluster.py

tests/manage/z_cluster/nodes/test_simultaneous_ocs_node_drains.py

tests/manage/z_cluster/upgrade/test_check_osd_mon_post_ocs_upgrade.py

ocs_ci/ocs/cluster.py

tests/manage/z_cluster/nodes/test_simultaneous_ocs_node_drains.py

tests/manage/z_cluster/upgrade/test_check_osd_mon_post_ocs_upgrade.py

ocs_ci/ocs/cluster.py

tests/manage/z_cluster/nodes/test_simultaneous_ocs_node_drains.py

tests/manage/z_cluster/upgrade/test_check_osd_post_ocs_upgrade.py

ocs_ci/ocs/cluster.py

PrasadDesala · 2021-08-16T06:37:51Z

tests/manage/z_cluster/upgrade/test_check_mon_pdb_post_upgrade.py


 log = logging.getLogger(__name__)


-@post_ocs_upgrade
-@skipif_external_mode
-@pytest.mark.polarion_id("OCS-2449")
 class TestToCheckMonPDBPostUpgrade(ManageTest):


Now the test file "test_check_mon_pdb_post_upgrade.py" consists of both mon and osd pdb test functions. Please also change the test file name accordingly or make it generic something like test_pdb_check_post_upgrade.py
Same for the class function also.

PrasadDesala · 2021-08-16T06:39:16Z

tests/manage/z_cluster/upgrade/test_check_mon_pdb_post_upgrade.py

+        2. Rook-ceph-drain-canary pods disappeared after upgrade of OCS version to ocs-operator.v4.6.2-233.ci
+
+        """
+        assert (


Should we also consider checking values for disruptions_allowed, min_available_osd, max_unavailable_osd, the way how we are checking for test_check_mon_pdb_post_upgrade?

Added validations in validate_existence_of_blocking_pdb(), Please check

ocs-ci

PR validation on existing cluster

Cluster Name: tdesala-a11
Cluster Configuration:
PR Test Suite: tier4b
PR Test Path: tests/manage/z_cluster/nodes/test_nodes_maintenance.py::TestNodesMaintenance::test_simultaneous_node_drains
Additional Test Params:
OCP VERSION: 4.8
OCS VERSION: 4.8

Job FAILED (installation failed, tests not executed).

ocs-ci

PR validation on existing cluster

Cluster Name: tdesala-a11
Cluster Configuration:
PR Test Suite: tier4b
PR Test Path: tests/manage/z_cluster/nodes/test_nodes_maintenance.py::TestNodesMaintenance::test_simultaneous_node_drains
Additional Test Params:
OCP VERSION: 4.8
OCS VERSION: 4.8

Job FAILED (installation failed, tests not executed).

OdedViner

LGTM

ocs-ci

PR validation on existing cluster

Cluster Name: tdesala-a11
Cluster Configuration:
PR Test Suite: tier4b
PR Test Path: tests/manage/z_cluster/nodes/test_nodes_maintenance.py::TestNodesMaintenance::test_pdb_check_simultaneous_node_drains
Additional Test Params:
OCP VERSION: 4.8
OCS VERSION: 4.8

Job FAILED (installation failed, tests not executed).

ocs-ci

PR validation on existing cluster

Cluster Name: tdesala-a11
Cluster Configuration:
PR Test Suite:
PR Test Path: tests/manage/z_cluster/nodes/test_nodes_maintenance.py::TestNodesMaintenance::test_pdb_check_simultaneous_node_drains
Additional Test Params:
OCP VERSION: 4.8
OCS VERSION: 4.8

Job UNSTABLE (some or all tests failed).

ocs-ci

PR validation on existing cluster

Cluster Name: tdesala-a11
Cluster Configuration:
PR Test Suite:
PR Test Path: tests/manage/z_cluster/nodes/test_nodes_maintenance.py::TestNodesMaintenance::test_pdb_check_simultaneous_node_drains
Additional Test Params:
OCP VERSION: 4.8
OCS VERSION: 4.8

Job UNSTABLE (some or all tests failed).

ocs-ci

PR validation on existing cluster

Cluster Name: tdesala-a11
Cluster Configuration:
PR Test Suite:
PR Test Path: tests/manage/z_cluster/nodes/test_nodes_maintenance.py::TestNodesMaintenance::test_pdb_check_simultaneous_node_drains
Additional Test Params:
OCP VERSION: 4.8
OCS VERSION: 4.8

Job UNSTABLE (some or all tests failed).

tests/manage/z_cluster/upgrade/test_check_pdb_post_upgrade.py

tests/manage/z_cluster/nodes/test_nodes_maintenance.py

Signed-off-by: Shrivaibavi Raghaventhiran <sraghave@redhat.com>

ocs-ci

PR validation on existing cluster

Cluster Name: tdesala-a11
Cluster Configuration:
PR Test Suite:
PR Test Path: tests/manage/z_cluster/nodes/test_nodes_maintenance.py::TestNodesMaintenance::test_pdb_check_simultaneous_node_drains
Additional Test Params:
OCP VERSION: 4.8
OCS VERSION: 4.8

Job PASSED.

Signed-off-by: Shrivaibavi Raghaventhiran <sraghave@redhat.com>

OdedViner

LGTM

dahorak

Just one note, but it is not a blocker for this PR.

dahorak · 2021-09-01T16:59:56Z

ocs_ci/ocs/cluster.py

+        allowed_disruptions = osd_pdb[osd].get("status").get("disruptionsAllowed")
+        maximum_unavailable = osd_pdb[osd].get("spec").get("maxUnavailable")


Not sure if it might happen, but if status or spec key will not be available, it will raise exception AttributeError: 'NoneType' object has no attribute 'get'.
If it is a valid scenario (that those keys might not be present) and we want to get some default value (None in this case), it is better to provide empty dict ({}) as default value for the get(...) method:

Suggested change

allowed_disruptions = osd_pdb[osd].get("status").get("disruptionsAllowed")

maximum_unavailable = osd_pdb[osd].get("spec").get("maxUnavailable")

allowed_disruptions = osd_pdb[osd].get("status", {}).get("disruptionsAllowed")

maximum_unavailable = osd_pdb[osd].get("spec", {}).get("maxUnavailable")

Or if this keys status and spec should be always present, you might use this way:

Suggested change

allowed_disruptions = osd_pdb[osd].get("status").get("disruptionsAllowed")

maximum_unavailable = osd_pdb[osd].get("spec").get("maxUnavailable")

allowed_disruptions = osd_pdb[osd]["status"].get("disruptionsAllowed")

maximum_unavailable = osd_pdb[osd]["spec"].get("maxUnavailable")

And if it will for some reason happen, that status or spec key are not present, it will raise KeyError: 'status' exception, which might be slightly more clear, than the AttributeError: 'NoneType' object has no attribute 'get' exception mentioned above.

prsurve · 2021-09-02T05:16:14Z

ocs_ci/ocs/cluster.py

+    for pdb in pdb_obj_get.get("items"):
+        if not any(
+            osd in pdb["metadata"]["name"]
+            for osd in [constants.MDS_PDB, constants.MON_PDB]


should we have osd pdb as well

We get only OSD PDBs from the list of PDBs so we exclude mons and mds PDBs

Shrivaibavi requested a review from a team as a code owner February 22, 2021 17:58

pull-request-size bot added the size/M PR that changes 30-99 lines label Feb 22, 2021

Shrivaibavi added Needs Testing Run tests and provide logs link Customer defects Defects automated aspart of GSS closed loop team/e2e E2E team related issues/PRs upgrade labels Feb 22, 2021

Shrivaibavi force-pushed the new_pdb_design branch from 6b0c0ba to 5796080 Compare May 5, 2021 10:32

pull-request-size bot added size/L PR that changes 100-499 lines and removed size/M PR that changes 30-99 lines labels May 5, 2021

OdedViner reviewed May 5, 2021

View reviewed changes

ocs_ci/ocs/cluster.py Show resolved Hide resolved

OdedViner reviewed May 5, 2021

View reviewed changes

tests/ecosystem/upgrade/test_upgrade.py Outdated Show resolved Hide resolved

OdedViner reviewed May 5, 2021

View reviewed changes

tests/manage/z_cluster/nodes/test_simultaneous_ocs_node_drains.py Outdated Show resolved Hide resolved

OdedViner reviewed May 5, 2021

View reviewed changes

tests/manage/z_cluster/nodes/test_simultaneous_ocs_node_drains.py Outdated Show resolved Hide resolved

yosibsh reviewed May 5, 2021

View reviewed changes

Shrivaibavi changed the title ~~[WIP] Automates BZ OCS podDisruptionBudget prevents successful OCP upgrades~~ Automates BZ OCS podDisruptionBudget prevents successful OCP upgrades May 5, 2021

PrasadDesala reviewed May 7, 2021

View reviewed changes

Shrivaibavi requested review from a team as code owners May 26, 2021 17:17

Shrivaibavi force-pushed the new_pdb_design branch 2 times, most recently from 91fcdad to 449dbac Compare June 3, 2021 14:44

Akarsha-rai reviewed Jun 7, 2021

View reviewed changes

yosibsh reviewed Jun 7, 2021

View reviewed changes

PrasadDesala reviewed Jun 23, 2021

View reviewed changes

Shrivaibavi force-pushed the new_pdb_design branch 2 times, most recently from 34ddf4c to 8cdf37a Compare August 10, 2021 15:08

yosibsh reviewed Aug 11, 2021

View reviewed changes

ocs_ci/ocs/cluster.py Outdated Show resolved Hide resolved

ocs_ci/ocs/cluster.py Outdated Show resolved Hide resolved

ocs_ci/ocs/cluster.py Outdated Show resolved Hide resolved

Shrivaibavi force-pushed the new_pdb_design branch from 8cdf37a to 91db156 Compare August 11, 2021 09:20

PrasadDesala reviewed Aug 16, 2021

View reviewed changes

ocs-ci reviewed Aug 16, 2021

View reviewed changes

OdedViner previously approved these changes Aug 17, 2021

View reviewed changes

ocs-ci reviewed Aug 17, 2021

View reviewed changes

Shrivaibavi dismissed OdedViner’s stale review via 8f9468c August 18, 2021 07:53

ocs-ci reviewed Aug 18, 2021

View reviewed changes

PrasadDesala reviewed Aug 18, 2021

View reviewed changes

tests/manage/z_cluster/upgrade/test_check_pdb_post_upgrade.py Show resolved Hide resolved

tests/manage/z_cluster/nodes/test_nodes_maintenance.py Show resolved Hide resolved

Automates BZ OCS podDisruptionBudget prevents successful OCP upgrades

f77b514

Signed-off-by: Shrivaibavi Raghaventhiran <sraghave@redhat.com>

Shrivaibavi force-pushed the new_pdb_design branch from b955224 to f77b514 Compare August 19, 2021 08:53

Shrivaibavi added the Verified Mark when PR was verified and log provided label Aug 19, 2021

ocs-ci reviewed Aug 19, 2021

View reviewed changes

Removed canary pod check as it is no more relevant

f3f2821

Signed-off-by: Shrivaibavi Raghaventhiran <sraghave@redhat.com>

Shrivaibavi removed the Needs Testing Run tests and provide logs link label Sep 1, 2021

Shrivaibavi added 2 commits September 1, 2021 19:54

Update test_check_pdb_post_upgrade.py

39e1df3

Create test_check_pdb_post_upgrade.py

12d92fb

OdedViner approved these changes Sep 1, 2021

View reviewed changes

Akarsha-rai approved these changes Sep 1, 2021

View reviewed changes

dahorak approved these changes Sep 1, 2021

View reviewed changes

prsurve reviewed Sep 2, 2021

View reviewed changes

prsurve approved these changes Sep 2, 2021

View reviewed changes

Shrivaibavi mentioned this pull request Sep 2, 2021

Automate BZ -Draining two nodes can cause mons to fall out of quorum temporarily until both nodes are up again #4793

Closed

PrasadDesala approved these changes Sep 2, 2021

View reviewed changes

PrasadDesala merged commit 836f8d9 into red-hat-storage:master Sep 2, 2021

Shrivaibavi mentioned this pull request Feb 9, 2022

Update drain and OCP upgrade scenarios to check for PDB states #3557

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automates BZ OCS podDisruptionBudget prevents successful OCP upgrades #3888

Automates BZ OCS podDisruptionBudget prevents successful OCP upgrades #3888

Shrivaibavi commented Feb 22, 2021

PrasadDesala Aug 16, 2021

Shrivaibavi Aug 16, 2021

PrasadDesala Aug 16, 2021

Shrivaibavi Aug 16, 2021

ocs-ci left a comment

ocs-ci left a comment

OdedViner left a comment

ocs-ci left a comment

ocs-ci left a comment

ocs-ci left a comment

ocs-ci left a comment

ocs-ci left a comment

OdedViner left a comment

dahorak left a comment

dahorak Sep 1, 2021

prsurve Sep 2, 2021

Shrivaibavi Sep 2, 2021

		allowed_disruptions = osd_pdb[osd].get("status").get("disruptionsAllowed")
		maximum_unavailable = osd_pdb[osd].get("spec").get("maxUnavailable")

Automates BZ OCS podDisruptionBudget prevents successful OCP upgrades #3888

Automates BZ OCS podDisruptionBudget prevents successful OCP upgrades #3888

Conversation

Shrivaibavi commented Feb 22, 2021

PrasadDesala Aug 16, 2021

Choose a reason for hiding this comment

Shrivaibavi Aug 16, 2021

Choose a reason for hiding this comment

PrasadDesala Aug 16, 2021

Choose a reason for hiding this comment

Shrivaibavi Aug 16, 2021

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

OdedViner left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

ocs-ci left a comment

Choose a reason for hiding this comment

OdedViner left a comment

Choose a reason for hiding this comment

dahorak left a comment

Choose a reason for hiding this comment

dahorak Sep 1, 2021

Choose a reason for hiding this comment

prsurve Sep 2, 2021

Choose a reason for hiding this comment

Shrivaibavi Sep 2, 2021

Choose a reason for hiding this comment