Fix test_update_forced_mgmt failed because check interfaces-config status with systemctl take 45 seconds. #12914

liuh-80 · 2024-05-21T07:31:04Z

Fix test_update_forced_mgmt failed because check interfaces-config status with systemctl take too much time.

Why I did it

Check interfaces-config status with systemctl some times take 45 seconds, however the wait timeout is 10 seconds, which cause test case break:

13/05/2024 23:01:03 base._run L0071 DEBUG | /var/src/sonic-mgmt_vms21-t0-2700_646f1402735219c3e5444094/tests/common/devices/multi_asic.py::_run_on_asics#128: [] AnsibleModule::command, args=["sudo systemctl show --no-pager interfaces-config -p ExecMainExitTimestamp --value"], kwargs={}

13/05/2024 23:01:45 base._run L0108 DEBUG | /var/src/sonic-mgmt_vms21-t0-2700_646f1402735219c3e5444094/tests/common/devices/multi_asic.py::_run_on_asics#128: [] AnsibleModule::command Result => {"changed": true, "stdout": "", "stderr": "", "rc": 0, "cmd": ["sudo", "systemctl", "show", "--no-pager", "interfaces-config", "-p", "ExecMainExitTimestamp", "--value"], "start": "2024-05-13 23:01:00.605392", "end": "2024-05-13 23:01:00.682419", "delta": "0:00:00.077027", "msg": "", "invocation": {"module_args": {"_raw_params": "sudo systemctl show --no-pager interfaces-config -p ExecMainExitTimestamp --value", "_uses_shell": false, "warn": false, "stdin_add_newline": true, "strip_empty_ends": true, "argv": null, "chdir": null, "executable": null, "creates": null, "removes": null, "stdin": null}}, "stdout_lines": [], "stderr_lines": [], "_ansible_no_log": null, "failed": false}

Work item tracking

Microsoft ADO: 27683903

How I did it

Increase wait timeout to 60 seconds, because the some time the check status command take 45 seconds.

How to verify it

Pass all test case.

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

Description for the changelog

Fix test_update_forced_mgmt failed because check interfaces-config status with systemctl take too much time.

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

qiluo-msft · 2024-05-21T21:29:14Z

tests/route/test_forced_mgmt_route.py

@@ -64,7 +64,7 @@ def hash_and_timestamp_changed(duthost, file):
        latest_timestamp = get_interface_reload_timestamp(duthost)
        return latest_hash != original_hash and latest_timestamp != last_timestamp

-    exist = wait_until(10, 1, 0, hash_and_timestamp_changed, duthost, file)
+    exist = wait_until(60, 1, 0, hash_and_timestamp_changed, duthost, file)
    pytest_assert(exist, "File {} does not change after 10 seconds.".format(file))


10

Conflicting message. Suggest use constant variable. #Closed

qiluo-msft · 2024-05-21T21:31:20Z

                    "/etc/network/interfaces",

What is your real waiting goal? Like eth0 having some expected attribute? Is it possible to wait for the real goal?

Refers to: tests/route/test_forced_mgmt_route.py:153 in 6ae8d21. [](commit_id = 6ae8d21, deletion_comment = False)

qiluo-msft · 2024-05-21T21:31:54Z

is it a typo "405 seconds"? #Closed

mssonicbld · 2024-05-22T02:43:20Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 tests/route/test_forced_mgmt_route.py:26:1: E302 expected 2 blank lines, found 1 flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

liuh-80 · 2024-05-22T07:50:13Z

                    "/etc/network/interfaces",
What is your real waiting goal? Like eth0 having some expected attribute? Is it possible to wait for the real goal?

Refers to: tests/route/test_forced_mgmt_route.py:153 in 6ae8d21. [](commit_id = 6ae8d21, deletion_comment = False)

After forced mgmt route change, hostcfgd will trigger 'interfaces-config' service to update /etc/network/interfaces.
Test case need wait for interface config update finish.

liuh-80 · 2024-05-22T07:50:26Z

is it a typo "405 seconds"?

Fixed, typo.

…atus with systemctl take 45 seconds. (sonic-net#12914) Fix test_update_forced_mgmt failed because check interfaces-config status with systemctl take too much time. #### Why I did it Check interfaces-config status with systemctl some times take 45 seconds, however the wait timeout is 10 seconds, which cause test case break: 13/05/2024 23:01:03 base._run L0071 DEBUG | /var/src/sonic-mgmt_vms21-t0-2700_646f1402735219c3e5444094/tests/common/devices/multi_asic.py::_run_on_asics#128: [] AnsibleModule::command, args=["sudo systemctl show --no-pager interfaces-config -p ExecMainExitTimestamp --value"], kwargs={} 13/05/2024 23:01:45 base._run L0108 DEBUG | /var/src/sonic-mgmt_vms21-t0-2700_646f1402735219c3e5444094/tests/common/devices/multi_asic.py::_run_on_asics#128: [] AnsibleModule::command Result => {"changed": true, "stdout": "", "stderr": "", "rc": 0, "cmd": ["sudo", "systemctl", "show", "--no-pager", "interfaces-config", "-p", "ExecMainExitTimestamp", "--value"], "start": "2024-05-13 23:01:00.605392", "end": "2024-05-13 23:01:00.682419", "delta": "0:00:00.077027", "msg": "", "invocation": {"module_args": {"_raw_params": "sudo systemctl show --no-pager interfaces-config -p ExecMainExitTimestamp --value", "_uses_shell": false, "warn": false, "stdin_add_newline": true, "strip_empty_ends": true, "argv": null, "chdir": null, "executable": null, "creates": null, "removes": null, "stdin": null}}, "stdout_lines": [], "stderr_lines": [], "_ansible_no_log": null, "failed": false} ##### Work item tracking - Microsoft ADO: 27683903 #### How I did it Increase wait timeout to 60 seconds, because the some time the check status command take 45 seconds. #### How to verify it Pass all test case. #### Description for the changelog Fix test_update_forced_mgmt failed because check interfaces-config status with systemctl take too much time.

mssonicbld · 2024-05-29T05:27:08Z

Cherry-pick PR to 202305: #13027

mssonicbld · 2024-05-29T05:27:09Z

Cherry-pick PR to 202311: #13028

…atus with systemctl take 45 seconds. (#12914) Fix test_update_forced_mgmt failed because check interfaces-config status with systemctl take too much time. #### Why I did it Check interfaces-config status with systemctl some times take 45 seconds, however the wait timeout is 10 seconds, which cause test case break: 13/05/2024 23:01:03 base._run L0071 DEBUG | /var/src/sonic-mgmt_vms21-t0-2700_646f1402735219c3e5444094/tests/common/devices/multi_asic.py::_run_on_asics#128: [] AnsibleModule::command, args=["sudo systemctl show --no-pager interfaces-config -p ExecMainExitTimestamp --value"], kwargs={} 13/05/2024 23:01:45 base._run L0108 DEBUG | /var/src/sonic-mgmt_vms21-t0-2700_646f1402735219c3e5444094/tests/common/devices/multi_asic.py::_run_on_asics#128: [] AnsibleModule::command Result => {"changed": true, "stdout": "", "stderr": "", "rc": 0, "cmd": ["sudo", "systemctl", "show", "--no-pager", "interfaces-config", "-p", "ExecMainExitTimestamp", "--value"], "start": "2024-05-13 23:01:00.605392", "end": "2024-05-13 23:01:00.682419", "delta": "0:00:00.077027", "msg": "", "invocation": {"module_args": {"_raw_params": "sudo systemctl show --no-pager interfaces-config -p ExecMainExitTimestamp --value", "_uses_shell": false, "warn": false, "stdin_add_newline": true, "strip_empty_ends": true, "argv": null, "chdir": null, "executable": null, "creates": null, "removes": null, "stdin": null}}, "stdout_lines": [], "stderr_lines": [], "_ansible_no_log": null, "failed": false} ##### Work item tracking - Microsoft ADO: 27683903 #### How I did it Increase wait timeout to 60 seconds, because the some time the check status command take 45 seconds. #### How to verify it Pass all test case. #### Description for the changelog Fix test_update_forced_mgmt failed because check interfaces-config status with systemctl take too much time.

…atus with systemctl take 45 seconds. (sonic-net#12914) Fix test_update_forced_mgmt failed because check interfaces-config status with systemctl take too much time. #### Why I did it Check interfaces-config status with systemctl some times take 45 seconds, however the wait timeout is 10 seconds, which cause test case break: 13/05/2024 23:01:03 base._run L0071 DEBUG | /var/src/sonic-mgmt_vms21-t0-2700_646f1402735219c3e5444094/tests/common/devices/multi_asic.py::_run_on_asics#128: [] AnsibleModule::command, args=["sudo systemctl show --no-pager interfaces-config -p ExecMainExitTimestamp --value"], kwargs={} 13/05/2024 23:01:45 base._run L0108 DEBUG | /var/src/sonic-mgmt_vms21-t0-2700_646f1402735219c3e5444094/tests/common/devices/multi_asic.py::_run_on_asics#128: [] AnsibleModule::command Result => {"changed": true, "stdout": "", "stderr": "", "rc": 0, "cmd": ["sudo", "systemctl", "show", "--no-pager", "interfaces-config", "-p", "ExecMainExitTimestamp", "--value"], "start": "2024-05-13 23:01:00.605392", "end": "2024-05-13 23:01:00.682419", "delta": "0:00:00.077027", "msg": "", "invocation": {"module_args": {"_raw_params": "sudo systemctl show --no-pager interfaces-config -p ExecMainExitTimestamp --value", "_uses_shell": false, "warn": false, "stdin_add_newline": true, "strip_empty_ends": true, "argv": null, "chdir": null, "executable": null, "creates": null, "removes": null, "stdin": null}}, "stdout_lines": [], "stderr_lines": [], "_ansible_no_log": null, "failed": false} ##### Work item tracking - Microsoft ADO: 27683903 #### How I did it Increase wait timeout to 60 seconds, because the some time the check status command take 45 seconds. #### How to verify it Pass all test case. #### Description for the changelog Fix test_update_forced_mgmt failed because check interfaces-config status with systemctl take too much time.

Update test_forced_mgmt_route.py

6ae8d21

liuh-80 marked this pull request as ready for review May 21, 2024 07:31

liuh-80 requested a review from qiluo-msft May 21, 2024 07:31

liuh-80 changed the title ~~Fix test_update_forced_mgmt failed because check interfaces-config status with systemctl take too much time.~~ Fix test_update_forced_mgmt failed because check interfaces-config status with systemctl take 45 seconds. May 21, 2024

qiluo-msft reviewed May 21, 2024

View reviewed changes

qiluo-msft requested a review from wangxin May 21, 2024 21:31

qiluo-msft requested a review from prsunny May 21, 2024 21:32

Update test_forced_mgmt_route.py

6e946b3

Update test_forced_mgmt_route.py

8e5cbc1

prsunny previously approved these changes May 22, 2024

View reviewed changes

Update test_forced_mgmt_route.py

bee9c8e

liuh-80 dismissed prsunny’s stale review via bee9c8e May 23, 2024 05:17

qiluo-msft approved these changes May 28, 2024

View reviewed changes

qiluo-msft merged commit c35beef into sonic-net:master May 28, 2024
14 checks passed

liuh-80 added Request for 202305 branch Approved for 202305 branch Approved for 202311 branch Request for 202311 branch labels May 29, 2024

mssonicbld added the Created PR to 202305 branch label May 29, 2024

mssonicbld added the Created PR to 202311 branch label May 29, 2024

mssonicbld mentioned this pull request May 29, 2024

[action] [PR:12914] Fix test_update_forced_mgmt failed because check interfaces-config status with systemctl take 45 seconds. #13027

Merged

9 tasks

mssonicbld mentioned this pull request May 29, 2024

[action] [PR:12914] Fix test_update_forced_mgmt failed because check interfaces-config status with systemctl take 45 seconds. #13028

Merged

9 tasks

mssonicbld added Included in 202305 branch Included in 202311 branch and removed Created PR to 202305 branch Created PR to 202311 branch labels May 29, 2024

AnantKishorSharma mentioned this pull request May 29, 2024

Increase timeout in test_forced_mgmt_route to avoid flakiness and debug info in wait_until utility #13000

Open

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix test_update_forced_mgmt failed because check interfaces-config status with systemctl take 45 seconds. #12914

Fix test_update_forced_mgmt failed because check interfaces-config status with systemctl take 45 seconds. #12914

liuh-80 commented May 21, 2024 •

edited

qiluo-msft May 21, 2024 •

edited

liuh-80 May 22, 2024

qiluo-msft commented May 21, 2024

qiluo-msft commented May 21, 2024 •

edited

mssonicbld commented May 22, 2024

liuh-80 commented May 22, 2024

liuh-80 commented May 22, 2024

mssonicbld commented May 29, 2024

mssonicbld commented May 29, 2024

Fix test_update_forced_mgmt failed because check interfaces-config status with systemctl take 45 seconds. #12914

Fix test_update_forced_mgmt failed because check interfaces-config status with systemctl take 45 seconds. #12914

Conversation

liuh-80 commented May 21, 2024 • edited

Why I did it

Work item tracking

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

qiluo-msft May 21, 2024 • edited

Choose a reason for hiding this comment

liuh-80 May 22, 2024

Choose a reason for hiding this comment

qiluo-msft commented May 21, 2024

qiluo-msft commented May 21, 2024 • edited

mssonicbld commented May 22, 2024

liuh-80 commented May 22, 2024

liuh-80 commented May 22, 2024

mssonicbld commented May 29, 2024

mssonicbld commented May 29, 2024

liuh-80 commented May 21, 2024 •

edited

qiluo-msft May 21, 2024 •

edited

qiluo-msft commented May 21, 2024 •

edited