Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Case test_encap_with_mirror_session of test_ipinip.py failed due to swss and other dockers could not start after invoking fixture setup_mirror_session #11744

Closed
echuawu opened this issue Feb 21, 2024 · 5 comments · Fixed by #12909
Closed

Case test_encap_with_mirror_session of test_ipinip.py failed due to swss and other dockers could not start after invoking fixture setup_mirror_session #11744

echuawu opened this issue Feb 21, 2024 · 5 comments · Fixed by #12909
Assignees
@bingwang-ms

Comments

@echuawu
Copy link
Contributor

echuawu commented Feb 21, 2024 •
edited

Description
After running the full cases of tests/test_pretest.py, all the container autorestart functions would be disabled due to case test_disable_container_autorestart.
Then running tests/dualtor/test_ipinip.py, case test_encap_with_mirror_session would failed for sure. Case test_encap_with_mirror_session would invoke fixture setup_mirror_session, and it would invoke fixture setup_uplink. There are steps in fixture setup_uplink, which could restart teamd docker, it would lead swss and syncd docker exits.

        # Unmask the service
        rand_selected_dut.shell_cmds(cmds="systemctl unmask teamd")
        # Restart teamd
        rand_selected_dut.shell_cmds(cmds="systemctl restart teamd")

Under the context of container autorestart functions disabled, so the affected dockers would not be up.

Steps to reproduce the issue:

  1. Run case test_disable_container_autorestart of tests/test_pretest.py
  2. Run case test_encap_with_mirror_session of tests/dualtor/test_ipinip.py

Describe the results you received:
Case test_encap_with_mirror_session would failed due to:

Failed: PortChannel didn't startup

Describe the results you expected:
Case pass even at the situation of docker autorestart function disabled

Additional information you deem important:

**Output of `show version`:**

```
(paste your output here)
```

**Attach debug file `sudo generate_dump`:**

```
(paste your output here)
```
@echuawu
Copy link
Contributor Author

echuawu commented Mar 25, 2024

Checked with MSFT team, they said they have not observed any test_ipinip.py failures. Technically speaking, if case test_disable_container_autorestart of tests/test_pretest.py run, then run case test_encap_with_mirror_session of tests/dualtor/test_ipinip.py in the same regression, then it would failed.

@bingwang-ms
Copy link
Collaborator

@echuawu Which container can't startup? Is it teamd or swss/syncd?

@echuawu
Copy link
Contributor Author

echuawu commented May 14, 2024

@echuawu Which container can't startup? Is it teamd or swss/syncd?

The reason why LAGs didn't startup - swss/syncd dockers are down after executing of systemctl restart teamd

These docker are down because there are "auto_restart": "disabled" for FEATURE in config_db:

    "teamd": {
        "auto_restart": "disabled",
   "syncd": {
        "auto_restart": "disabled",

    "swss": {
        "auto_restart": "disabled",

@bingwang-ms bingwang-ms mentioned this issue May 21, 2024
Merged
8 tasks
@bingwang-ms
Copy link
Collaborator

Hi @echuawu, I checked the test record internally but didn't see the same error. I believe that's because the autorestart is enabled somehow by some test cases. But the issue makes sense to me. I raised a PR to fix it #12909. Can you please help review? Thanks

@echuawu
Copy link
Contributor Author

echuawu commented May 21, 2024

Hi @echuawu, I checked the test record internally but didn't see the same error. I believe that's because the autorestart is enabled somehow by some test cases. But the issue makes sense to me. I raised a PR to fix it #12909. Can you please help review? Thanks

Thank you for debugging and updating! Sure, reviewed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bingwang-ms bingwang-ms

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Enable feature autorestart before test_encap_with_mirror_session bingwang-ms/sonic-mgmt
2 participants
@bingwang-ms @echuawu