Understanding Passwordless SSH Dependency in SONiC Management Pytest Execution Environment

[Shubh-csk]

🏷️ Discussion Type

Question

Body

While working on SONiC FDB testcase execution inside the docker-sonic-mgmt environment, I encountered an interesting issue during pytest initialization and wanted to understand the expected behavior and recommended best practices from the community.

Initially, all FDB related testcases were failing during duthosts fixture initialization with the following error:

Host unreachable in the inventory

During troubleshooting, I verified the following successfully:

Inventory parsing and host mapping
DUT/PTF ansible connectivity
Manual SSH accessibility
Pytest execution and testcase discovery
Testbed configuration loading

Even though manual ansible connectivity and SSH verification were working correctly, the SONiC pytest framework was still failing internally during testcase setup.

Later, the issue was resolved after configuring passwordless SSH using:

ssh-copy-id

between:

docker-sonic-mgmt
DUT
PTF server

After enabling passwordless SSH, the pytest execution progressed successfully beyond the earlier Host unreachable stage.

I wanted to understand the following from experienced SONiC users and contributors:

Does the SONiC pytest/ansible framework internally require passwordless SSH for fixture initialization and background ansible operations?
Are there specific pytest fixtures or SONiC management utilities that depend on non-interactive SSH authentication?
What are the recommended pre-validation steps before running SONiC pytest suites in custom or new lab environments?
Is there any documented best practice for validating SSH trust relationships between docker-sonic-mgmt, DUT, and PTF before testcase execution?

Would appreciate any guidance, recommendations, or documentation references from the community regarding this workflow and environment setup behavior.

[cofeezz]

Yeah, what you ran into makes total sense once you understand how the SONiC mgmt framework works under the hood.
To answer your questions:

1. Yes, passwordless SSH is essentially a hard requirement. The docker-sonic-mgmt framework uses Ansible internally to initialize fixtures like duthosts, and Ansible by default expects non-interactive SSH. Even if manual SSH works fine, the framework spawns connections in the background during fixture setup without any TTY or prompt handling. So if it hits a password prompt, it just fails silently or throws "Host unreachable".

2. Which fixtures depend on it? Pretty much anything that touches the DUT or PTF during setup — duthosts, ptfhost, localhost (for ansible calls), and most topology-related fixtures. They all fire off ansible commands internally that require passwordless access between docker-sonic-mgmt → DUT and docker-sonic-mgmt → PTF.

3. Recommended pre-validation checklist before running pytest:

• Run ssh-copy-id from docker-sonic-mgmt to both DUT and PTF
• Verify with ssh -o BatchMode=yes <DUT_IP> echo ok. if it returns ok without prompting, you're good
• Also confirm the ansible inventory resolves correctly: ansible -i <inventory> <dut_hostname> -m ping
• Check that to ~/.ssh/known_hosts inside the container already has the DUT/PTF fingerprints (first-time connections can stall on host key confirmation)

4. Documentation reference: The SONiC mgmt repo has some setup notes in ansible/README.md and docs/testbed/README.testbed.Setup.md — worth checking those for your specific testbed type. The community Slack (sonic-dev channel on the OCP Slack) is also pretty active if you hit more edge cases.

[v-ember]

Hi @Shubh-csk , thanks for posting in GitHub Discussions!

The Repositories category is meant for questions related to repositories on GitHub. A repository contains all of your project's files and each file's revision history. I've gone ahead and moved this post to our Programming Help category, as this topic may be more relevant there and might help someone give you a nudge in the right direction. 😁

[Luke-Corwin]

Yes, passwordless SSH is generally expected. Many SONiC-mgmt pytest workflows rely on Ansible executing commands non-interactively across DUTs, PTF hosts, fanouts, and localhost. Password prompts cannot be handled during automated fixture initialization.
2. Several fixtures and utilities depend on non-interactive SSH, including duthosts, ptfhost, setup fixtures, topology discovery, command execution helpers, file transfers, log collection, and various Ansible modules invoked throughout test execution.
3. Recommended pre-validation steps before running pytest:
* Verify inventory parsing.
* Confirm ansible all -m ping succeeds.
* Verify SSH connectivity from docker-sonic-mgmt to DUT and PTF.
* Validate testbed information using testbed utilities.
* Confirm required SSH keys are present and authorized.
* Ensure Ansible can execute commands without prompts.
4. Passwordless SSH should be validated in all directions required by the testbed. At minimum:
* docker-sonic-mgmt → DUT
* docker-sonic-mgmt → PTF
* Any additional hosts referenced by the topology or fixtures
5. The “Host unreachable” error can be misleading. In some cases, inventory and basic connectivity are correct, but Ansible fails because authentication requires interactive input. From the framework’s perspective, the host becomes unreachable even though manual SSH works.
6. Best practice for new labs: Treat passwordless SSH setup as a prerequisite, not a troubleshooting step. Before running any SONiC pytest suite, verify that all required hosts can be reached via SSH and Ansible without passwords, prompts, or host-key confirmation requests.

Based on your findings, the behavior you observed is consistent with how SONiC-mgmt and Ansible are typically deployed in CI, lab, and production validation environments, where passwordless SSH is effectively a requirement for reliable test execution.