Harden libvirt connection keepalive handling - #4539
Merged
Merged
Conversation
🤖 AI Test SelectionNo test cases were selected for this PR. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens the libvirt SUT orchestrator’s connection handling to avoid stale shared virConnect objects after libvirtd closes the client socket (e.g., keepalive timeouts), and adds targeted reconnect/retry logic around key domain operations.
Changes:
- Disables daemon-side keepalive while LISA owns the host (via
libvirtd.conf) and disables client-side keepalive on opened connections. - Replaces direct
defineXML/lookupByNameusage with helpers that reopen the libvirt connection and retry when a closed-socket error is detected. - Adds reconnect-and-retry handling for domain destroy/undefine when a stale socket is encountered.
The Cloud Hypervisor passthrough run failed during deployment after libvirtd closed the client connection with a keepalive timeout. Once that happened, the shared virConnect stayed stale and later environments failed immediately with 'client socket is closed'. Configure libvirtd to stop sending daemon-side keepalive probes while LISA owns the host, disable client-side keepalive probes on opened connections, and reconnect/retry focused domain operations when a stale client socket is detected.
vyadavmsft
force-pushed
the
vyadav_hune_fix5
branch
from
June 16, 2026 17:15
5a41818 to
2e10343
Compare
🤖 AI Test SelectionNo test cases were selected for this PR. |
Collaborator
Author
|
@copilot fix check errors |
Contributor
Fixed in commit |
🤖 AI Test SelectionNo test cases were selected for this PR. |
vyadavmsft
force-pushed
the
vyadav_hune_fix5
branch
from
June 16, 2026 18:26
e88e2af to
7a7d7a6
Compare
🤖 AI Test SelectionNo test cases were selected for this PR. |
…efine_domain helpers
vyadavmsft
force-pushed
the
vyadav_hune_fix5
branch
from
June 16, 2026 18:39
7a7d7a6 to
09267c8
Compare
🤖 AI Test SelectionNo test cases were selected for this PR. |
LiliDeng
reviewed
Jun 18, 2026
LiliDeng
reviewed
Jun 18, 2026
LiliDeng
reviewed
Jun 18, 2026
anirudhrb
reviewed
Jun 20, 2026
Release devices back to the libvirt passthrough pool without duplicating entries, then clear the node context so later deployments do not emit stale hostdev XML. This prevents follow-on device-passthrough tests from failing with 'Hostdev already exists in the domain configuration' after a previous deployment has released its devices.
🤖 AI Test SelectionNo test cases were selected for this PR. |
Extract the closed-connection retry flow into a shared helper and use it for domain define, lookup, destroy, and undefine operations. Add explicit domain assertions and keep cleanup retry exceptions logged through the standard exception variable.
🤖 AI Test SelectionNo test cases were selected for this PR. |
🤖 AI Test SelectionNo test cases were selected for this PR. |
vyadavmsft
force-pushed
the
vyadav_hune_fix5
branch
from
June 21, 2026 00:14
07e3305 to
395d0dd
Compare
🤖 AI Test SelectionNo test cases were selected for this PR. |
vyadavmsft
force-pushed
the
vyadav_hune_fix5
branch
from
June 21, 2026 00:18
395d0dd to
ce7bab1
Compare
🤖 AI Test SelectionNo test cases were selected for this PR. |
Route domain.interfaceAddresses through the shared libvirt reconnect helper so IP discovery can recover if the client socket closes after domain lookup.
vyadavmsft
force-pushed
the
vyadav_hune_fix5
branch
from
June 21, 2026 00:27
ce7bab1 to
8034aec
Compare
🤖 AI Test SelectionNo test cases were selected for this PR. |
Cloud Hypervisor passthrough deployments can lose the libvirt SSH transport while attaching the serial console. In the failing run, domain.openConsole() returned a virt-ssh-helper EOF/connection-reset error, which left the shared libvirt connection stale. Later deployments then failed during defineXML() with 'client socket is closed'. Treat the virt-ssh-helper EOF/reset messages as reconnectable libvirt connection errors. Wrap CH domain start plus console attach in the existing reconnect helper so the retry reopens libvirt, looks up the already-defined domain, starts it only if needed, and attaches the console again. Also clean up partially-created console streams and log files when openConsole() fails, so retries do not inherit a half-initialized console logger.
🤖 AI Test SelectionNo test cases were selected for this PR. |
anirudhrb
approved these changes
Jun 24, 2026
LiliDeng
approved these changes
Jun 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Cloud Hypervisor passthrough run failed during deployment after libvirtd closed the client connection with a keepalive timeout. Once that happened, the shared virConnect stayed stale and later environments failed immediately with 'client socket is closed'.
Configure libvirtd to stop sending daemon-side keepalive probes while LISA owns the host, disable client-side keepalive probes on opened connections, and reconnect/retry focused domain operations when a stale client socket is detected.
Description
Related Issue
Type of Change
Checklist
Test Validation
Key Test Cases:
Impacted LISA Features:
Tested Azure Marketplace Images:
Test Results