Collect log using SerialConsole when SshShell fails to connect #3845
Conversation
ssh may not succeed due to many reasons, at present if ssh fails it is either very difficult or impossible to triage the issue without reproducing it. This change uses SerialConsole to run commands for log collection.
|
@squirrelsc @LiliDeng , this PR is for early design review. This will enhance triaging of failures for both engineers and AI |
|
|
||
| # Prepare the RunCommandInput for Azure | ||
| command = RunCommandInput( | ||
| command_id="RunShellScript", |
There was a problem hiding this comment.
Is this Linux specific? Consider whether you can add support for Windows. If not possible, think about how to exclude the Feature for non-Posix systems.
adityagesh
force-pushed
the
aditya/serial_log_after_test_failure
branch
from
5a994dd to
ff40ee5
Compare
adityagesh
force-pushed
the
aditya/serial_log_after_test_failure
branch
from
ff40ee5 to
bc92811
Compare
adityagesh
force-pushed
the
aditya/serial_log_after_test_failure
branch
from
df59804 to
a69b551
Compare
Introduce NonSshExecutor. It Internally uses SerialConsole/RunCommand
adityagesh
force-pushed
the
aditya/serial_log_after_test_failure
branch
from
a69b551 to
84d4f4a
Compare
adityagesh
force-pushed
the
aditya/serial_log_after_test_failure
branch
from
ed1e6f5 to
741e95e
Compare
adityagesh
force-pushed
the
aditya/serial_log_after_test_failure
branch
from
741e95e to
4ff4695
Compare
lisa/features/non_ssh_executor.py
Outdated
| out = [] | ||
|
|
||
| serial_console = self._node.features[SerialConsole] | ||
| if not serial_console.enabled(): |
There was a problem hiding this comment.
use self._node.features.is_supported to check the feature is available or not before self._node.features[SerialConsole]
lisa/node.py
Outdated
| f"NonSshExecutor is not supported on {self.name}, " | ||
| "cannot collect logs using non-ssh executor." | ||
| ) | ||
| return None |
There was a problem hiding this comment.
It doesn't need to return None at the end.
There was a problem hiding this comment.
My bad, I wasn't thinking straight
adityagesh
force-pushed
the
aditya/serial_log_after_test_failure
branch
from
4ff4695 to
e033ae1
Compare
|
Testing with non-Linux environments are pending. |
lisa/features/non_ssh_executor.py
Outdated
| _ = serial_console.read() | ||
| for command in commands: | ||
| serial_console.write(self._add_newline(command)) | ||
| out.append(serial_console.read()) |
There was a problem hiding this comment.
read() may not get all output. How about use previous _ = serial_console.read() to get prompt, and check if the prompt is printed out.
There was a problem hiding this comment.
Makes sense, added logic
serial_console.write("\n")
response = serial_console.read()
if not response or "$" not in response and "#" not in response:
Do you recommend the above or response.strip().endswith()?
lisa/features/non_ssh_executor.py
Outdated
| _ = serial_console.read() | ||
| for command in commands: | ||
| serial_console.write(self._add_newline(command)) | ||
| out.append(serial_console.read()) |
There was a problem hiding this comment.
besides return the output, also printed them out.
There was a problem hiding this comment.
def _collect_logs_using_non_ssh_executor (the caller) is currently printing it.
Should I add an optional argument to print it in this function itself?
ssh may not succeed due to multiple reasons, at present if ssh fails it is either very difficult or impossible to triage the issue without reproducing it.
This change uses NonSshExecutor to run commands for log collection.