Introduce remote spawner functionality

[pevogam]

This pull request is in similar style and spirit to #4158 but makes use of remote aexpect sessions (to remote hosts or equivalently remote containers behind remote hosts forwarded via specific ports) as slots to schedule test runs on.

TODO: Currently highly experimental and rebased on our LXC spawner branch, to be considered for review and completion once the LXC spawner pull request is merged.

[pevogam]

@richtja @clebergnu As the concept and changes of the remote spawner is very similar to those of the LXC spawner but simply uses aexpect sessions and API instead of LXC API I wonder what is your overall opinion about such a possibility. Are you for or against supporting remote test spawning? What do you think about the aexpect (potentially in-house) backend and thus dependency for this implementation?

I have some linting to fix and a few isolation tests to add but right now I am just curious about your overall thoughts about this.

[clebergnu]

Hi @pevogam,

Thanks for yet another spawner! Please take a look at my comments.

And thanks again!

[clebergnu]

This is identical (besides the "RemoteSpawner" reference that can become a variable) to the version on the lxc spawner. Maybe come up with a version that suits both and share it?

[pevogam]

Yes, I guess we could refactor a shared mechanism for using "available slots" for these types of spawners. Will take a look.

[clebergnu]

This is also pretty much identical to the lxc version... It makes an even stronger point towards sharing more of this code IMO.

[clebergnu]

Avocado doesn't have a dependency on aexpect. So we have two options (in order of preference):

1. Make it an optional plugin
2. Protect against import errors (and when/if Spawner: require runtime operation suitability #5649 is merged, signal that in is_operational())

[pevogam]

Indeed, I can do something similar to the LXC_AVAILABLE where I also suggested it gets into the is_operational method here. I think then I will wait for the above pull request to get merged and will update this pull request accordingly right after.

[clebergnu]

So, the is_operational method will go a long way and will probably solve this. But, depending on another external library in a "built in" plugin would:

• require either an "import protection" (try/import/except)
• require development mode of the core avocado to have aexpect

IMO, the best solution here is to have this plugin (and any other with extra Python package deps) as an optional_plugin. I tried to come up with an example, and ended up with this working version: https://github.com/clebergnu/avocado/commits/pevogam_remote_spawner_review

Please consider that approach, and feel free to cherry-pick those commits.

[pevogam]

Alright, I am not sure about your opinion about having an optional LXC plugin then and the comments above about refactoring code from an optional and core plugin (or perhaps two optional plugins) seems harder if at least one of them is optional. Where would you recommend we refactor functionality as in your comment #5621 (comment) and #5621 (comment)? If you prefer of course we might as well not refactor anything from the plugins yet until both of them are fully stabilized over the longer term and simply end this with your effort to migrate the remote spawner to an optional plugin.

[clebergnu]

Hi @pevogam,

Please consider the idea of making this an optional plugin, as given as an example here.

With that, I'd be more than fine with including it on 103.0.

Thanks!

[pevogam]

Hi @clebergnu, I will make sure our integration checks pass and squash all remaining commits by the end of the day. Until then, the optional plugin usage you suggested causes static checks to fail like:

  File "/usr/lib/python3.11/site-packages/pkg_resources/__init__.py", line 800, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (aexpect 1.6.1 (/github/home/.local/lib/python3.11/site-packages), Requirement.parse('aexpect>=1.6.2'))
Error: Process completed with exit code 1.

[pevogam]

Ok, I pushed a fix on top of 102.0 at first but then noticed that the aexpect dependency is missing from most recent master so decided to simply rebase there. I also reviewed your optional plugin commit and I hope you don't mind that I patched the descriptions (seems to be a copy paste from a different plugin) as follows:

diff --git a/python-avocado.spec b/python-avocado.spec
index 080073cc4..13d5caf4d 100644
--- a/python-avocado.spec
+++ b/python-avocado.spec
@@ -384,13 +384,12 @@ a dedicated sever.
 %{python3_sitelib}/avocado_framework_plugin_result_upload*
 
 %package -n python3-avocado-plugins-spawner-remote
-Summary: Avocado Plugin to propagate Job results to a remote host
+Summary: Avocado Plugin to spawn tests on a remote host
 License: GPLv2+
 Requires: python3-avocado == %{version}-%{release}
 
 %description -n python3-avocado-plugins-spawner-remote
-This optional plugin is intended to upload the Avocado Job results to
-a dedicated sever.
+This optional plugin is intended to spawn tests on a remote host.
 
 %files -n python3-avocado-plugins-spawner-remote
 %{python3_sitelib}/avocado_spawner_remote*
diff --git a/requirements-dev.txt b/requirements-dev.txt
index 997dc3ad0..60b4bfcc6 100644

Finally, I squashed all commits to a few final ones (no draft commits left) and added a few of your leftover comments from the first review which were temporarily outdated before resolving them.

The only thing left from the CI that I can see is a few failing rpm-build variants for CentOS and similar distros.

[pevogam]

Hey, thanks for the prompt response! Maybe I misunderstood 'remote-spawner' a little? I thought it to be a way to execute tests on a remote hardware connected by some means (e.g. serial console) utilizing 'expect'. So I could run a set of tests on my embedded boards.

No, you understand this exactly right.

But it still seems to be tied to LXC containers? First day of avocado here, still trying to get the grasp of it.

What I mean is simply that the "slot approach" is similar to the way it is done for LXC containers and that you will have to define "persistent slots" (your embedded boards that persist and remain the same the testing) in a similar fashion. So it should be fairly easy to adapt for your use case. We do lack some more significant documentation for the time being since these features are too new, experimental, and purely additive to the code base so far but I have used such remote spawners for some distributed testing for almost a year now without having much trouble.

[wsakernel]

Okay, I guess I see the big picture better now. I will try. Thanks for your pointers!

[wsakernel]

Okay, I got it somewhat working, but boy, this needs documentation :) If someone else wants to try it, first add the slot to your avocado config file:

[spawner.remote]
slots = ['board']

Then you need a JSON file of the same name as the slot. Its contents are the command line parameters of the remote_login function of class remote, e.g.:

{
    "client": "telnet",
    "host": "192.168.64.2",
    "port": "23",
    "username": "root",
    "password": "",
    "prompt": "#"
}

The detail I missed so far: the remote site also needs avocado installed. It makes sense (considering e.g. python native tests), but is sadly a show stopper for me and my limited systems under test. Also, accessing remotes via serial port would be nice to have in my case. That seems to be still missing in aexpect.

So, thanks for the help up to here. But I think I am better off writing my tests in expect directly, handling the serial connection there. Makes the tests "remote-only", but this is okay for me.

[wsakernel]

Oh forgot to say, seeing avocoado finally accessing the board was fun!

[pevogam]

Okay, I got it somewhat working, but boy, this needs documentation :) If someone else wants to try it, first add the slot to your avocado config file:

[spawner.remote]
slots = ['board']

Then you need a JSON file of the same name as the slot. Its contents are the command line parameters of the remote_login function of class remote, e.g.:

{
    "client": "telnet",
    "host": "192.168.64.2",
    "port": "23",
    "username": "root",
    "password": "",
    "prompt": "#"
}

Great investigation and we will make to use the documentation you provided. On my side we use a lot of additoinal automation that not only prepares and provides such configuration but also manages the remote hosts entirely so I could not provide much simpler hints before.

The detail I missed so far: the remote site also needs avocado installed.

I am not so sure about that though, perhaps the avocado maintainers could provide better advice than me here but I think you would only need a task runner and a few other modules to be copied over. Avocado also considers every executable a potential test so you don't need python native tests for instance.

It makes sense (considering e.g. python native tests), but is sadly a show stopper for me and my limited systems under test. Also, accessing remotes via serial port would be nice to have in my case. That seems to be still missing in aexpect.

I am also not sure about this, Avocado VT for instance has serial sessions that are created via aexpect like so

    def _create_serial_console(self):
        """
        Establish a session with the serial console.

        Let's consider the first serial port as serial console.
        Note: requires a version of netcat that supports -U
        """
        if self.serial_session_device is None:
            LOG.warning("No serial ports defined!")
            return
        log_name = "serial-%s-%s.log" % (
            self.serial_session_device, self.name)
        self.serial_console_log = os.path.join(utils_misc.get_log_file_dir(),
                                               log_name)
        file_name = self.get_serial_console_filename(
            self.serial_session_device)
        self.serial_console = aexpect.ShellSession(
            "nc -U %s" % file_name,
            auto_close=False,
            output_func=utils_misc.log_line,
            output_params=(log_name,),
            prompt=self.params.get("shell_prompt", "[\#\$]"),
            status_test_command=self.params.get("status_test_command",
                                                "echo $?"))

@ldoktor Perhaps you could say much more than me especially since you mentioned before lots of users might be interested to apply aexpect in a minimal (often embedded systems) setting.

[ldoktor]

It makes sense (considering e.g. python native tests), but is sadly a show stopper for me and my limited systems under test. Also, accessing remotes via serial port would be nice to have in my case. That seems to be still missing in aexpect.

I am also not sure about this, Avocado VT for instance has serial sessions that are created via aexpect like so

    def _create_serial_console(self):
        """
        Establish a session with the serial console.

        Let's consider the first serial port as serial console.
        Note: requires a version of netcat that supports -U
        """
        if self.serial_session_device is None:
            LOG.warning("No serial ports defined!")
            return
        log_name = "serial-%s-%s.log" % (
            self.serial_session_device, self.name)
        self.serial_console_log = os.path.join(utils_misc.get_log_file_dir(),
                                               log_name)
        file_name = self.get_serial_console_filename(
            self.serial_session_device)
        self.serial_console = aexpect.ShellSession(
            "nc -U %s" % file_name,
            auto_close=False,
            output_func=utils_misc.log_line,
            output_params=(log_name,),
            prompt=self.params.get("shell_prompt", "[\#\$]"),
            status_test_command=self.params.get("status_test_command",
                                                "echo $?"))

@ldoktor Perhaps you could say much more than me especially since you mentioned before lots of users might be interested to apply aexpect in a minimal (often embedded systems) setting.

Hello @wsakernel I don't know the history, but if you're used to python than aexpect should offer greater capabilities then pure expect, therefore more reliable and faster to develop tests (not talking about easier self-recovery on failures). It allows greater flexibility and allows easier debugability than pure bash (I could recommend pydevd).

As for this runner I can't tell (haven't studied it much) but overall avocado should give you the separation and uniformity as well as various result formats.

[wsakernel]

I am not so sure about that though, perhaps the avocado maintainers could provide better advice than me here but I think you would only need a task runner and a few other modules to be copied over. Avocado also considers every executable a potential test so you don't need python native tests for instance.

Okay, it may not be a complete Avocado install, but for some use cases, even installing Python3 on the target is already too much. Think of testing bootloaders of an embedded device. But agreed, for tests covering the Linux Kernel, Python3 and a bit of Avocado is doable.

I am also not sure about this, Avocado VT for instance has serial sessions that are created via aexpect like so

    def _create_serial_console(self):
        """
        Establish a session with the serial console.

        Let's consider the first serial port as serial console.
        Note: requires a version of netcat that supports -U

Well, this needs nc -U which is again not helpful for bootloaders. Accessing it directly is desired for some of my cases.

But it is not too bad for me currently. I can do custom expect now, so I will have my test results by the end of next week as planned. Avocado will help me greatly by scheduling the tests, making nice HTML output and pushing them to my server. This is good. Converting the tests to aexpect and/or remote spawners may come later as a second step.

[wsakernel]

Hello @wsakernel I don't know the history, but if you're used to python than aexpect should offer greater capabilities then pure expect, therefore more reliable and faster to develop tests (not talking about easier self-recovery on failures). It allows greater flexibility and allows easier debugability than pure bash (I could recommend pydevd).

Hey, ldoktor. Thank you for helping me as well. Much appreciated! The thing is, I am not used to Python. I can read and modify it somewhat, but I am far more fluent in C and shell scripting. To meet my deadlines next week, I will stick to that for now. I am definately interested in using aexpect later but I'd need documentation. So far, I have not found it. Do you know of some? Or examples?

[ldoktor]

Unfortunatelly there is none, just the docstrings. The best you can do is install aexpect, run python and use help command:

>>> import aexpect
>>> import time
>>> dir(aexpect)
['Expect', 'ExpectError', 'ExpectProcessTerminatedError', 'ExpectTimeoutError', 'ShellCmdError', 'ShellError', 'ShellProcessTerminatedError', 'ShellSession', 'ShellStatusError', 'ShellTimeoutError', 'Spawn', 'Tail', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'client', 'exceptions', 'kill_tail_threads', 'remote', 'rss_client', 'run_bg', 'run_fg', 'run_tail', 'shared', 'utils']
>>> session = aexpect.ShellSession("bash")
>>> session.cmd("ls /tmp/b")
'1  2\n'
>>> session.cmd("cat /tmp/b/1")
'Hello\n'
>>> session.cmd("cat /tmp/b/2")
'World\n'
>>> dir(session)
['_ShellSession__RE_STATUS', '__class__', '__del__', '__delattr__', '__dict__', '__dir__', '__doc__', '__enter__', '__eq__', '__exit__', '__format__', '__ge__', '__getattribute__', '__getinitargs__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_add_close_hook', '_add_reader', '_aexpect_helper', '_close_aexpect_helper', '_close_reader_fds', '_get_fd', '_join_thread', '_read_nonblocking', '_start_thread', '_tail', 'a_id', 'auto_close', 'close', 'close_hooks', 'closed', 'cmd', 'cmd_output', 'cmd_output_safe', 'cmd_status', 'cmd_status_output', 'command', 'ctrlpipe_filename', 'echo', 'encoding', 'get_command_output', 'get_command_status', 'get_command_status_output', 'get_id', 'get_output', 'get_pid', 'get_status', 'get_stripped_output', 'inpipe_filename', 'is_alive', 'is_defunct', 'is_responsive', 'kill', 'linesep', 'lock_client_starting_filename', 'lock_server_running_filename', 'log_file', 'log_file_fd', 'match_patterns', 'match_patterns_multiline', 'output_filename', 'output_func', 'output_params', 'output_prefix', 'prompt', 'read_nonblocking', 'read_until_any_line_matches', 'read_until_last_line_matches', 'read_until_last_word_matches', 'read_until_output_matches', 'read_up_to_prompt', 'reader_fds', 'reader_filenames', 'readers', 'remove_command_echo', 'remove_last_nonempty_line', 'send', 'send_ctrl', 'sendcontrol', 'sendline', 'server_log_filename', 'set_linesep', 'set_log_file', 'set_output_func', 'set_output_params', 'set_output_prefix', 'set_prompt', 'set_status_test_command', 'set_termination_func', 'set_termination_params', 'shell_pid_filename', 'status_filename', 'status_test_command', 'tail_thread', 'termination_func', 'termination_params', 'thread_name']
>>> session.sendline("for I in $(seq 10); do echo $I; sleep 1; done")
>>> session.read_nonblocking(0.1, 2)
>>> time.sleep(10)
>>> session.read_nonblocking(0.1, 2)
'2\n3\n4\n5\n6\n7\n8\n9\n10\n[medic@fedora ~ \x1b[1;31m\x1b[0m]$ '
>>> session.sendline("for I in $(seq 10); do echo $I; sleep 1; done")
>>> session.read_nonblocking(1.5, 2)
'1\n2\n3\n4\n'
>>> session.sendline("for I in $(seq 10); do echo $I; sleep 1; done")
>>> session.read_until_output_matches("3", timeout=10)
(0, '1\n2\n3\n')
>>> session.sendline("for I in $(seq 10); do echo $I; sleep 1; done")
>>> session.read_until_output_matches(["5", "7", "2", "foo"], timeout=10)
(2, '1\n2\n')
>>> session.cmd_status("true")
0
>>> session.cmd_status("false")
1
>>> session.is_alive()
True
>>> session.is_responsive()
True
>>> session.sendline("exit")
>>> session.is_alive()
False
>>> session.is_responsive()
False

[wsakernel]

@ldoktor: Thank you for this starting point! It is easily understandable, so if I get to it somewhen, I will start here.

[ldoktor]

@ldoktor: Thank you for this starting point! It is easily understandable, so if I get to it somewhen, I will start here.

Btw the best benefit of python is the debugging. So let me recommend you Eclipse with pydev, then on the machine where you intend to debug python code install pydevd (pip install pydevd) and use insert import pydevd; pydevd.settrace("127.0.0.1", True, True) to the place you want to insert a breakpoint (it works remotely too if you replace the address with yours), then switch your Eclipse to debug view and start pydev server (Pydev->Start debug server). Then you run your application whatever way you are used to and once it executes the pydevd.settrace code it'll call to your Eclipse instance and you'll be able to single-step things.

I find this very useful and easier than the usual way where you need to run debug mode from IDE, this way you can run multiple iterations at once and only when it hits the pydevd.settrace it'll show up in your eclipse (provided you don't forget to start the server). Also it works well with multiple threads and debugging concurency issues.

[pevogam]

Btw the best benefit of python is the debugging. So let me recommend you Eclipse with pydev, then on the machine where you intend to debug python code install pydevd (pip install pydevd) and use insert import pydevd; pydevd.settrace("127.0.0.1", True, True) to the place you want to insert a breakpoint (it works remotely too if you replace the address with yours), then switch your Eclipse to debug view and start pydev server (Pydev->Start debug server). Then you run your application whatever way you are used to and once it executes the pydevd.settrace code it'll call to your Eclipse instance and you'll be able to single-step things.

This is a great piece of advice in fact, the only downside I have with pydev here is the lack of pure command line frontend. For a more extensive list of alternatives I recommend checking out:

https://blog.koehntopp.info/2023/03/14/tracing-python.html

[clebergnu]

@pevogam about the packit build failures:

• centos-stream-8 and centos-stream-9 do not have python3-aexpect (yet?), so they're unable to build there
• only epel-8 has python3-aexpect (not available on epel-9)

So, the most logic thing to do is to disable the spawner plugin RPM packages on all EL* distros until the situation is normalized. If you could, please add the following:

diff --git a/python-avocado.spec b/python-avocado.spec
index 2df3ee6b1..c4437243f 100644
--- a/python-avocado.spec
+++ b/python-avocado.spec
@@ -46,7 +46,9 @@ BuildRequires: python3-docutils
 BuildRequires: python3-lxml
 BuildRequires: python3-psutil
 BuildRequires: python3-setuptools
+%if ! 0%{?rhel}
 BuildRequires: python3-aexpect
+%endif
 
 %if ! 0%{?rhel}
 %if ! 0%{?fedora} > 35
@@ -132,9 +134,11 @@ popd
 pushd optional_plugins/result_upload
 %py3_build
 popd
+%if ! 0%{?rhel}
 pushd optional_plugins/spawner_remote
 %py3_build
 popd
+%endif
 rst2man man/avocado.rst man/avocado.1
 
 %install
@@ -170,9 +174,11 @@ popd
 pushd optional_plugins/result_upload
 %py3_install
 popd
+%if ! 0%{?rhel}
 pushd optional_plugins/spawner_remote
 %py3_install
 popd
+%endif
 mkdir -p %{buildroot}%{_mandir}/man1
 install -m 0644 man/avocado.1 %{buildroot}%{_mandir}/man1/avocado.1
 mkdir -p %{buildroot}%{_pkgdocdir}
@@ -382,6 +388,7 @@ a dedicated sever.
 %{python3_sitelib}/avocado_result_upload*
 %{python3_sitelib}/avocado_framework_plugin_result_upload*
 
+%if ! 0%{?rhel}
 %package -n python3-avocado-plugins-spawner-remote
 Summary: Avocado Plugin to spawn tests on a remote host
 License: GPLv2+
@@ -393,6 +400,7 @@ This optional plugin is intended to spawn tests on a remote host.
 %files -n python3-avocado-plugins-spawner-remote
 %{python3_sitelib}/avocado_spawner_remote*
 %{python3_sitelib}/avocado_framework_plugin_spawner_remote*
+%endif
 
 %package -n python3-avocado-examples
 Summary: Avocado Test Framework Example Tests

I'd also suggest a squash of the commits. IMO, there's no really any benefit to keeping the "original" location of the plugin (it will often require a --follow to see the full history).

Other than those, we're ready to merge the plugin IMO.

[pevogam]

Alright, adding it now 👍

[clebergnu]

Alright, adding it now 👍

Thanks @pevogam ! Looks like you need a rebase against the latest master (that should fix the readthedocs.org failure).

[pevogam]

Alright, adding it now 👍

Thanks @pevogam ! Looks like you need a rebase against the latest master (that should fix the readthedocs.org failure).

Was wondering whether to write about that first but wanted to see the tests complete - I think merging against the most current master will make this test pass anyway but I always advocate for rebasing since it keeps this the multi-branch git history much cleaner.

[pevogam]

Now I seem to get

suite nrunner-requirement doesn't have 12 tests it has 16.
If you made some changes into selftests please update `TEST_SIZE` variable in `check.py`. If you haven't done any changes to selftests this behavior is an ERROR, and it needs to be fixed.
error: Bad exit status from /var/tmp/rpm-tmp.HYrtFD (%check)
    Bad exit status from /var/tmp/rpm-tmp.HYrtFD (%check)

but this pull request introduced no self-tests I can think of.

[clebergnu]

Now I seem to get

suite nrunner-requirement doesn't have 12 tests it has 16.
If you made some changes into selftests please update `TEST_SIZE` variable in `check.py`. If you haven't done any changes to selftests this behavior is an ERROR, and it needs to be fixed.
error: Bad exit status from /var/tmp/rpm-tmp.HYrtFD (%check)
    Bad exit status from /var/tmp/rpm-tmp.HYrtFD (%check)

but this pull request introduced no self-tests I can think of.

The addition of the new spawner on selftests/check.py has a multiplication effect on that suite. It should be a simple matter of updating the value.

[pevogam]

Alright everything passed, thanks for the hints at various points!

[clebergnu]

LGTM @pevogam. Thanks for the patience and all the work here!